Delayed notification through telegram

[Kevin Lewis]

Alerts are coming in 10 minutes (or more) after the event - is this typical

[Brian Robertson]

I had a notification today come in about 25 minutes late. Definitely hope you all can figure out what’s going on soon because that really puts a wrench in things.

[Will Stillwell]

This is funny.  6:26 person in driveway

6:28 car is leaving

6:38 person in driveway.   As in impossible to be at 6:38 because the car pulled out at 6:28.   (Middle gif shows car pulling out)  

~Will

On Fri, May 1, 2020 at 10:31 AM Brian Robertson <brianmr...@gmail.com> wrote:

I had a notification today come in about 25 minutes late. Definitely hope you all can figure out what’s going on soon because that really puts a wrench in things.

--
You received this message because you are subscribed to the Google Groups "Camect User Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@camect.com.
To view this discussion on the web visit https://groups.google.com/a/camect.com/d/msgid/forum/71402fb9-8ca0-4be7-9f34-f61419743ddd%40camect.com.

[Will Stillwell]

Look at the time stamps on each image.  The one at 6:38 actually should have been the first.  Interesting side note, the EMAIL alert also came in last at 6:38.   So it was not just telegram this time being late, it was email too!   

~Will

[Wayne Wilson]

On Friday, May 1, 2020 at 1:30:00 PM UTC-4, Kevin Lewis wrote:
> Alerts are coming in 10 minutes (or more) after the event - is this typical

my telegrams are also about 25 minutes late. I checked telegrams web page and twitter feeds and saw no notice of server problems.

[jereme lovekin]

For the past day or so, my alerts to both email and telegram are severely delayed.  One alert/email occurred nearly an hour after the actual event:

Camera timestamp          10:40am

Telegraph Alert received: 11:34am

Email received:                11:34am

Some alerts were received within a few minutes, others were 20 minutes or longer.  The instance above took almost an hour to receive an alert.  Is the AI detection local to the Camect, or does this go out to a server for processing?  Does anyone have a fix for this?  A few seconds of delay is acceptable and expected, and anything more than that is nearly useless.

Thanks,

-Jereme

On Friday, May 1, 2020 at 1:30:00 PM UTC-4, Kevin Lewis wrote:

[Patric Sandelin]

Same here. Up to 25 minutes delay today.

[Will Stillwell]

This appears to have started after the latest Camect update. 

~Will

--

You received this message because you are subscribed to the Google Groups "Camect User Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@camect.com.

To view this discussion on the web visit https://groups.google.com/a/camect.com/d/msgid/forum/868e0090-eadd-46ca-886b-d30f74016571%40camect.com.

[KP]

Same issue, if you check the attached screenshots, you'll see an alert was detected and the telegram notification was delivered almost 40 minutes later.

This is after disabling and re-enabling the telegram bot as well.

On Friday, May 1, 2020 at 11:39:40 AM UTC-7, Will wrote:
> This appears to have started after the latest Camect update. 
>
>
> ~Will
>
>
>
>
> On Fri, May 1, 2020 at 11:19 AM jereme lovekin <motor...@gmail.com> wrote:
>
> For the past day or so, my alerts to both email and telegram are severely delayed.  One alert/email occurred nearly an hour after the actual event:
>
>
> Camera timestamp          10:40am
> Telegraph Alert received: 11:34am
> Email received:                11:34am
>
>
> Some alerts were received within a few minutes, others were 20 minutes or longer.  The instance above took almost an hour to receive an alert.  Is the AI detection local to the Camect, or does this go out to a server for processing?  Does anyone have a fix for this?  A few seconds of delay is acceptable and expected, and anything more than that is nearly useless.
>
>
> Thanks,
> -Jereme
>
> On Friday, May 1, 2020 at 1:30:00 PM UTC-4, Kevin Lewis wrote:Alerts are coming in 10 minutes (or more) after the event - is this typical
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Camect User Forum" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to fo...@camect.com.

[CamectArup]

We just made a change that should fix the email delays. The telegram delays were holding up the email too. Let us know if you continue to see email delays. 

Brian, and anyone else that has tried the new bots ... I am seeing zero evidence of delay on the new bots (at least not yet, as very few people are on them) and I am not experiencing any delay myself on them, whereas I was on the old bots. Could you give me the time of the most recent alert you got, when it should have been sent (according to the timestamp on the gif), and when you actually received it? 

To make sure you're looking at an alert that came from the new bot, click "Camect Alerts" at the top of the chat in which you're viewing your alerts, so you'll see details about the sender. The bot username should look something like "@CamectAlertsBotX4Y29V2Bot" rather than "@CamectAlertsBot"

KP: You didn't say when you switched bots relative to the time of the alert. If the alert occurred before you switched but was delivered after you switched, it could have been an alert that was already in transit to the old bot and finally got delivered. Alerts from the new bot will arrive in a new telegram chat. 

[KP]

Sorry, should have specified. I switched the bot yesterday, and it was still getting delayed today. I just verified the bot name on my telegram is @CamectAlertsX4Y29V4Bot.

[CamectArup]

Thanks.... Please use "Report Bug" from the home server settings, and provide the time an alert occurred and when you got it via telegram.  The more recent the example the better ... so if you're getting alerts frequently, wait for the next delayed one and let us know. If not, just mention the latest one you have. 

[KP]

Bug submitted. It seems like telegram alerts are currently being delivered on time, so maybe it happens sporadically.

[Will Stillwell]

I have an email alert today that was delayed 15+ minutes, that did not have telegram involved as that camera doesn't do telegram during the day. 

What is the version number for the update coming today that has this change?

~Will

--
You received this message because you are subscribed to the Google Groups "Camect User Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@camect.com.
To view this discussion on the web visit https://groups.google.com/a/camect.com/d/msgid/forum/0cb2b1d4-0db6-4d67-9005-166c53105ac4%40camect.com.

[CamectArup]

There was no update to the software on your device. The telegram and email delays were occurring in our cloud, at the point where our cloud tries to hand off the alert to telegram's cloud ... and the changes we made were entirely within the cloud. 

What time was your delayed email alert? The email delays should have stopped around 1pm Pacific time today. Telegram alerts for people on the old bots are still being delayed (not much we can do about that unless they respond), but so far I haven't seen any delays on the new ones. 

[Will Stillwell]

Ok, last delay was  12:16 to 12:34.   Email came at 12:34 time stamp was 12:16 I believe so be before 1pm.   Since then email has all been solid.   FYI I am on the new telegram bot.    Also curious WHERE in the world are you located Arup?  

~Will

On Fri, May 1, 2020 at 5:06 PM CamectArup <ar...@camect.com> wrote:

There was no update to the software on your device. The telegram and email delays were occurring in our cloud, at the point where our cloud tries to hand off the alert to telegram's cloud ... and the changes we made were entirely within the cloud. 

What time was your delayed email alert? The email delays should have stopped around 1pm Pacific time today. Telegram alerts for people on the old bots are still being delayed (not much we can do about that unless they respond), but so far I haven't seen any delays on the new ones. 

--
You received this message because you are subscribed to the Google Groups "Camect User Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@camect.com.

To view this discussion on the web visit https://groups.google.com/a/camect.com/d/msgid/forum/664a159d-8ab0-41d0-b01f-6f4802a1a950%40camect.com.

[CamectArup]

On Friday, May 1, 2020 at 6:12:00 PM UTC-7, Will wrote:

Ok, last delay was  12:16 to 12:34.   Email came at 12:34 time stamp was 12:16 I believe so be before 1pm.   Since then email has all been solid.   FYI I am on the new telegram bot.    Also curious WHERE in the world are you located Arup?  

Los Altos, California. Why do you ask? 

[Niten Jaiswal]

How do you know if you’re on the old bot?

On May 1, 2020, at 8:06 PM, CamectArup <ar...@camect.com> wrote:

There was no update to the software on your device. The telegram and email delays were occurring in our cloud, at the point where our cloud tries to hand off the alert to telegram's cloud ... and the changes we made were entirely within the cloud. 

What time was your delayed email alert? The email delays should have stopped around 1pm Pacific time today. Telegram alerts for people on the old bots are still being delayed (not much we can do about that unless they respond), but so far I haven't seen any delays on the new ones. 

--
You received this message because you are subscribed to the Google Groups "Camect User Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@camect.com.

To view this discussion on the web visit https://groups.google.com/a/camect.com/d/msgid/forum/664a159d-8ab0-41d0-b01f-6f4802a1a950%40camect.com.

[Will Stillwell]

I was curious!   Someone we / I interact with.   You mentioned the time zone.   Was just curious.   I know Wyze is head quartered in Seattle.   Just thought I would ask.   Oh, and I wanted to say I really appreciate  your effort and responses here in the forum.   You have been great to work with.   

I didn't realize that the email and telegram sending all happened in the cloud.  Though maybe it originated on device.   

~Will

--
You received this message because you are subscribed to the Google Groups "Camect User Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@camect.com.

To view this discussion on the web visit https://groups.google.com/a/camect.com/d/msgid/forum/4355e6f2-f809-4859-8f7d-9a5b4261e76c%40camect.com.

[Will Stillwell]

Niten, look at the name of the bot.   If it is @CamectAlertsbot that is the main one.  If it is @camectAlertsX4Y29V3Bot    or something similar that is new bot.    

~Will

To view this discussion on the web visit https://groups.google.com/a/camect.com/d/msgid/forum/D6A08745-96EE-4A9F-9A62-5BA648B28A49%40gmail.com.

[Will Stillwell]

From the other thread and Arup 

Moving from one of the old bots to a newer one requires cancelling and signing up with a bot again. If you want to try this, send a message saying "/cancel" in the existing chat. You should get a confirmation that you'll no longer receive alerts. Then, click your profile pic in the top bar and choose "receive alerts in telegram" as you did before ... telegram should open and then you need to click the "start" button, and you should get a confirmation that you're signed up.  (Signing up again is easiest done on a phone, where you already have the telegram app installed. Clicking the link will open the telegram app, and you should see the "start" button at the bottom of the screen.)   

~Will

[Patric Sandelin]

Nice. Seems to work like it should now.

I have had two notifications today where timestamp on cam and Telegram is the same, but it popped up on the phone 5 minutes after. I'm guessing that is a Telegram or local connection issue?

[David S.]

After struggling with the same delay issue and then trying to deal with the frustration of Telegraph and it's obvious internal technical issues... I was finally able to cancel the first bot and reinstall a new bot. After emailing with Arup he noticed that my alert settings in Camect had oddly been reset to not sending alerts.

If anyone else is having delay issue or no alerts at all... check your setting under "alerts" and make sure the "send alerts" field is checked. 

Everything is now working as it should. I am once again getting alerts and in a timely manner.

On Friday, May 1, 2020 at 1:30:00 PM UTC-4, Kevin Lewis wrote:

[KP]

> frustration of Telegraph and it's obvious internal technical issues..

Lol...

1) it's telegram, not telegraph
2) I've been using this app daily for almost 5 years, it's one of the most reliable apps I've ever used. Just because you struggle with it doesn't mean Telegram is the issue...

[CamectArup]

2) I've been using this app daily for almost 5 years, it's one of the most reliable apps I've ever used. Just because you struggle with it doesn't mean Telegram is the issue...

Given the length of time the problem has lasted, I'd agree it's probably not an internal Telegram issue, but it is certainly the case that something changed such that our original bots are now unable to submit anything more than about 1% of the stated load limits. We have attempted to contact their bot support twice about this, and got no response. (There are plenty of threads on the net saying that you usually won't hear back from them.)  

Our next guesses are that perhaps they are trying to reduce load due to increased Covid-related usage, or simply that they thought the traffic was spammy, or unevenly scheduled, or something like that and decided to punish us for it.  It's unfortunate that they don't feel inclined to document what's going on, and thus we currently have no way to know what we have to do to deliver a reliable experience on top of it :-( 

[Will Stillwell]

Instead of using a bot is it possible to have your system send a message from the person to the person.   As in my Camect sends the massage from ME to ME.   

I don't know much about Telegram but I know I can send a message to myself in the desktop app.  Shows as "Saved Messages"  

~Will

--
You received this message because you are subscribed to the Google Groups "Camect User Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to forum+un...@camect.com.

To view this discussion on the web visit https://groups.google.com/a/camect.com/d/msgid/forum/9ae739f9-2cd2-4c0e-b824-2f5c5124c87f%40camect.com.

[Rob Taft]

I’ve said it before but this is another reason...Camect needs it’s own app. I know that’s a resource heavy task but not having control of critical aspects of your service is not sustainable.

[Den]

FYI - I had to do this as I noticed the delayed notifications in the past couple of days. It worked great. Will keep my eye out on it if it appears again.

Thanks.

[ZzyzxOh]

While changing Bots helped restore most (if not all) notifications, I still get notifications out of sequence and some are significantly delayed. For example, a vehicle crossing two camera fields will have the second alert arrive first and the first alert arrives second. And an intermediate alert of the occupant dropping off a package might come after I've already retrieved the package.

So, whatever is going on with Alerts is still going on. It's just not as bad.

[CamectChao]

On Sunday, May 10, 2020 at 11:22:54 AM UTC-7, ZzyzxOh wrote:

While changing Bots helped restore most (if not all) notifications, I still get notifications out of sequence and some are significantly delayed. For example, a vehicle crossing two camera fields will have the second alert arrive first and the first alert arrives second. And an intermediate alert of the occupant dropping off a package might come after I've already retrieved the package.

So, whatever is going on with Alerts is still going on. It's just not as bad.

 

Sorry about all the inconvenience..   How often does it happen? I assume not too frequently.

We are slowly pushing up the QPS limit on our side. Things will get better and back to normal eventually. It'll take several days perhaps.

This is what we found and did recently:

- Telegram started to throttle our traffic aggressively about 2 weeks before. The public document claims 30 qps. What we got is about 0.03 qps.

- We changed our code to send MP4 instead of GIF. The former is much smaller than the latter. This has helped a lot.

- We don't know whether Telegram is still throttling our traffic and how much the QPS limit is.

  We are very cautious to avoid being flagged as spam by Telegram. So far, we have been using a relatively low QPS limit on our side and slowly pushing it up.

We'll make some changes to avoid the reorder of alerts.

[ZzyzxOh]

I will begin to collect more data for you. I assume their Throttle is on you as the Source so our Telegram account level (free) does not matter?

Are you expecting that Telegram will be replaced as the recommendation as you scale out to more devices in the field?

[CamectChao]

On Sunday, May 10, 2020 at 1:29:03 PM UTC-7, ZzyzxOh wrote:

I will begin to collect more data for you. I assume their Throttle is on you as the Source so our Telegram account level (free) does not matter?

Right. It's on our side not per user account. 

Are you expecting that Telegram will be replaced as the recommendation as you scale out to more devices in the field?

Not in the near term. 

Currently, it seems Telegram only throttle if we are sending big messages.

We don't know what's the cutoff yet. What we know is that GIF alerts are bigger than the cutoff and MP4 alerts are below it.

Without throttling, Telegram allow 30 QPS per bot and we can have multiple bots.

So as long as we don't increase the alert size dramatically, we should be fine.

Of course, we don't know whether Telegram will change the policy in the future.

In long term, we do have other plans, when we have more resources as the company grow.