Long delay on subscribe / send MQTT Messages

[Roland Koch]

Hy,

I´m using node-red 0.17.5 on a raspberry pi 3. During last test while developing on an esp8266 i found out that sending or receiveing mqtt (mosquitto) messages takes very long. 

It is totally independed how log the meassage is (1 Byte ... 250 Bytes ...). 

During investigation I also checked if the message was received with a long delay using terminal tools on the raspberry pi. I found out that the message arrived immediate from my esp8266 devices when sent from them.

I guess the delay somehow comes from node-red. Does anybody know such an issue?

Thanks and best regards,

Roland

[Colin Law]

On 6 Oct 2017 6:32 a.m., "Roland Koch" <rlnd...@gmail.com> wrote:

Hy,

I´m using node-red 0.17.5 on a raspberry pi 3. During last test while developing on an esp8266 i found out that sending or receiveing mqtt (mosquitto) messages takes very long. 

It is totally independed how log the meassage is (1 Byte ... 250 Bytes ...). 

During investigation I also checked if the message was received with a long delay using terminal tools on the raspberry pi. I found out that the message arrived immediate from my esp8266 devices when sent from them.

That doesn't quite make sense to me, you say it arrived immediately *from* ESP when sent *from* command line. Can you give a more detailed example of exactly what you see please?

Also where is the mqtt server?

Colin

I guess the delay somehow comes from node-red. Does anybody know such an issue?

Thanks and best regards,

Roland

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/5553ed74-9221-4e08-80d3-49f14c63da79%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Roland]

Hy Colin,

thanks for your answer. Please let me describe it in detail:

Devices are in local network

Different ESP8266 devices

On RPI3 is running

- Mosquitto MQTT Server

- Node Red (allowed memory extended to 384 MB)

- Samba

Example:

ESP8266 Device 1 -> Send Message 1

RPI 3 MQTT Service -> Receives Message 1 immediate

RPI 3 NR Server -> Receives Message 1 immediate and react

ESP8266 Device 1 -> Send Message 2

RPI 3 MQTT Service -> Receives Message 2 immediate

RPI 3 NR Server -> no Message receives

ESP8266 Device 1 -> Send Message 3

RPI 3 MQTT Service -> Receives Message 3 immediate

RPI 3 NR Server -> no Message receives

ESP8266 Device 1 -> Send Message 4

RPI 3 MQTT Service -> Receives Message 4 immediate

RPI 3 NR Server -> no Message receives

After about 5 - 15 seconds

RPI 3 NR Server -> Receives Message 2 and react

RPI 3 NR Server -> Receives Message 3 and react

RPI 3 NR Server -> Receives Message 4 and react

ESP8266 Device 1 -> Send Message 5

RPI 3 MQTT Service -> Receives Message 5 immediate

RPI 3 NR Server -> Receives Message 5 immediate and react

...

Roland

You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/nQX-YbKAnGI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CAL%3D0gLt6uZ-W4htefoR-p4OLvvkUsHAN3%2BH9tgPeHOBrXgmMHQ%40mail.gmail.com.

[Colin Law]

That is much clearer. First I suggest putting a debug node directly
on the mqtt-in node to check whether that node passes the message on
immediately. Perhaps you have done that already.

If that is showing the delay then disconnect the other wires from the
mqtt in so only the debug is connected and see if that makes a
difference.

Also run top in a terminal and see whether the pi processor
utilisation is high during the delay, and if so then which process is
consuming the processor.

Also tell us which version of node.js are you using and which version
of the mqtt node.

Finally run mosquitto-sub to watch all topics and make sure there is
not an mqtt avalanche going on.

Please make sure you answer all the suggestions above.

Colin

>> email to node-red+u...@googlegroups.com.

>> To post to this group, send email to node...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/node-red.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/node-red/5553ed74-9221-4e08-80d3-49f14c63da79%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> http://nodered.org
>>
>> Join us on Slack to continue the conversation: http://nodered.org/slack
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Node-RED" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/node-red/nQX-YbKAnGI/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> node-red+u...@googlegroups.com.

>> To post to this group, send email to node...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/node-red.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/node-red/CAL%3D0gLt6uZ-W4htefoR-p4OLvvkUsHAN3%2BH9tgPeHOBrXgmMHQ%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to node-red+u...@googlegroups.com.

> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/CAMFvL73-jUoWL42k4JCkgotydUh4ndjkwZnKiCFDria%2BeSbX%2BA%40mail.gmail.com.

[Roland]

Hy,

found the Problem and partially solved.

But step by step

+ Debug nodes and mosquitto_sub I already used. With them i found that a delay occured.

+ But with top i forgot to check the cpu load.

I noticed following effect:

Every 1, 2 and 2.5 minutes the cpu load of the node-red instance raised up to 100 % for about 15 - 20 seconds.

Reason: In node-red I use 1, 2, and 2.5 minute timers to collect MIN, MAX data from a database (SQLite) and doing some other stuff. Disconnecting them, the load decreases down to 0.3 ... 2 % and node-red reactions with MQTT as expected.

SQL Select with MAX and MIN seams to be a very expensive operation.

Thanks for pushing to use top

Regards

Roland

>> email to node-red+unsubscribe@googlegroups.com.

>> To post to this group, send email to node...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/node-red.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/node-red/5553ed74-9221-4e08-80d3-49f14c63da79%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> http://nodered.org
>>
>> Join us on Slack to continue the conversation: http://nodered.org/slack
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Node-RED" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/node-red/nQX-YbKAnGI/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> node-red+unsubscribe@googlegroups.com.

>> To post to this group, send email to node...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/node-red.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/node-red/CAL%3D0gLt6uZ-W4htefoR-p4OLvvkUsHAN3%2BH9tgPeHOBrXgmMHQ%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to node-red+unsubscribe@googlegroups.com.

> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/CAMFvL73-jUoWL42k4JCkgotydUh4ndjkwZnKiCFDria%2BeSbX%2BA%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
http://nodered.org

Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/nQX-YbKAnGI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send an email to node...@googlegroups.com.

Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CAL%3D0gLsH-__SU4LXe%3D2st2B5O_QOdarosZg0hu76fx0fuqsCdA%40mail.gmail.com.

[Colin Law]

If you have not got an index on the column then max and min are very expensive as it has to read all the records and compare them. If you add an index to the column it should make a huge difference.

Colin

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CAMFvL70mxtfLU%3DjzeMxM%3D2eR4X_UWtanhLUSpSr9qNnfzYZUow%40mail.gmail.com.

[Julian Knight]

In addition, you also have to check carefully the data size. SQLite needs to keep everything in memory, if your db or worse, your index, gets cached out to swap space, that is very slow indeed.

After looking at the indexes, you might want to try prepared statements and views to help optimise your queries. Triggers are another tool you can use to help keep summarised tables up-to-date when writing to your main table.

>> email to node-red+u...@googlegroups.com.

>> To post to this group, send email to node...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/node-red.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/node-red/5553ed74-9221-4e08-80d3-49f14c63da79%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> http://nodered.org
>>
>> Join us on Slack to continue the conversation: http://nodered.org/slack
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Node-RED" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/node-red/nQX-YbKAnGI/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> node-red+u...@googlegroups.com.

>> To post to this group, send email to node...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/node-red.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/node-red/CAL%3D0gLt6uZ-W4htefoR-p4OLvvkUsHAN3%2BH9tgPeHOBrXgmMHQ%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to node-red+u...@googlegroups.com.

> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/CAMFvL73-jUoWL42k4JCkgotydUh4ndjkwZnKiCFDria%2BeSbX%2BA%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
http://nodered.org

Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/nQX-YbKAnGI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to node-red+u...@googlegroups.com.

To post to this group, send an email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CAL%3D0gLsH-__SU4LXe%3D2st2B5O_QOdarosZg0hu76fx0fuqsCdA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.