Node-red throughput limits

[Andrea Giordano]

Hi, 

I built a node-red-environment on a Raspberry Pi3 with 10 mosquitto consumers and 10 kafka producers.

Then I send 10 different streams of data (messages of 113 bytes) at 50 Hz.

I noted that sometimes during the execution the message (lost connection) of node-red appears, or registers a lack of connection with kafka and both disappearing after some seconds.

I though that the problem was my internet connection but I tried with others and the issue is stil there. So I tried using sending just 2 streams

and the errors don't appear anymore.

Could be the problem concerns node-red? Is it not able to handle the previous load?

[Ben Hardill]

What is the CPU usage while this lot is running?

Without knowing exactly what transformations you are doing on the messages it's impossible to guess what the max throughput of the system may be.

[Andrea Giordano]

the raspberry has a quad core cpu.

The cpu usage measured with SAR is always under 60%. The memory also is not saturate.

[Andrea Giordano]

The trasformation I apply on data is very simple, I receive a string of text and send the same one enriched. Essentially towards Kafka the message's size is 1 KB.

So in input there is a throughput of 10*0,113*50 = 57 KB/s while in output it is 10*1*50 = 500 KB/s.

[Colin Law]

Remember that node and node-red are single threaded so will not make
full use of all cores.

A common reason for mqtt disconnection is if you have used the same
client id in multiple connections. Each time a new connection for that
id is made the original is chucked off.

If not that then I suggest trying it with the same connections but a
slower rate and see if that makes a difference.

Colin

On 5 November 2017 at 17:11, Andrea Giordano

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/node-red/82b7f54d-69c6-41e3-9ebb-6e5be838fcb5%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Andrea Giordano]

Actually the disconnection regards node-red (the lost connection error appears in the top of the browser page in the red box).

I tried also with an ingestion rate of 25Hz and the situation do not present errors.

I'm quite wondered node-red can't handle this limited throughput.

[Julian Knight]

Exactly how are you receiving this data? What nodes are you using. An example flow would be helpful.

As Colin says, so far we can't tell whether 1 core has been saturated. Nor do we know whether the way you are receiving the data is having a negative impact.

Trying to process 500 messages per second is quite high for a simple Pi. Especially if you are having to process in the messages, send them through a flow and then send them on to the broker which then also has to process them.

You could also write a small node.js programme to send the messages direct to the broker and see how well that performs.

[Julian Knight]

Oh, you also need to check how much disk activity there is. This is a real killer on a Pi with just an SD card.

[Andrea Giordano]

Essentially I do everything on node-red. I have 10 pipelines in a flow. They are composed of a default mqtt node to receive data, a json node, a javascript function node and the kafka node found here: node-red-contrib-kafka-node. the first three nodes were already installed in node-red.

I'm thinking to implement a service with python which perform the same operations since I'm not so good with js.

What do you mean exactly for disk activity? I have still the SAR logs I can check rapidly.

[Colin Law]

Have you got retain set when writing to mqtt (assuming the server is on the pi)

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/01744842-4777-42a2-8266-9f8b852272cc%40googlegroups.com.

[Colin Law]

Also what QOS?

[Nick O'Leary]

One tip would be to avoid the Function node unless you really need it. This is because the Function node does have a performance overhead due to the way it runs code in a sandbox.

For example, the Change node can be used to modify the message payload in lots of ways - particularly if you use the jsonata expression option.

Nick

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/2cc2b17d-8c80-4dde-9771-54eb82387546%40googlegroups.com.

[Andrea Giordano]

What do you mean for retain set? 

I used the standard mqtt node without any configuration. 

The broker is installed on the raspberry, I enlarged just the limit about number of input connections and messages. 

I set also inflight_messages to 0, I would like to set QOS to 0 but I didn't find any config in mosquitto.conf to do it.

[Andrea Giordano]

What is executed in the function is essential but fundamentally it is an append to a string. I never heard of "Change" node, I'm looking for it on google but I don't find anything about it.

[Nick O'Leary]

The Change node is one of the core nodes already in your palette. If you provide an example of what your Function is doing we can advise how best to do it with other core nodes.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/a7b30bfb-1a62-4136-9103-65a9b94b7b05%40googlegroups.com.

[Julian Knight]

You need to check what disk (filing system) activity is going on when you run your flow. There are lots of things that might touch the filing system including all the many logs that a typical Pi is accumulating (including Node-RED and Mosquitto), Mosquitto's retention database, etc. But also, if you start to get close to your memory limit, Linux will start to try and offload programme memory to file. This is called paging and will utterly kill the performance of your Pi because the Pi uses an SD card which is terribly slow compared to other storage types.

You need to run a tool to collect the OS metrics in order to see this stuff. The simplest would be a command line application called "top". There are various takes and improvements on that (atop, htop) or you could go with something more sophisticated such as glances (python based). If you want your Pi running smoothly for a long period, you might consider a proper metrics tool such as telegraf which you can use with Graphana for producing dashboards.

[Julian Knight]

There is an option in MQTT to retain the last message on any topic. That message is replayed whenever a client subscribes to the topic. In order to retain the message, the broker (Mosquitto in this case) has to write it to the filing system.

[Julian Knight]

You can append to a string in the change node. Either using a regular expression (which allows you to capture part/all of your original string and include it in the replacement) or using JSONata.

[Andrea Giordano]

Here a template of my code:

The input of the function is the outcome of a json node while the output is the kafka producer.

jsonPayload = msg.payload;

timestampInDate = jsonPayload["timestamp"];

splitted = timestampInDate.split(".");

ts = splitted[0];

ms = splitted[1];

parsed = Date.parse(ts)/1000;

millisecondsValue = parsed + Number(ms);

sensing = "https://iotdb.org/pub/iot-facet#sensor.spatial.acceleration"

familiarSensorName = "Acceleration Sensor x-y-z"

commercialSensorName = "ACC_SensorXYZ"

graphID = millisecondsValue.toString()+commercialSensorName+"-"+"LC"

location = "http://www.semanticweb.org/mimu-wear/wear-body#LeftCalf"

x_comp = jsonPayload["LC-acc-x"];

y_comp = jsonPayload["LC-acc-y"];

z_comp = jsonPayload["LC-acc-z"];

send_to_Kafka = '{"@context":{"generatedAt":{"@id":"http://www.w3.org/ns/prov#generatedAtTime","@type":"http://www.w3.org/2001/XMLSchema#date"},"Sensor":"http://purl.oclc.org/NET/ssnx/ssn#Sensor","SensorName":"http://purl.oclc.org/NET/ssnx/ssn#SensingDevice","ObservedParam":"http://purl.oclc.org/NET/ssnx/ssn#MeasurementProperty","Observes":"http://purl.oclc.org/NET/ssnx/ssn#observes","hasPosition":"http://www.loa-cnr.it/ontologies/DUL.owl#hasLocation","hasValue":"http://purl.oclc.org/NET/ssnx/ssn#hasValue"},"@id":"'+graphID+'","generatedAt":"'+timestampInDate+'","@graph":[{"@id":"http://www.sensor-market-example/sensor/'+commercialSensorName+'","@type":"Sensor","SensorName":"'+familiarSensorName+'","Observes":"'+sensing+'"}, {"@id":"'+sensing+'","@type":"ObservedParam","hasValue":"'+x_comp+'#'+y_comp+'#'+z_comp+'"},{"@id":"http://www.sensor-market-example/sensor/'+commercialSensorName+'","@type":"Sensor","SensorName":"'+familiarSensorName+'","hasPosition":"'+location+'"}]}';

var out = { _msgid:"code", payload:send_to_Kafka};

return out;

[Andrea Giordano]

Understand. I tried to found an option to disable retain in mosquitto but I didn't find it. In my mosquitto.conf I only set

 

persistence false 
connections_messages true
max_inflight_messages 0
max_queued_messages 0
max_connections -1

Moreover I quit and restart mosquitto often so the retained messages should be deleted at every restart.

[Julian Knight]

Retain is set per topic. Once you send a message with the retain flag, it will be! To get rid of the retained msg, you send a blank message to the topic WITH RETAIN SET (somewhat counter-intuitive).

[Julian Knight]

I think all of that could be done using JSONata. JSONata does take a bit of getting used to though.

If you want a quick win, maybe try switching from the function node to the contrib unsafe function node (look it up in the flows site). That doesn't use the VM so may be faster.

JSONata probably the best long term solution though.

[Andrea Giordano]

I checked a bit.

When the stream is launched node-red process has a cpu% always greater than 96%. When the stream in node.red output is stuck or show an error finding the kafka leader map process appears with a 100%. Mosquitto is always under 20% and in general memory is always under the 60-70%.

[Andrea Giordano]

Done. mmm I noted that mqtt nodes in node-red has an option retain set to false by default. Is it not enough?

Moreover, in topics options I set QOS to 0.

[Colin Law]

On 6 November 2017 at 00:31, Andrea Giordano

<andrea.g...@gmail.com> wrote:
> I checked a bit.
> When the stream is launched node-red process has a cpu% always greater than
> 96%. When the stream in node.red output is stuck or show an error finding
> the kafka leader map process appears with a 100%. Mosquitto is always under
> 20% and in general memory is always under the 60-70%.

What happens if you stop the mosquitto server when it is locked up? I
have suggested trying that before but I don't think you have
responded.

Colin

[Julian Knight]

Interesting. It would appear that neither Node-RED nor kafka processes are behaving nicely.

On the NR front, I'm not sure that is a common issue - I know that some other people are processing streams and I've not heard them complain. To understand where that is happening would need some time spent digging through a debugger I think.

[Andrea Giordano]

It's the node-red server that is locked, not the mosquitto one.

Anyway I tried, simply when node-red restart itself the mosquitto consumers are offline.

Clearly the mosquitto and node-red process disappear from top since there are no more data to compute.

[Nick O'Leary]

Andrea - thanks for sharing your Function code. If you could provide a simple message you are passing into that and what the result looks like, it would help see what needs doing. JSONata within the change node would be able to do most of that... only part that may be tricky is the date handling - something JSONata doesn't have in it today.

Nick

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/5c872e15-80c6-46c0-ab51-67b004671647%40googlegroups.com.

[Colin Law]

On 6 November 2017 at 11:01, Andrea Giordano
<andrea.g...@gmail.com> wrote:

> It's the node-red server that is locked, not the mosquitto one.

I am aware of that, well don't try stopping mosquitto if you don't
want to, its up to you. I suggest trying it though. If the problem is
an mqtt flood then that will tell you, it is easy to lock up node-red
and other processes using the server with a flood, The mosquitto
server itself is not necessarily going to be show any issues when
processes using it are overloaded.

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/5c872e15-80c6-46c0-ab51-67b004671647%40googlegroups.com.