Serial Port reliability

[Peter Scargill]

I don't really know if this is the right place to ask this question - but one has to start somewhere.

I am now thousands of miles away from home and I've left Node-Red in charge of various duties around the house - there are Two Raspberry Pis running Node-Red and the secondary one is using the serial port to talk to an Arduino - which is fastened to a 433Mhz receiver setup to read all sorts of sensors.

So Node-Red Serial is reading the 433Mhz incoming data.  Well, about an hour after we left the UK, that bit stopped working - and I spent the entire journey griping about crappy Arduino software. It was not until last night that I sat down and remoted into the Node-Red doing the job.  All was well but no incoming data in the serial port (this had worked well for weeks before we left).  

Though I was happy to blame  the Arduino, just because I could, I reset the Raspberry Pi remotely - it powered up, Node-Red powered up and LO AND BEHOLD the data started coming into the serial port - nothing to do with the Arduino after all.   Of course until it fails again, at which point I will ONLY restart Node-Red in an effort to pin this down, I won't be able to take this much further.

I was just wondering if anyone has any idea through experience of how I could either now or later, try to narrow down this problem and how to "bodge" a solution (can't physically do anything as I'm no-where near the machine for the rest of the summer).

Pete.

[Dave C-J]

Good enough place to ask... but not one I've personally seen of course :-) My Pi at home has 4 USB serial things reporting into it and touch wood - so far all seem to keep running... - though I must admit I do tend to restart it now and again so I can test all those updates you all want ;-)

But yes - more seriously - the serialport is supposed to retry if it detects disconnection and so on - but of course there may be an edge case you have found where for some reason it failed... or indeed may be some other totally unrelated reason (like a memory issue or ???) . Tricky to tell until you can see the actual logs or anything. 

[Peter Scargill]

Failed again and this is what I see in syslog......  seems to me this might be useful to someone who knows their stuff..

[Peter Scargill]

So one minute - serial stuff is coming into the serial port - no problem - text from my weather station then.... bang - it isn't.

[Colin Law]

Googling for the kernel message yields some possibly interesting hits such as
https://github.com/raspberrypi/linux/issues/1187

Possibly worth having a look through some of them.

Colin

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/32d67a4a-2d50-4d32-a0a4-8799a3b24739%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dave C-J]

good spot Colin

and another - https://www.esp32.com/viewtopic.php?t=255

same suggested bandaid.

[Peter Scargill]

And, gentlemen, friend Antonio independently pointed out this option - and sad though it is to have to go down this route  - I have indeed lobotomised the Pi USB down to USB 1.1

Time will tell, thank you very much for your input.

Pete.

[Dave C-J]

please let us know - this may be a tip that others will use...

[Peter Scargill]

Well, in my fantasy world I'd like it to keep working until my return to the UK in October....

Possibly more realistically I've made a note in my diary to comment back in here in 2 weeks - that will hopefully be 2 weeks of night and day operation

[Peter Scargill]

Erm, you know what I said about 2 weeks.

Apr 24 22:10:38 pi433 Node-RED[377]: 24 Apr 22:10:38 - [warn] [function:Process incoming] Wind Speed: 0km/h

Apr 24 22:10:38 pi433 Node-RED[377]: 24 Apr 22:10:38 - [warn] [function:Process incoming] Wind Dir: 247.5 degrees

Apr 24 22:10:51 pi433 kernel: [ 4908.645033] ch341-uart ttyUSB0: usb_serial_generic_read_bulk_callback - urb stopped: -32

Apr 24 22:14:05 pi433 dbus[381]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service'

Apr 24 22:14:05 pi433 systemd[1]: Starting PackageKit Daemon...

Apr 24 22:14:05 pi433 PackageKit: daemon start

Apr 24 22:14:05 pi433 dbus[381]: [system] Successfully activated service 'org.freedesktop.PackageKit'

Apr 24 22:14:05 pi433 systemd[1]: Started PackageKit Daemon.

Apr 24 22:17:01 pi433 CRON[2301]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

With my limited knowledge of log files (syslog) exactly the same error within an hour..... third line down.....

Depressed...

[Dave C-J]

Sound like the USB chip is falling.

[Csongor Varga]

Let me know how it goes. I am also struggling with the modbus serial node. It kinda works, I deploy a new version of the code and the Modbus read node goes to waiting state and stays there forever.

I know there is another thread on the modbus serial specifically, but I could not find much help there.

Csongor

[Garry Hayne]

Peter,

could this be a power supply problem on the Pi?

Garry

[Peter Scargill]

Hi Garry

Funny enough - power on the Pi was the subject of one of the best discussions we've had on my blog - http://tech.scargill.net/a-question-of-lifespan/  where we started discussing SD lifespan and branched out into power and (some of us) discovered all sorts of things about Pi power - including tests and that board WAS on excellent power.

But... while I was in the UK the whole thing worked a treat for weeks - but just before I left the UK, I (probably stupidly) took my excellent 5amp USB supply with me and plugged that Pi into a normal USB mains adaptor - granted one able to give out reasonable power.

SO:  Dave C-J asks if if could be a duff USB chip... if he means a duff USB chip on the Arduino, NO, because restarting Node-Red on the Pi always brings it back to life - indicating that the data is working on the Arduino constantly. The fact that I don't have to reset the Pi also suggests it is something to do with the UART and it's interface with Node-Red.

Last night when I was having trouble, I'd added 

dwc_otg.speed=1

onto a new line in /boot/cmdline.txt

and it made no difference. Last thing last night- I moved it to the first line - which then became in total..

dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait dwc_otg.speed=1

Related or not, it has not failed since then.  Someone who knows more about that file than I will have to say whether putting it back onto the first line would actually make a difference or whether I've been lucky (in each case I rebooted after doing this).

So - could it be power causing the issue? Possibly - sadly as I'm nowhere near that board until October my testing ability is limited other than to run the power tests we came up with in the blog.

But hopefully this conversation will be helpful to anyone else struggling with this.   The Pi2 incidentally is fully up to date as to operating system, Node-Red etc...

For me now, if indeed it fails again - my only viable scenario is, given that a Node-Red restart fixes the issue, to find a somewhat less dramatic way to re-start the Node-Red serial software other than to kill Node-Red. Monitoring is easy - I can put a time on the output of the UART which SHOULD be generating traffic every 30 seconds or less.... but what to actually DO if it stops....

[Dave C-J]

does a re-deploy work enough to kick it?

or a partial re-deploy ?

[Simone Bonacini]

Hi, maybe I am wrong, but if I remember well when a serial connection is established on the Arduino, doesn't it reset? Could it be the Arduino just freezing and getting reset when you restart node-red that is re-establishes the connection?

[Garry Hayne]

Hi Pete,

the reason I suggested this was because I had similar problems with the network and after changing the power supply did the trick.

There must be a way for the pi to self-test it's power supply?

Garry

[Peter Scargill]

Absolutely - we covered this on the blog - the next step would be to read the registers via the script that TKAISER came up with - in Node-Red - and report by email if anything went wrong. I ran out of time to do all of that but yes you can to some extent test the supply.

[Peter Scargill]

I can't tell you that until the next time it fails. After rebooting yesterday with that fix to cmdline.txt it has not failed.... so I've left that note in my diary in case it all goes swimmingly well and I forget about it - to write back  in here (last time I said that it promptly stopped).

[Peter Scargill]

If I remember the reset on Arduino has something to do with the DTR line for programming...  Not sure - but that too is a point to look at.

[Walter Kraembring]

Someone who knows more about that file

Peter, I think you have to have it on one single line with a space between each, otherwise I fear it will not be recognized correctly as command line parameters. The suggested fix also mentioned to add it to the existing line. I have made the same change on one of my Pi's where I have noticed a similar problem but not that frequent. My case is also a bit special. Using a USB to RS232 converter listening to data packages spitted out from our heat pump once a minute at 38400. To start I used the serial node, everything just based on Node-RED *things*. Suddenly data stopped coming in, the port seemed to be "hanging". I had to restart Node-RED to make it work again. The port could work for weeks so the problem might not be related to yours but anyway...

So after a while experiencing those stops, I decided to try a different approach; I wrote a small python program instead of using the serial node. Polling the serial port from Python, pushing data packages to Node-RED via MQTT. But result is more or less the same, could work for weeks, then suddenly stop.

Now I have made the same modification as you, so let's see how this will continue...  

[Walter Kraembring]

For me this change did not work out well, my Pi stopped responding on the network after a couple of hours (The fix does also limit the performance of the NIC, just so that you are aware of it)

Right now testing with another param to see if it works better (dwc_otg.fiq_fix_enable=0). I found the following explanation to this param:




dwc_otg.fiq_fix_enable -> 1 (default now) give about 10% extra performance to ARM when USB is not busy by lowering the number of interrupts USB does

 

[Csongor Varga]

Hi Guys,

My fresh observation using Modbus serial RTU (with node-red-contrib-modbus) is the following:

1. node is working correctly, I am getting the data I am expecting.

2. I change something in the flow (not the modbus node), I deploy and the modbus node goes to waiting state and stays there forever.

3. I change the modbus settings from 8 bit to 7 bit. I deploy the changes, the modbus node is now communicating, but fails since the settings are wrong.

4. I change the settings back to 8 bit, redeploy and everything works just fine. I am back to step 1.

Do you think the above symptom is related to your problem? Or it appears to be related to the node itself?

Regards,

Csongor

[Dave C-J]

from the sidelines... in that case I'd guess that the node isn't closing down / restarting cleanly - so the close method would be the place to start looking.