ttyAMA0: 1 input overrun(s)

[John Beale]

Should I expect to see this "input overrun" line when I type 'dmesg' after ssh-ing into my R-Boom?  Curious if this is a sign of a potential problem, or just a harmless glitch?  

I do not have any additional software or hardware installed, this is just the factory-stock Raspberry Boom board attached to a Raspberry Pi Model 2B.

myshake@raspberryshake:/opt $ dmesg | tail -10

[   52.154766] eth0: renamed from veth401cda1

[   52.213610] docker0: port 3(vethf1422da) entered blocking state

[   52.213629] docker0: port 3(vethf1422da) entered forwarding state

[15624.356052] ttyAMA ttyAMA0: 1 input overrun(s)

[31594.277484] ttyAMA ttyAMA0: 1 input overrun(s)

[35669.778510] ttyAMA ttyAMA0: 1 input overrun(s)

[36714.755657] ttyAMA ttyAMA0: 1 input overrun(s)

[44174.731972] ttyAMA ttyAMA0: 1 input overrun(s)

[44234.731906] ttyAMA ttyAMA0: 1 input overrun(s)

[45744.731126] ttyAMA ttyAMA0: 1 input overrun(s)

myshake@raspberryshake:/opt $ uname -a

Linux raspberryshake 4.9.24-v7+ #993 SMP Wed Apr 26 18:01:23 BST 2017 armv7l GNU/Linux

myshake@raspberryshake:/opt $ date

Tue 31 Jul 17:54:53 UTC 2018

Running the 'top' command shows the major process is 'flask' with 14% of CPU.

-----

top - 17:57:54 up 13:03,  1 user,  load average: 0.13, 0.44, 0.48

Tasks: 169 total,   1 running, 168 sleeping,   0 stopped,   0 zombie

%Cpu(s): 14.2 us,  4.6 sy,  0.0 ni, 79.9 id,  0.5 wa,  0.0 hi,  0.8 si,  0.0 st

KiB Mem:    945524 total,   884188 used,    61336 free,   206504 buffers

KiB Swap:        0 total,        0 used,        0 free.   506960 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 1461 root      20   0   92764  34592  21604 S  14.1  3.7  40:03.63 flask

 1216 root      20   0   18888  14844   6264 S   7.2  1.6  21:23.95 python

 4367 root      20   0   63152   4664   4176 S   1.6  0.5  14:49.08 odf_SL_plugin

 1060 myshake   20   0    5112   2488   2100 R   1.3  0.3   0:00.34 top

  564 root      20   0  921928  33228  19572 S   1.0  3.5   4:19.66 docker

 1456 www-data  20   0   15856   3636   2632 S   0.7  0.4   0:29.77 nginx

 1455 www-data  20   0   15932   3644   2632 S   0.3  0.4   0:26.49 nginx

 4301 root      20   0    5236   2996   2640 S   0.3  0.3   0:14.71 purge_datafiles

 4342 root      20   0    5536   3752   3364 S   0.3  0.4   1:31.61 seedlink

22084 root      20   0       0      0      0 S   0.3  0.0   0:02.65 kworker/u8:2

22231 root      20   0       0      0      0 S   0.3  0.0   0:00.19 kworker/0:0

26843 root      20   0       0      0      0 S   0.3  0.0   0:00.56 kworker/2:2

27727 myshake   20   0   11480   3208   2604 S   0.3  0.3   0:00.12 sshd

    1 root      20   0    5448   3880   2788 S   0.0  0.4   0:04.90 systemd

    2 root      20   0       0      0      0 S   0.0  0.0   0:00.07 kthreadd

    3 root      20   0       0      0      0 S   0.0  0.0   0:06.88 ksoftirqd/0

    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H

[Branden Christensen]

John:

Buenas tardes. 

I am not sure. Is there a problem for us to solve here?

Yours,

Branden Christensen

Director, Raspberry Shake project, Social Media: @raspishake

"Vive a tu manera" - Herencia de Timbiquí

--
Some useful links:
 
Manual: http://manual.raspberryshake.org/
Do It YourSelf Page: http://raspberryshake.org/do-it-yourself
Shop: https://shop.raspberryshake.org/
Website: http://raspberryshake.org/
 
Instagram: https://www.instagram.com/raspishake/
Facebook: https://www.facebook.com/raspishake/
Twitter: https://twitter.com/raspishake/
Hashtag: #rasperryshake, @raspishake
DOI: https://doi.org/10.7914/SN/AM
---
You received this message because you are subscribed to the Google Groups "RaspberryShake" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raspberryshake+unsubscribe@googlegroups.com.
To post to this group, send email to raspberryshake@googlegroups.com.
Visit this group at https://groups.google.com/group/raspberryshake.
To view this discussion on the web visit https://groups.google.com/d/msgid/raspberryshake/2c3116a5-ce55-463f-9a92-5db9048047f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Beale]

I'm not familiar with all the inner workings of the Raspberry Boom but I had the impression that the data was transferred serially to the Raspberry Pi using the Pi's UART, which may be what /dev/ttyAMA0 represents (?)  If that is true, it seems possible that "1 input overrun(s)" means there was a data overflow coming from the Boom to the Pi board, that is the serial port buffer filled up before whatever process on the Pi read it out. If so, it seems possible there might be some loss of data. Not continually, I notice the errors do not occur all the time, only at some intervals. If there was just one sample out of 10000 that was dropped, I'm not sure it would be obvious looking at a plot of the output.  But not knowing how it all works in detail, this is just a conjecture. 

[Branden Christensen]

×

Branden Christensen

Director, Raspberry Shake project, Social Media: @raspishake

"Vive a tu manera" - Herencia de Timbiquí

--
Some useful links:
 
Manual: http://manual.raspberryshake.org/
Do It YourSelf Page: http://raspberryshake.org/do-it-yourself
Shop: https://shop.raspberryshake.org/
Website: http://raspberryshake.org/
 
Instagram: https://www.instagram.com/raspishake/
Facebook: https://www.facebook.com/raspishake/
Twitter: https://twitter.com/raspishake/
Hashtag: #rasperryshake, @raspishake
DOI: https://doi.org/10.7914/SN/AM
---
You received this message because you are subscribed to the Google Groups "RaspberryShake" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raspberryshake+unsubscribe@googlegroups.com.
To post to this group, send email to raspberryshake@googlegroups.com.
Visit this group at https://groups.google.com/group/raspberryshake.

To view this discussion on the web visit https://groups.google.com/d/msgid/raspberryshake/ec795afc-ce79-4f6e-b1a3-227fb6f153d5%40googlegroups.com.

[Chris Burton]

Hi, 

I'm not familiar with all the inner workings of the Raspberry Boom but I had the impression that the data was transferred serially to the Raspberry Pi using the Pi's UART, which may be what /dev/ttyAMA0 represents (?)

/dev/ttyAMA0 is one of the two UART on the Raspberry Pi, in this case it's the higher spec PL011 UART with a 16 byte hardware buffer - the lower spec mini UART (ttyS0) only has an 8 byte buffer.

If you're not using hardware flow control and data is sent when the 16/8 byte buffers are full the Raspberry Pi will drop bytes (this *seems* to be more of an issue with later kernels on busy Pi).

If that is true, it seems possible that "1 input overrun(s)" means there was a data overflow coming from the Boom to the Pi board, that is the serial port buffer filled up before whatever process on the Pi read it out. If so, it seems possible there might be some loss of data. Not continually, I notice the errors do not occur all the time, only at some intervals. If there was just one sample out of 10000 that was dropped, I'm not sure it would be obvious looking at a plot of the output.  But not knowing how it all works in detail, this is just a conjecture.

I don't know what the protocol of the data going over the UART is but I assume it's in a format forgiving of dropped bytes or the software knows how to synchronise to it again?

Chris.

[Richard]

hi john,

i would like ot see your log files, if possible.

it is possible that a buffer overflow would result in a lost data packet (or two), but from the dmesg output, this isn't obvious.  the log files will provide a more complete picture.

in any case, under normal operating circumstances, this shouldn't normally happen, though given that this is a coordination effort between the OS and hardware interrupts, anything goes.

thanks in advance,

richard

[John Beale]

Log files attached.

Also 'dmesg' seems to contain the input overrun message on a recurring basis, for example here's the last few lines currently. I'm not used to seeing this type of thing on the various other (non-Shake/Boom) Raspberry Pi machines that I run.  Also I have an original 50 Hz 1D Shake on the same LAN with the same software rev: System Version 0.13 and it doesn't do this either.

[753915.734493] ttyAMA ttyAMA0: 1 input overrun(s)

[754075.734171] ttyAMA ttyAMA0: 1 input overrun(s)

[754605.734189] ttyAMA ttyAMA0: 1 input overrun(s)

[754625.734195] ttyAMA ttyAMA0: 1 input overrun(s)

[754755.734570] ttyAMA ttyAMA0: 1 input overrun(s)

[758480.715171] ttyAMA ttyAMA0: 1 input overrun(s)

[764730.693550] ttyAMA ttyAMA0: 1 input overrun(s)

[765460.664283] ttyAMA ttyAMA0: 1 input overrun(s)

[766720.693189] ttyAMA ttyAMA0: 1 input overrun(s)

[766770.693170] ttyAMA ttyAMA0: 1 input overrun(s)

[766830.693149] ttyAMA ttyAMA0: 1 input overrun(s)

[767350.693053] ttyAMA ttyAMA0: 1 input overrun(s)

[768082.692959] ttyAMA ttyAMA0: 1 input overrun(s)

[781932.370608] ttyAMA ttyAMA0: 1 input overrun(s)

[798757.318834] ttyAMA ttyAMA0: 1 input overrun(s)

[820007.730621] ttyAMA ttyAMA0: 1 input overrun(s)

[820577.728414] ttyAMA ttyAMA0: 1 input overrun(s)

[830524.689044] ttyAMA ttyAMA0: 1 input overrun(s)

[834064.676869] ttyAMA ttyAMA0: 1 input overrun(s)

[836828.676779] ttyAMA ttyAMA0: 1 input overrun(s)

[837768.676920] ttyAMA ttyAMA0: 1 input overrun(s)

[838178.676890] ttyAMA ttyAMA0: 1 input overrun(s)

[John Beale]

Update with key bit of information: there was one non-standard thing I did with my R-Boom unit. About a week ago, after reading the discussion here about battery power, I switched to using an external USB-Ethernet adaptor instead of the built-in ethernet port. This appeared to reduce total power draw of the unit by 100 to 200 mW, based on my external "Kill A Watt (R)" AC power monitor.  This particular ethernet adapter is a cheap one from ebay and uses a Kontron DM9601 (USB: 0fe6:9700)  which is a USB 1.1 chip. It draws low power at the cost of lower speed. It *seemed* to work OK, but something about the adapter or its Linux driver might somehow cause more processor load and/or latency leading to an occasional serial port overrun.

To check this, I removed the USB adapter and went back to the built-in ethernet port. I did first shut down the system from the web page, and waited for the Pi's activity LED to blink at a fixed rate indicating the power down mode.  Even so, when I tried to restart, the system would not boot up, presumably the memory card was dead. I reflashed it again and restarted normally, now with the standard configuration and no outside USB adapter.  There has only been 1 hour uptime since then, but so far, no overrun messages reported from 'dmesg' so I may have found the cause, if not the exact mechanism.

[Marco Walther]

On 08/10/2018 10:07 AM, John Beale wrote:
> Update with key bit of information: there was one non-standard thing I

> did with my R-Boom unit. After reading the discussion about battery

> power, I switched to using an external USB-Ethernet adaptor instead of
> the built-in ethernet port. This appeared to reduce total power draw of

> the unit by 200 to 200 mW based on the external AC power monitor.  This
> particular  adapter is a cheap one from ebay and uses a Kontron DM9601
> which is a USB1.1 chip. It draws low power at the cost of lower speed.

> It *seemed* to work OK, but something about the adapter or its Linux
> driver might somehow cause more processor load and/or latency leading to
> an occasional serial port overrun.
>
> To check this, I removed the USB adapter and went back to the built-in
> ethernet port. I did first shut down the system from the web page, and
> waited for the Pi's activity LED to blink at a fixed rate indicating the
> power down mode.

When the Pi is down, there should be NO blinking at all any more:-( The
red power-led would stay solid and the green led should not blink at all
(for >= 30 seconds).

-- Marco

> Even so, when I tried to restart, the memory card was

> dead. I reflashed it again and restarted normally, now with the standard
> configuration and no outside USB adapter.  There has only been 1 hour
> uptime since then, but so far, no overrun messages reported from 'dmesg'
> so I may have found the cause, if not the exact mechanism.
>

> --
> Some useful links:
>
> Manual: http://manual.raspberryshake.org/
> Do It YourSelf Page: http://raspberryshake.org/do-it-yourself
> Shop: https://shop.raspberryshake.org/
> Website: http://raspberryshake.org/
>
> Instagram: https://www.instagram.com/raspishake/
> Facebook: https://www.facebook.com/raspishake/
> Twitter: https://twitter.com/raspishake/
> Hashtag: #rasperryshake, @raspishake
> DOI: https://doi.org/10.7914/SN/AM
> ---
> You received this message because you are subscribed to the Google
> Groups "RaspberryShake" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to raspberryshak...@googlegroups.com
> <mailto:raspberryshak...@googlegroups.com>.
> To post to this group, send email to raspber...@googlegroups.com
> <mailto:raspber...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/raspberryshake.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/raspberryshake/d77ee7e8-5c8c-47cb-b935-c298a23f993c%40googlegroups.com
> <https://groups.google.com/d/msgid/raspberryshake/d77ee7e8-5c8c-47cb-b935-c298a23f993c%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Richard]

hi john,

yes, your logfiles indicate at least two problems happening on the serial port (odf_SL_plugin.err, if you're interested):

1. a data loss of 260 milliseconds / 26 samples (this has occurred more than once) resulting because no data is read off of the serial port.  this is a major problem and means that, for whatever reason, no data has arrived to the serial port from the Shake board, or is not available for reading. 
   the transmission rate from the board to the SP is every 250 msecs, so the data loss is one full data packet (25 samples) plus one extra sample (this is likely due to syncing with NTP).
   
   
2. every now and then a data packet is read but is marked invalid due to a byte or two that is missing, so is thrown away, resulting again in a loss of 26 samples.

this is also evidenced by the following plot of data for yesterday into today: (yellow flags are gaps)

your power supply story is interesting when regarded in conjunction with the time-syncing algorithm that is employed.  when the software does this sync, it will, for one or two seconds, use much more CPU than normal, while also reading very intesively off the serial port.  if this spike in resource usage is not able to be supported by the power supply, then it's entirely possible we could see these types of gaps in the data.

please monitor the situation now that you have replaced the power supply and send your log files again in a couple of days.  so far, this problem has been a daily occurrance over the last 10 days.

cheers,

richard

[John Beale]

Hi Richard,

Thank you very much for looking into this! May I enquire how you generated the plot with the yellow flags showing the missing data? So far I've only used SWARM to visualize the output, but that looks like some other software. I'm curious if it is something proprietary, or if I could also use it.

Note: I am still using the same power supply, although it looks like I should try replacing it. What I did change today was a non-standard external USB-Ethernet adapter that I had previously been using for the network connection instead of the built-in port.

[Richard]

hi john,

that is my own software, SQLX, which will, in the next month or two, be made available on the SQLX website, an announcement will be made here when this happens.

like other display software, it has all the functionality you would expect from a waveform viewer, but concentrates more on analysis of the data than making helicorder plots.

how are you tty overruns going?  any change?

richard

[John Beale]

I will look forward to seeing SQLX when it is ready.  The first thing I changed was removing the external USB-Ethernet adapter which I had used, but I still had some overruns the next day. Then, based on your comments here, I replaced the power supply with a different one. Now 36 hours after restarting, I've had 0 overruns reported, so perhaps that was the problem.

The old (and now suspect) supply was a phone charger labelled for 5V at 1.8 A, meaning it should be able to deliver 5 * 1.8 A = 9 W, and it was plugged into an AC wattmeter showing the average power draw (over a few-second time frame) was always under 2 W.  So that seems to be more than a 4x safety margin, but perhaps there were very high short-term peaks in current, and/or the power supply  label was not entirely truthful, and/or the USB power cable was not good enough.

[Branden Christensen]

What are the specs on the 2 powet supplies? Thanks John.

Branden

On Tue, Aug 14, 2018, 23:59 John Beale <beale...@gmail.com> wrote:

I will look forward to seeing SQLX when it is ready.  The first thing I changed was removing the external USB-Ethernet adapter which I had used, but I still had some overruns the next day. Then, based on your comments here, I replaced the power supply with a different one. Now 36 hours after restarting, I've had 0 overruns reported, so perhaps that was the problem.

--
Some useful links:
 
Manual: http://manual.raspberryshake.org/
Do It YourSelf Page: http://raspberryshake.org/do-it-yourself
Shop: https://shop.raspberryshake.org/
Website: http://raspberryshake.org/
 
Instagram: https://www.instagram.com/raspishake/
Facebook: https://www.facebook.com/raspishake/
Twitter: https://twitter.com/raspishake/
Hashtag: #rasperryshake, @raspishake
DOI: https://doi.org/10.7914/SN/AM
---
You received this message because you are subscribed to the Google Groups "RaspberryShake" group.

To unsubscribe from this group and stop receiving emails from it, send an email to raspberryshak...@googlegroups.com.
To post to this group, send email to raspber...@googlegroups.com.

Visit this group at https://groups.google.com/group/raspberryshake.

To view this discussion on the web visit https://groups.google.com/d/msgid/raspberryshake/6afa5f35-f32e-4ba8-815b-70152164543a%40googlegroups.com.

[John Beale]

They are both compact USB power supplies intended to charge phones. It was an older one that apparently caused the issue, and I have several similar-looking ones. I originally thought it claimed to provide 5V at 1.8 A on the label, but I may have mis-remembered; looking at the unit which MAY be the one I used, says 5V 1A.  Now, even that rating I would expect to be adequate for the R-Boom, if it really delivered the full 1A, since I measured on average the unit took less than 2W total (average) power, which means only 400 mA at 5V.  Now that is average power, and maybe the instantaneous peak draw is the issue, if it exceeds 1A. I could also blame high resistance in the USB cord, but I'm still using the same one now just on a different supply, and it's working ok.

The 5V USB power supply that is working OK now (over 3 days with no overruns at all reported on the R-Boom) is a newer device that came with a 2017 model phone (ZTE z978) and claims to be capable of 10W output for fast charge mode.

-john

[David J Taylor]

They are both compact USB power supplies intended to charge phones.

[]
-john
==================================

Probably best to get the officially approved Raspberry Pi PSU, rated at 5.1V
at 2.5 A.

https://thepihut.com/products/official-raspberry-pi-universal-power-supply

Cheers,
David
--
SatSignal Software - Quality software for you
Web: http://www.satsignal.eu
Email: david-...@blueyonder.co.uk
Twitter: @gm8arv

[Branden Christensen]

Hi John:

Buenos días. 

I am surprised that you even got the Shake to work on a 1 Amp supply.

Our official requirement/ recommendation for the Shakes is a 5V 2.5 Amp supply. Cell phone chargers do not meet that spec. The vast majority are 2 Amp supplies.

Have a good weekend. 

Yours, 

Branden Christensen

Director, Raspberry Shake project, Social Media: @raspishake

"Vive a tu manera" - Herencia de Timbiquí

--
Some useful links:

Manual: http://manual.raspberryshake.org/
Do It YourSelf Page: http://raspberryshake.org/do-it-yourself
Shop: https://shop.raspberryshake.org/
Website: http://raspberryshake.org/

Instagram: https://www.instagram.com/raspishake/
Facebook: https://www.facebook.com/raspishake/
Twitter: https://twitter.com/raspishake/
Hashtag: #rasperryshake, @raspishake
DOI: https://doi.org/10.7914/SN/AM
--- You received this message because you are subscribed to the Google Groups "RaspberryShake" group.

To unsubscribe from this group and stop receiving emails from it, send an email to raspberryshake+unsubscribe@googlegroups.com.

To post to this group, send email to raspber...@googlegroups.com.
Visit this group at https://groups.google.com/group/raspberryshake.

To view this discussion on the web visit https://groups.google.com/d/msgid/raspberryshake/B3A27FFC1921434FBDEFD828A3DA329F%40Alta.

[Branden Christensen]

Hi John:

Would you be open to shipping me the power supply that you think caused the issue? I have not succeeded in reproducing this condition in the lab so I want to go right to the source.

Branden

On Thu, Aug 16, 2018, 17:16 John Beale <beale...@gmail.com> wrote:

They are both compact USB power supplies intended to charge phones. It was an older one that apparently caused the issue, and I have several similar-looking ones. I originally thought it claimed to provide 5V at 1.8 A on the label, but I may have mis-remembered; looking at the unit which MAY be the one I used, says 5V 1A.  Now, even that rating I would expect to be adequate for the R-Boom, if it really delivered the full 1A, since I measured on average the unit took less that 2W total (average) power which means only 400 mA at 5V.  Now that is average power, and maybe the instantaneous peak draw is the issue, if it exceeds 1A. I could also blame high resistance in the USB cord, but I'm still using the same one now just on a different supply, and it's working ok.

The 5V USB power supply that is working OK now (over 3 days with no overruns at all reported on the R-Boom) is a newer device that came with a 2017 model phone (ZTE z978) and claims to be capable of 10W output for fast charge mode.

-john

On Wednesday, August 15, 2018 at 4:48:45 AM UTC-7, Branden Christensen wrote:

What are the specs on the 2 powet supplies? Thanks John.

--

Some useful links:
 
Manual: http://manual.raspberryshake.org/
Do It YourSelf Page: http://raspberryshake.org/do-it-yourself
Shop: https://shop.raspberryshake.org/
Website: http://raspberryshake.org/
 
Instagram: https://www.instagram.com/raspishake/
Facebook: https://www.facebook.com/raspishake/
Twitter: https://twitter.com/raspishake/
Hashtag: #rasperryshake, @raspishake
DOI: https://doi.org/10.7914/SN/AM
---
You received this message because you are subscribed to the Google Groups "RaspberryShake" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raspberryshak...@googlegroups.com.
To post to this group, send email to raspber...@googlegroups.com.
Visit this group at https://groups.google.com/group/raspberryshake.

To view this discussion on the web visit https://groups.google.com/d/msgid/raspberryshake/60d3c917-6d92-4f53-b76f-732c81289401%40googlegroups.com.

[Branden Christensen]

John:

Buenas noches.

With v.15, tty overruns in dmesg are now logged to /opt/log so would now never miss our attention.

Great suggestion on your part. Thank you.

Change Log

Kind regards from Panama,

Branden Christensen

CEO, Raspberry Shake, Social Media: @raspishake

"Vive a tu manera" - Herencia de Timbiquí

--
Some useful links:
 
Manual: http://manual.raspberryshake.org/
Do It YourSelf Page: http://raspberryshake.org/do-it-yourself
Shop: https://shop.raspberryshake.org/
Website: http://raspberryshake.org/
 
Instagram: https://www.instagram.com/raspishake/
Facebook: https://www.facebook.com/raspishake/
Twitter: https://twitter.com/raspishake/
Hashtag: #rasperryshake, @raspishake
DOI: https://doi.org/10.7914/SN/AM
---
You received this message because you are subscribed to the Google Groups "RaspberryShake" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raspberryshak...@googlegroups.com.
To post to this group, send email to raspber...@googlegroups.com.
Visit this group at https://groups.google.com/group/raspberryshake.

To view this discussion on the web visit https://groups.google.com/d/msgid/raspberryshake/2c3116a5-ce55-463f-9a92-5db9048047f7%40googlegroups.com.