Weird behaviour: input/output error and no commands after few minutes

[Niels Jakob Buch]

Hi there, new to Beaglebone Blue but not happy so far.

Using newest image bone-debian-8.7-iot-armhf-2017-03-19-4gb

Flashing to MMC, working fine. 

Configuring wifi to my local AP in /etc/network/interfaces

Running apt-get update, apt-get upgrade without problems.

Rebooting from that, all is ok, connecting wlan0 automatically

But after a while, without myself being able to pinpoint why, the terminal on 192.168.7.2 starts to behave strangely. 

I cannot run commands, they only return "input/output error"

Cloud9 and NodeRed returns "Cannot / GET" in the browser and there is only one way which is a powercycle.

How do I debug that?

[William Hermans]

Get, and connect a serial debug cable. Connect to your support PC, use puTTY or another serial client software to view the kernel output as it's running, and what errors are being displayed, when this happens.

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beagleboard/1d8a155d-75d7-4aa1-8ad6-ffaf194a4aac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Niels Jakob Buch]

That makes sense, will try that.

But are there no log-file where this kernel output is stored?

On Wednesday, 17 May 2017 22:48:24 UTC+2, William Hermans wrote:

Get, and connect a serial debug cable. Connect to your support PC, use puTTY or another serial client software to view the kernel output as it's running, and what errors are being displayed, when this happens.

On Wed, May 17, 2017 at 1:25 PM, Niels Jakob Buch <njb...@gmail.com> wrote:

Hi there, new to Beaglebone Blue but not happy so far.

Using newest image bone-debian-8.7-iot-armhf-2017-03-19-4gb

Flashing to MMC, working fine. 

Configuring wifi to my local AP in /etc/network/interfaces

Running apt-get update, apt-get upgrade without problems.

Rebooting from that, all is ok, connecting wlan0 automatically

But after a while, without myself being able to pinpoint why, the terminal on 192.168.7.2 starts to behave strangely. 

I cannot run commands, they only return "input/output error"

Cloud9 and NodeRed returns "Cannot / GET" in the browser and there is only one way which is a powercycle.

How do I debug that?

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.

[William Hermans]

On Wed, May 17, 2017 at 2:12 PM, Niels Jakob Buch <njb...@gmail.com> wrote:

That makes sense, will try that.

But are there no log-file where this kernel output is stored?

Well it could be, but I do not think Robert persists logs between reboots. Maybe I'm wrong. So then assuming if what I say is true, you'd have to hope the error is actually logged. Then the problem is that if you lose all connectivity, how do you propose to see that information ?

With a serial debug cable, you'll be directly connected to a tty terminal, so networking issues will not effect that. Which means not only can you get additional debug info, but you can also log into the system, for further troubleshooting if needed.

You can try to use:
root@wgd:~# more /var/log/messages

or
root@wgd:~# more /var/log/kern.log

But I do not think either of these will do you any good. You can check to make sure though. Do keep in mind there will be a lot of info in there that does not have anything to do with what you're looking for. So you may run out of time before you get done reading through the files.

However, if you have another Linux machine, you could scp those files to a remote system. If you are using Windows, there is a program called WinSCP I believe that should allow you to traverse into the /var/log directory, and copy those files out to your desktop.

 

[William Hermans]

A couple of other things you could try.

1) Occasionally installing Linux to an sdcard for this purpose can fail silently. You could try to re-image the sdcard in hopes that this might be the case in your situation.

2) Pick a slightly newer, or slightly older sdcard image, install that. Then see if the problem persists.

[Niels Jakob Buch]

Will try that if attaching the cable does not work....

Now comes the noob question: How do I attach the cable, seems that there is plenty BeagleBone Black tutorials on that, but not sure how to do it on the blue one.

[Dennis Lee Bieber]

On Thu, 18 May 2017 06:01:49 -0700 (PDT), Niels Jakob Buch
<njb...@gmail.com> declaimed the following:

>Will try that if attaching the cable does not work....
>
>Now comes the noob question: How do I attach the cable, seems that there is
>plenty BeagleBone Black tutorials on that, but not sure how to do it on the
>blue one.
>

Based on the spec sheet, you'll likely have to wire the interface to a
4-pin JST-SH connector, since UART0 appears to be the fourth socket on the
outside long edge of the card.

I can't tell if the card has male or female connectors, so I don't know
if it may be possible to just run bare jumpers into the connector.
--
Wulfraed Dennis Lee Bieber AF6VN
wlf...@ix.netcom.com HTTP://wlfraed.home.netcom.com/

[William Hermans]

UART0 looks to be tied to a 4pin JST-SH connector. How exactly that works out, you'll have to tell me. I do not own a beaglebone blue. Software  wise though, it should be exactly the same.

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beagleboard/700db256-78a9-4198-8a41-4f9f477df553%40googlegroups.com.

[Niels Jakob Buch]

Argh, waiting on a pile of JST-XX connectors to get the Blue-world up and running. But thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.

[William Hermans]

On Thu, May 18, 2017 at 2:06 PM, Niels Jakob Buch <njb...@gmail.com> wrote:

Argh, waiting on a pile of JST-XX connectors to get the Blue-world up and running. But thanks!

So, I actually have no idea of the pin pitch on those JST headers. But the serial debug cable I bought years ago, and still use. Has fairly thin slip on connectors. You only need RXD, TXD, and ground connected. The cable I bought was a Prolific PL2303HX cable very similar to the one stocked at Adafruit https://www.adafruit.com/product/954

I actually bought mine off ebay for less than $3, but shipping took a lot longer too. Adafruit is stating on their page there that they're now using SilLabs something or another because of driver stability. But I've never experienced a problem with the PL2303HX cable.

Anyway, my point is that you may not need a JST connector for this purpose.

[Niels Jakob Buch]

I just checked and the JST-SH connector pins, inside the terminal are only 1mm apart. https://is.alicdn.com/img/pb/863/022/524/524022863_862.jpg

I guess I will wait for the wire assemblies.

[jef...@obadal.com]

Any update on this problem. I ordered a beaglebone blue from Arrow and I'm having the same problems. Writing/reading to the eMMC just randomly fails. They sent me another one, this ones a little better... but eventually i get input/output errors. I thought it was the power supply maybe at first, but I hooked them up to a known googd power supply that's 12V 5A (60w) and it still does this.

[msard...@gmail.com]

I also am having the same issue. I thought it was broken hardware so I bought a new one and the same thing happened again. I found that I can usually induce the errors by transferring a large amount of files using scp. 

I'm going to check UART0 for additional information once I get the cabling set up. I really hope this gets patched soon, it's a very irritating bug.

One time I was able to get some debug information before the I/O errors consumed everything, not sure if it's helpful in any way.

Here's the last few lines given from dmesg

[14188.582576] wlan0: RX AssocResp from 06:18:d6:57:8c:90 (capab=0x431 status=0 aid=2)

[14188.593611] wlan0: associated

[14188.641133] wlcore: Association completed.

[25043.096569] hrtimer: interrupt took 136276 ns

[30864.563462] mmcblk1: error -110 sending status command, retrying

[30864.597101] mmcblk1: error -110 sending status command, retrying

[30864.638167] mmcblk1: error -110 sending status command, aborting

[30864.644451] blk_update_request: I/O error, dev mmcblk1, sector 3487760

[30864.651172] blk_update_request: I/O error, dev mmcblk1, sector 3487768

[30864.657840] blk_update_request: I/O error, dev mmcblk1, sector 3487776

[30864.664510] blk_update_request: I/O error, dev mmcblk1, sector 3487784

[30864.671146] blk_update_request: I/O error, dev mmcblk1, sector 3487792

[30864.677767] blk_update_request: I/O error, dev mmcblk1, sector 3487800

[30864.684382] blk_update_request: I/O error, dev mmcblk1, sector 3487808

[30864.690996] blk_update_request: I/O error, dev mmcblk1, sector 3487816

[30864.697652] blk_update_request: I/O error, dev mmcblk1, sector 3487824

[30864.708291] Aborting journal on device mmcblk1p1-8.

[30864.769510] mmcblk1: error -110 sending status command, retrying

[30864.817684] mmcblk1: error -110 sending status command, retrying

[30864.865245] mmcblk1: error -110 sending status command, aborting

[30864.871498] blk_update_request: I/O error, dev mmcblk1, sector 8192

[30864.877900] Buffer I/O error on dev mmcblk1p1, logical block 0, lost sync page write

[30864.886607] EXT4-fs error (device mmcblk1p1): ext4_journal_check_start:56: Detected aborted journal

[30864.895940] EXT4-fs (mmcblk1p1): Remounting filesystem read-only

[30864.902042] EXT4-fs (mmcblk1p1): previous I/O error to superblock detected

[30865.101035] mmcblk1: error -110 sending status command, retrying

[30865.115710] mmcblk1: error -110 sending status command, retrying

And here are the last few lines of /var/log/syslog

Aug  4 20:37:33 beaglebone connmand[794]: ntp: time slew -0.003904 s

Aug  4 20:37:33 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:39:03 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:39:04 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:40:34 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:40:35 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:42:05 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:42:14 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:43:44 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:43:56 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:45:26 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:45:27 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:46:57 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:46:57 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:48:27 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:48:32 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:50:02 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:50:13 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:51:43 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:51:47 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:53:17 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:53:17 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:54:47 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:54:37 beaglebone connmand[794]: ntp: time slew -0.010474 s

Aug  4 20:54:47 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:56:17 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:56:17 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:57:47 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:57:47 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 20:59:17 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 20:59:22 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:00:52 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:01:02 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:02:32 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:02:32 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:04:02 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:04:03 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:05:33 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:05:33 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:07:03 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:07:03 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:08:33 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:08:33 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:10:03 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:10:12 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:11:42 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:11:41 beaglebone connmand[794]: ntp: time slew -0.009679 s

Aug  4 21:11:51 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:13:21 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:13:22 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:14:52 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:14:53 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:16:23 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:16:35 beaglebone rsyslogd-2007: action 'action 17' suspended, next retry is Fri Aug  4 21:18:05 2017 [try http://www.rsyslog.com/e/2007 ]

Aug  4 21:17:01 beaglebone CRON[4839]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Aug  4 21:17:01 beaglebone kernel: [30864.563462] mmcblk1: error -110 sending status command, retrying

Aug  4 21:17:01 beaglebone kernel: [30864.597101] mmcblk1: error -110 sending status command, retrying

Aug  4 21:17:01 beaglebone kernel: [30864.638167] mmcblk1: error -110 sending status command, aborting

Aug  4 21:17:01 beaglebone kernel: [30864.644451] blk_update_request: I/O error, dev mmcblk1, sector 3487760

Aug  4 21:17:01 beaglebone kernel: [30864.651172] blk_update_request: I/O error, dev mmcblk1, sector 3487768

Aug  4 21:17:01 beaglebone kernel: [30864.657840] blk_update_request: I/O error, dev mmcblk1, sector 3487776

Aug  4 21:17:01 beaglebone kernel: [30864.664510] blk_update_request: I/O error, dev mmcblk1, sector 3487784

Aug  4 21:17:01 beaglebone kernel: [30864.671146] blk_update_request: I/O error, dev mmcblk1, sector 3487792

Aug  4 21:17:01 beaglebone kernel: [30864.677767] blk_update_request: I/O error, dev mmcblk1, sector 3487800

Aug  4 21:17:01 beaglebone kernel: [30864.684382] blk_update_request: I/O error, dev mmcblk1, sector 3487808

Aug  4 21:17:01 beaglebone kernel: [30864.690996] blk_update_request: I/O error, dev mmcblk1, sector 3487816

Aug  4 21:17:01 beaglebone kernel: [30864.697652] blk_update_request: I/O error, dev mmcblk1, sector 3487824

[Jeffrey Obadal]

I don't have my logs with me, but I believe mine looked pretty much just like that. Unfortunately both of mine spits out errors just trying to update Debian. Using an SD Card delays the error a little longer, but eventually it gets some disk access errors and goes into read only mode. I wonder if there is a way to slow down read/write and if that would fix it. I mean, besides updating it I don't really care about writing much after that.

I contacted the RMA email on the beagleboard site, and Arrow. Never heard back from beagleboard. Arrow said they would have the RMA people contact me, been a week already. I bought this for testing until their new X15 comes out, but at this rate I think I'll wait a major revision of X15 after it first comes out.

Jeff

--

For more options, visit http://beagleboard.org/discuss
---

You received this message because you are subscribed to a topic in the Google Groups "BeagleBoard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beagleboard/TgPgtCMf87U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beagleboard+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beagleboard/e41f0e13-3458-4614-b78c-f6c7b1265ab0%40googlegroups.com.

[megl...@gmail.com]

Same problems on new Beagle Bone Blue.
Unpacked, powered up, worked like 5 minutes until IO errors.

Kind regards,

Bm

To unsubscribe from this group and all its topics, send an email to beagleboard...@googlegroups.com.

[megl...@gmail.com]

I have tried:

- 4 distinct power supplies (3 x 5V, 1x 12V)

- SD card and built in memory

- 3 different OS images 

-- latest 'iot' image from https://beagleboard.org/latest-images

-- image referenced in BeagleBoneBlue FAQ, https://debian.beagleboard.org/images/bone-debian-8.7-iot-armhf-2017-03-19-4gb.img.xz.

-- image referenced on Straswon Design,  https://rcn-ee.com/rootfs/bb.org/testing/2017-03-07/iot/bone-debian-8.7-iot-armhf-2017-03-07-4gb.img.xz 

Nothing helped. The input/output errors show usually after some writes to mmc and dmesg is flooded with errors.

I have also found on StrawsonDesign github this thread: https://github.com/StrawsonDesign/Robotics_Cape_Installer/issues/87 and StrawsonDesign mentioning that:

I've found this on about 10 out of 70 beaglebone blues, many of those boards also show flash errors when running badblocks. I think it's a hardware issue with the emmc as I don't see it when running off an SD card.

In my case it also doesn't work with SD cards. I concluded that my BBBL is faulty and returned it.

Kind regards

[shr...@gmail.com]

I have the same problem. I ordered two of those but both of them have I/O error. 

This happens with the eMMC and the SD card as well. 

dmesg shows memory block errors.

Is it the hardware? Did I get 2 out of those 10 faulty ones?

[mnc...@googlemail.com]

Just received BeagleBone Blue and had the same issue as described here when trying to perform a distribution upgrade, which led me here.

My steps to recreate the problem:-

1) I created this image https://rcn-ee.net/rootfs/bb.org/testing/2019-03-24/stretch-console/bone-debian-9.8-console-armhf-2019-03-24-1gb.img.xz on 16GB SD card using https://etcher.io/

2) Booted beagle using the new image on SD card

3) ssh to the beaglebone blue

4) enabled wifif to connect to my wifi

5) sudo apt-get -y update

6) sudo apt-get -y dist-upgrade


After running step 6 it fails with input\output error after about 5-10 seconds.




Did anyone have any success on finding a solution?

How did you guys get on when receiving a new BeagleBone blue.

after this point the device is servilely hindered as not many commands work

[Jeffrey Obadal]

The company I ordered from replaced it. Their guess was it had bad memory somewhere, or some other faulty component.  Replacement had the same issue. Gave up after that.

--

For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to a topic in the Google Groups "BeagleBoard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beagleboard/TgPgtCMf87U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beagleboard...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beagleboard/9e52eca1-375e-4898-8225-56a08194c968%40googlegroups.com.

[Matt Carless]

I think I managed to get past this issue. I tried another SD card, it was a really old one I had lying around, 2GB in size. I'm pretty sure there was nothing wrong with the previous 16GB card as it was used in my action camera. Fsck also found no errors.

Using the 2GB card I went through the same process as when using the 16GB card. This time round it went through the distribution upgrade procedure and seems to be working ok.

I no face another problem with arducopter but that's for another forum.

Maybe it's the issue is something to do with the SD cards class?

To view this discussion on the web visit https://groups.google.com/d/msgid/beagleboard/CAM0CNLPF6OBxkZdv5HjmxM6GpPLeGwv0LOfKDKAHMO7LxZLgTA%40mail.gmail.com.

[Bartosz Meglicki]

I requested and got a refund the next day.

As far as I remember I tested it with original and updated BBB image and the problem persisted.

To view this discussion on the web visit https://groups.google.com/d/msgid/beagleboard/CAM0CNLPF6OBxkZdv5HjmxM6GpPLeGwv0LOfKDKAHMO7LxZLgTA%40mail.gmail.com.

[Jason Kridner]

On Thu, May 18, 2017 at 5:06 PM Niels Jakob Buch <njb...@gmail.com> wrote:

Argh, waiting on a pile of JST-XX connectors to get the Blue-world up and running. But thanks!

I happen to really like the https://www.seeedstudio.com/Grove-Universal-4-Pin-to-Beaglebone-Blue-4-Pin-Female-JST-SH-Conversion-Cable-10-pcs-pack-p-3026.html to get to 100-mill connections where I can simply plug in jumper wires. I use those connectors for Grove modules with Blue as well.

Regards to your challenges with what sounds like your root file system becoming unreadable (unable to execute commands), are you monitoring heat and power in any way? Is there free room on your root file system? Indeed getting a serial log should help you see what kernel errors are happening, but I'd be cognizant of what environment your running in when creating posts.

BTW, the network connection is typically supported through connmanctl rather than /etc/network/interfaces. 

 

To view this discussion on the web visit https://groups.google.com/d/msgid/beagleboard/456811c9-7078-4c39-8297-88c2a33ddcb5%40googlegroups.com.