[Issue] BeagleBone Black Random Reboot

[Illutian Kade]

Has anyone experienced their BBB rebooting at random?

It was running solid for two days straight and this morning it just keeps rebooting. This last time the power light even went out.

*I've checked and all cables are secured. The entire setup is running of a PFC UPS; so power fluctuations shouldn't be an issue.

*CPU doesn't seem to be under load; ~20% utilization

*I've done a total unplug-let-sit and restart

*From what logs I've looked through (message and kern.log) the board does a total restart and loses even the date (goes form Oct 5 to Aug 26)

*I've not made any changes other than unplugging the Syba C-media USB sound card and transferring it back to my main computer after I wake up (BBB is doubling as a media player)

-This has worked for a grand total of 'uptime' 2 days and some odd hours with no issues.

..I really hope this doesn't mean the board is defective :\

OS: Debian Wheezy (BBB-eMMC-flasher-debian-7.1-2013-08-26.img)

Power: AC Adapter (http://www.adafruit.com/products/276) [specifically recommended by Adafruit]

[Illutian Kade]

UPDATE

Appears the DC port on the board failed.

[Gerald Coley]

That would be a first! I suspect it may be a grounding issue.

Gerald

On Mon, Oct 28, 2013 at 7:45 AM, Illutian Kade <delv...@gmail.com> wrote:

UPDATE

Appears the DC port on the board failed.

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Illutian Kade]

Ya, figured it was something wrong with the port. The AC Adapter works fine; 5.1v on a meter.

The board works fine; when powered by USB using the Debug Port.

I wonder if I can use a 5v,2a "Fast Charger" port or if it'll burn out the BBB's Debug Port....

[kavitha bk]

Good you know the reason
But It is always better if you print your PMIC last boot status during next boot or even the Processor last boot reason so that next boot you can see the reason for reboot
https://android.googlesource.com/kernel/omap.git/+/android-omap-tuna-3.0-ics-mr1/arch/arm/mach-omap2/resetreason.c

Kavitha

[Maxim Podbereznyy]

You will not burn the board but Nothing will change because when powering the board by a usb port the software limits the current to some level and when this limit is exceeded then the board reboots. Therefore even if you give 100A to the usb port the pmic will limit it

29 окт. 2013 г. 9:47 пользователь "Illutian Kade" <delv...@gmail.com> написал:

[Illutian Kade]

That's a bit of a relief.

[Illutian Kade]

And now the board won't power on from the Debug Port, but will form the DC jack. At this point I'm about to say "fuck this shit".

[lei...@gmail.com]

I have similar issue. I have (2) versions of BBB (A5A and A5C). They both randomly reboot themselves while I am running TI prebuilt BBB android image (from a couple hours to ten to fifteen hours). When I plug in the USB (DC is still powered) for logging with logcat, the reboot issue seems to disappear.

I don't have problem with BBB Angstrom image (based on 3.8 kernel). I don't have problem with Andrew Henderson's android image (based on 3.8 kernel) either.

Another issue is that when I run TI BBB android image, the clock randomly jumps forward 2^17 seconds. This happens on both of my BBB boards. The problem goes away when I run Angstrom or Andrew's android. 

I suspect it has something to do with the processor, DDR3 (BBB: AM3359 1GHz + 512MB DDR3), the configuration, or apply workaround of errata. We have an AM335x EVM kit (AM3359 720MHz + 256MB DDR2). I also loaded TI prebuilt android image. I have run it for several months. It is rock solid. I never had problem with it.

Here are the links to my other posts in regarding to this issue.

https://groups.google.com/forum/#!category-topic/beagleboard/advanced/5qSJ4dQdar4

http://e2e.ti.com/support/embedded/android/f/509/t/297726.aspx

[Robert Nelson]

On Tue, Nov 12, 2013 at 8:37 AM, <lei...@gmail.com> wrote:
> I have similar issue. I have (2) versions of BBB (A5A and A5C). They both
> randomly reboot themselves while I am running TI prebuilt BBB android image
> (from a couple hours to ten to fifteen hours). When I plug in the USB (DC is
> still powered) for logging with logcat, the reboot issue seems to disappear.
>
> I don't have problem with BBB Angstrom image (based on 3.8 kernel). I don't
> have problem with Andrew Henderson's android image (based on 3.8 kernel)
> either.
>
> Another issue is that when I run TI BBB android image, the clock randomly
> jumps forward 2^17 seconds. This happens on both of my BBB boards. The
> problem goes away when I run Angstrom or Andrew's android.

This time 'issue' was fixed in u-boot sometime last year, so I'm
guessing the TI image has an un-patched u-boot..

>
> I suspect it has something to do with the processor, DDR3 (BBB: AM3359 1GHz
> + 512MB DDR3), the configuration, or apply workaround of errata. We have an
> AM335x EVM kit (AM3359 720MHz + 256MB DDR2). I also loaded TI prebuilt
> android image. I have run it for several months. It is rock solid. I never
> had problem with it.
>
> Here are the links to my other posts in regarding to this issue.
> https://groups.google.com/forum/#!category-topic/beagleboard/advanced/5qSJ4dQdar4
> http://e2e.ti.com/support/embedded/android/f/509/t/297726.aspx
>

Regards,

--
Robert Nelson
http://www.rcn-ee.com/

[Lei Wang]

I think Robert is correct. I fetched a snapshot of u-boot (2013.10), build and dropped it in. It seems that the time jump problem is going away.

I also did some testing with TI's kernel. I disabled the OPPs at 1G, 720M, and 600MHz. So the highest OPP is 500MHz. It seems that BBB is running much more robust. I ran the system overnight without seeing random reboots.

I wonder if someone has some insight into this random reboot problem. 

Why it seems a slower clock helps? 

Why a USB connection also seems help (it doesn't seem to decrease the BogoMIPs)?

What is the main difference between 3.8 kernel and 3.2 kernel?

Thanks!

[Illutian Kade]

It could be a power 'level' issue. Most modern sound systems have a built-in kill switch. That if the volume spikes the speakers are turned off to prevent damage.. A similar system is probably in the BBB that prevents it from *receiving* to much power.

It has to be something like that because on a whim I tried to run my BBB before going to bed. In the time it took to get ready for bed (~5min) the BBB had powered down on it's own (no power LED light). Sometimes it runs for days with no issue...other times, like previously mentioned, mere minutes.

And this happens on both USB and AC power (adapter is the recommended one by Adafruit).

[ance...@gmail.com]

I confirm that I have the same issue with a BBB A5B using TI 3.2 kernel.

After reading the full thread I guess that the posible workarounds are:

- Using Angstrom 3.8 kernel.
- Powering the BBB from USB.
- Limiting the CPU frequency.

Can anybody confirm it?

Best regards.

[Lei Wang]

It seems that Angstrom 3.8 kernel is more robust. But I couldn't confirm that it will resolve the random reboot issue. I haven't done long term testing. I remember seeing someone also has problem with it. It could be just that kernel is more optimized which draws less current.

Limiting the CPU frequency only made BBB random reboot less often. I can confirm that. I believe it is also related to the current draw. The slower the clock the less the power is drawn from PMIC.

By plugging in both DC power and USB cable (mini connector), it increases the PMIC current capacity. PMIC has two power paths (AC->SYS and USB->SYS), which reduces the stress placed on a single path (AC->SYS). I am suspecting the PMIC thermal shutdown causes the random reboot. Please take a look at the other thread I mentioned in a previous post. Jakub suggested putting in a Mosfet to bypass VDD_5V to SYS_5V path after boot up. 

BTW, could you tell me what is your setup besides using TI prebuild kernel? Are you driving a HDMI display or a LCD cape? what else do you connect to the BBB?

Thanks,

Lei

[ance...@gmail.com]

After reading the Jakub thread the new conclusion is that this seems to bee a hardware related problem (related to PMIC). This may explain why changing kernel or changing CPU frequency doesn't resolve the problem (only minimizes it).

 I will test the Jakub hardware workaround (external MOSFET) in my own system.

About your questions:

- I'm using a BBB A5B with a custom cape that includes: a LVDS converter for a LCD, a resistive touchscreen, a MAX3232 converter for a RS232 port and a buffer for some push buttons.

- I have recompiled the TI 3.2 kernel (from LINUXEZSDK-BONE v06.00) to include the proper LCD initialization.

- The system runs a Qt application with CPU load below 10%. The CPU works all time at default 1 GHz frequency.

- I have two identical systems under test and I got one reboot each 2 to 3 hours but some days both systems work without reboots for 10 to 15 hours.


Best regards.

[Antonio Cebrián]

I have captured a BBB A5B random reboot with the oscilloscope (see attached image).

  Ch1 → VDD_3V3B (P9.4)
  Ch2 → VDD_5V (P9.6)
  Ch3 → SYS_5V (P9.8)
  Ch4 → SYS_RESETn (P9.10)

This confirms that the random reboot is produced by 1 second SYS_5V fall.

Best regards.

2013/11/20 <ance...@gmail.com>

[Lei Wang]

Thanks for sharing the captured trace. It is very helpful. I wonder if you could further zoom in on the falling edge. I am really interested in the order of SYS_5V voltage drop and SYS_RESETn voltage drop. 

Also I noticed in TPS65217 datasheet, if the nRESET pin is pulled low, there will be a minimum 1s delay before the PMIC returns to Active state (page 15). The nRESET pin is not connected on BBB. It should have an internal pull-up resistor of 100k (supposed to an alway-on supply). TI's engineer may have better insight on this.

I did some measurement on the current consumptions of different Kernels (showing below). As you can tell 3.8 Kernels do draw less current.

image	Angstrom (3.8 kernel)	Android(3.8 kernel)	Android(3.2 kernel)
clock (Hz)	1G	1G	1G
Display type	HDMI	HDMI	HDMI
current (mA)	290	300	375

[peter...@gmail.com]

My guess is that some piece of the kernel is setting PWR_EN low (see pages 15 & 16 in the TPS65217 Datasheet). This would explain the 1 second duration of the off state. 

The system could restart due to PWR_EN floating high. 

It does not explain why connecting to the USB power prevents the random reset.

The PMIC_POWR_EN is driven by the RTC module in the AM335x (see section 20.3.3.8 in the AM335x TRM). 

Is there a difference in the way the 3.2 and 3.8 kernels handle the RTC or sleep modes? 

On Thursday, November 21, 2013 7:53:20 AM UTC-5, Antonio Cebrián wrote:

[Maxim Podbereznyy]

I doubt the issue is connected with the floating nRESET pin. If you have a look at the "Figure 1. Global State Diagram" at page 16 the PMIC always waits for 1 second after a *FAULT*. Please read the following:

FAULT = UVLO || OTS || PGOOD low|| PWR_EN pin not asserted within 5s of Wakeup event.
If no battery is present, OVP on AC input also leads to OFF mode. With battery present, device switches
automatically from AC to BAT if AC is>6.5V and back to AC when voltage recovers to<6.5V.
Device will remain in RESET state for at least 1s.

UVLO = Under Voltage Lockout

OTS = Over Temperature Shutdown

OVP = Over Voltage Protection

2013/11/21 Lei Wang <lei...@gmail.com>

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

LinkedIn - http://www.linkedin.com/in/maximpodbereznyy

Company - http://www.linkedin.com/company/mentorel

Facebook - https://www.facebook.com/mentorel.company

[Maxim Podbereznyy]

I don't think that connecting a USB cable reduces the current load to the PMIC because there are two different switches in the power path for the AC and USB inputs. They can't be opened at the same time because the PMIC can't be sure that two sources have the same voltage. If both switches are opened then current can flow backwards to a weaker source IMHO. Probably it's really a ground issue as Gerald said before

2013/11/21 Maxim Podbereznyy <lisa...@gmail.com>

[ance...@gmail.com]

I have soldered a 15 ohm power resistor between SYS_5V (P9.8) and DGND (P9.2). This increases SYS_5V load by 333mA. After three days of test I can confirm that random reboot frequency have not changed so it seems that the current load is not the problem.

BTW: I have found and unexpected TPS65217C behavior related to the USB power detection. I have posted this issue at http://e2e.ti.com/support/arm/sitara_arm/f/791/t/305879.aspx

Best regards.

[Lei Wang]

Follow your finding on the unexpected TPS65217C behavior, I patched the tps65217 driver with irq handling. The 3.2 kernel does not handle nNMI/PMIC_INT interrupt; the 3.8 kernel does. I placed printk in the interrupt handler and got the same result as yours. The PMIC_INT was issued every 2 seconds which is caused by the USBI flag in the TPS65217C interrupt register. 

[  217.095367] tps65217_irq: USB power status change

[  217.259338] tps65217_irq: USB power status change

[  219.096801] tps65217_irq: USB power status change

[  219.256103] tps65217_irq: USB power status change

[  221.094177] tps65217_irq: USB power status change

[  221.262084] tps65217_irq: USB power status change

[  223.095611] tps65217_irq: USB power status change

[  223.259033] tps65217_irq: USB power status change

[  225.097045] tps65217_irq: USB power status change

[  225.255859] tps65217_irq: USB power status change

[  227.094024] tps65217_irq: USB power status change

[  227.262908] tps65217_irq: USB power status change

[  229.095489] tps65217_irq: USB power status change

I paste part of the log above. The actual timing of USBI status change is at a inteval of 0.16, 1.84 seconds interval. I lowered the CPU frequency from 1G to 300MHz. I don't observe any change in this timing.

I also loaded an Angstrom (3.8 kernel) on the BBB. By inspecting the interrupts (cat /proc/interrupts), it seems that 3.8 kernel does NOT have the same behavior. I will be checking the PMIC configuration difference between 3.2 and 3.8 kernel.

Also, if USB is connected to the mini USB connector. This behavior is going away.

Thanks.

[Maxim Podbereznyy]

By the way when I connected 5vdc directly to the additional ldo ic I have grounded Vusb as well and the board worked without any reboot issue during 1 week. Probably I was in a wrong direction thinking that it was totally depending on the current overload at tps65217. One of the ideas was about the USB grounding and I combined all solutions at once

Will test only grounded Vusb to check your hypothesis

03 дек. 2013 г. 21:32 пользователь "Lei Wang" <lei...@gmail.com> написал:

[dek...@gmail.com]

On Tuesday, December 3, 2013 12:32:03 PM UTC-5, Lei Wang wrote:

Follow your finding on the unexpected TPS65217C behavior, I patched the tps65217 driver with irq handling. The 3.2 kernel does not handle nNMI/PMIC_INT interrupt; the 3.8 kernel does. I placed printk in the interrupt handler and got the same result as yours. The PMIC_INT was issued every 2 seconds which is caused by the USBI flag in the TPS65217C interrupt register. 

 

 

This appears to be related to USB OTG. The VBUS, ID, and D+ pins on USB0 are all generating 0.5 Hz signals. The signal on VBUS that the TPS65217C is detecting may be OTG probing.

 

This may not be the cause of the random reboots, but it's worth some examination.

[Maxim Podbereznyy]

The abstract from the TPS65217 datasheet to describe what is going on here:

The linear charger periodically applies a 10-mA current source to the BAT pin to check for the presence of a

battery. This will cause the BAT terminal to float up to > 3 V which may interfere with AC removal detection and

the ability to switch from AC to USB input. For this reason, it is not recommended to use both AC and USB

inputs when the battery is absent.

2013/12/4 <dek...@gmail.com>

--

For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[dek...@gmail.com]

On Wednesday, December 4, 2013 1:45:02 PM UTC-5, lisarden wrote:

The abstract from the TPS65217 datasheet to describe what is going on here:

The linear charger periodically applies a 10-mA current source to the BAT pin to check for the presence of a

battery. This will cause the BAT terminal to float up to > 3 V which may interfere with AC removal detection and

the ability to switch from AC to USB input. For this reason, it is not recommended to use both AC and USB

inputs when the battery is absent.

I wonder when the BAT terminal drifts > 3 V, if the PMIC behaves as if V_BAT > V_UVLO.  

 

If so, I wonder what happens if AM335x USB-OTG probing drives VBUS > V_BAT + 190 mV. That would exceed V_IN(DT).

 

[Lei Wang]

I can confirm that the pulsing detected by PMIC on USB_DC signal is the probing from USB-OTG. 

After I disabled the USB-OTG in the kernel, the system has never rebooted. Btw I also re-loaded Angstrom image (3.8 kernel) and Andrew's Android image (with 3.8 kernel). I did not observe USB-OTG probing pulses on the VBus. I believe in the 3.8 kernel, the USB-OTG has not been implemented/enabled. That might be reason why it seems that 3.8 kernel doesn't have the random reboot behavior.

[Lei Wang]

In case anyone wants to test it out, here is the change in the source code (NOTE: ignore the line and column numbers; just search for the struct "static struct omap_musb_board_data musb_board_data" ):

...

--- a/arch/arm/mach-omap2/board-am335xevm.c

+++ b/arch/arm/mach-omap2/board-am335xevm.c

...

@@ -3956,7 +4125,8 @@ static struct omap_musb_board_data musb_board_data = {

  * mode[4:7] = USB1PORT's mode

  * AM335X beta EVM has USB0 in OTG mode and USB1 in host mode.

  */

- .mode           = (MUSB_HOST << 4) | MUSB_OTG,

+// .mode           = (MUSB_HOST << 4) | MUSB_OTG,

+ .mode           = (MUSB_HOST << 4) | MUSB_PERIPHERAL,

  .power = 500,

  .instances = 1,

 };

Please let me know the results.

Thank.

[Illutian Kade]

Crap...now to figure out if this is doable (by me) for Debian Wheezy.

...if computers become sentient, it's probably because I goofed something up.

[Illutian Kade]

Sigh, nope, Angstrom 3.8 (disabled OTG detection) didn't fix the issue either.

As of late, the BBB actually powers down completely (power LED is off). And pressing the power button, reset button, even BOOT button does nothing. Hell, even unplugging and plugging back in the power doesn't do anything. If I wait several minutes I can get the thing to power on......for about 50seconds then it goes dead again.

Appears that it is, as others suggested, a physical issue.....crap.

[Gerald Coley]

Please request an RMA so we can look at it. Make sure it is in the failed state and that you let the RMA team know how to get it in that state and recover.

Gerald

--

[Thomas J]

Hi

any news on this issue ? I have the same problem with my beagle xm, with linux not running anything and it still reboots after 1 to 5 minutes

If there is anything electric to do to make this work (add a capacity in front of the 5V power supply, ...) I can do it, I have lots of electronic tools at work

Btw, I am logged in through tty and I see

Broadcast message from root@arm
        (unknown) at 11:40 ...

The system is going down for reboot NOW!
[  143.036193] Restarting system.

so somehow it might not be a complete physical problem, the system knows about it

(Linux version 3.7.10-x9 on Ubuntu 12.10 and beagleboard xm) 

[Illutian Kade]

Turns out it was a faulty PMIC. It's been fixed and is happily blinking away :D

[Jukka Mykkänen]

Hey,

I have to BBB:s (A6) that boots randomly with Ubuntu installed. How the PMIC was fixed, did you do it or send it back to shop?`

-Jukka

[Illutian Kade]

I RMA-ed it using the Beagle Board website. Took about 20 days, round trip, to get the board back. I doubt even if I had the skills to replace circuit chips, I'd would have been able to find it....I have no idea how they diagnose issues for their boards.

But you may have another issue. As mine eventually stopped booting back up after the randomly rebooting for several days. Might be harder for them to find the issue. As they told me to leave the board in a "failed state" and ship it to them. But if your's is rebooting, then it will never be in a 'failed state'.

[Thomas Jannaud]

The problem was fixed on my side with the new kernel

https://github.com/RobertCNelson/armv7-multiplatform

[Damien]

Hi Robert,
Do you know more detail about the time jump issue in uboot? for example, fix commit number, or some words used for the commit?
I am interested to find out what exactly the fixes are, but there are too many commits in uboot and I need some thing to search with.

Regards,
Damien

On Wednesday, November 13, 2013 1:45:37 AM UTC+11, RobertCNelson wrote:

On Tue, Nov 12, 2013 at 8:37 AM,  <lei...@gmail.com> wrote:

> I have similar issue. I have (2) versions of BBB (A5A and A5C). They both
> randomly reboot themselves while I am running TI prebuilt BBB android image
> (from a couple hours to ten to fifteen hours). When I plug in the USB (DC is
> still powered) for logging with logcat, the reboot issue seems to disappear.
>
> I don't have problem with BBB Angstrom image (based on 3.8 kernel). I don't
> have problem with Andrew Henderson's android image (based on 3.8 kernel)
> either.
>
> Another issue is that when I run TI BBB android image, the clock randomly
> jumps forward 2^17 seconds. This happens on both of my BBB boards. The
> problem goes away when I run Angstrom or Andrew's android.

This time 'issue' was fixed in u-boot sometime last year, so I'm
guessing the TI image has an un-patched u-boot..

>

> I suspect it has something to do with the processor, DDR3 (BBB: AM3359 1GHz
> + 512MB DDR3), the configuration, or apply workaround of errata. We have an
> AM335x EVM kit (AM3359 720MHz + 256MB DDR2). I also loaded TI prebuilt
> android image. I have run it for several months. It is rock solid. I never
> had problem with it.
>
> Here are the links to my other posts in regarding to this issue.
> https://groups.google.com/forum/#!category-topic/beagleboard/advanced/5qSJ4dQdar4
> http://e2e.ti.com/support/embedded/android/f/509/t/297726.aspx
>

[Robert Nelson]

Probably.. http://git.denx.de/?p=u-boot.git;a=commit;h=000820b5835c2b8b863af992b66dc973dc4bd202

--

[Damien]

Strange ... I checked the commit and found it had been included within the v2013.04 uboot release .. but my BB board is still experienced with this time jump issue one/two times per day. Could there be anything else?
Regards.
Damien

[Lei Wang]

Damien,

Check the link on TI e2e: 

http://e2e.ti.com/support/embedded/android/f/509/t/308616.aspx

It basically switched the clock source from 32K to 24MHz clock. It eventually fixed my time jump problem.

Good luck!

Lei

You received this message because you are subscribed to a topic in the Google Groups "BeagleBoard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beagleboard/xPxzYyNsA78/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beagleboard...@googlegroups.com.

[Lei Wang]

Damien,

Check the link on TI e2e: 

http://e2e.ti.com/support/embedded/android/f/509/t/308616.aspx

It basically switched the clock source from 32K to 24MHz clock. It eventually fixed my time jump problem.

Good luck!

[Damien]

Thanks Lei.
this fix works! I ran the BB for 8 days without problem.

[Thomas O]

Hey i do have some updates regarding this! We have now ran 10 bbb boards with a strap between 5v vcc and the v+ input on the USB connector. These now all have an uprime of 10+ days.

The controls that are powered from the same power supply but do not have the strap has an average uptime of 8-9 hours.

This makes me think that this has to do with the driver initialisation of the PMIC.

I am hoping that this information will help someone to continue the research in the restart/ reboot problem.

 

[terry.b...@googlemail.com]

I think I have found the reason for this. See the "3.17.1-rc4 sudden reset" thread.