BBB: after powerfailure BBB is no longer booting from eMMC

[Martin Sperl]

Hi!

A few weeks ago I had debian installed on my BeagleBoneBlack, booting from emmc, which worked fine.

Now a few days ago I had to move the board to a different location - I just disconnected power.
Since then the system is not booting from eMMC flash.

I have tried to reflash the system - Angstrom and Debian.

But both are not working after writing the eMMC (debian does not even start)

And I have even connected to the serial console to see that the emmc.sh script has finished its work and nothing mounted any more...

The current situation is that I can only boot a complete Angstrom system from SD-card with the stock Angstrom image.

From the Angstrom SD image I have partitioned the emmc the same way as the SD card (same sizes,...)
and then copied the boot partition from SD to EMMC via dd and checked checksums for the partition.

Here what I got:

root@beaglebone:~# fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 3963 MB, 3963617280 bytes
4 heads, 16 sectors/track, 120960 cylinders, total 7741440 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1   *          63      144584       72261    c  W95 FAT32 (LBA)
/dev/mmcblk0p2          147456     7127039     3489792   83  Linux
root@beaglebone:~# fdisk -l /dev/mmcblk1

Disk /dev/mmcblk1: 1920 MB, 1920991232 bytes
4 heads, 16 sectors/track, 58624 cylinders, total 3751936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x2e79218b

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk1p1   *          63      144584       72261    c  W95 FAT32 (LBA)
root@beaglebone:~# md5sum /dev/mmcblk?p1
cb808e0e8fd5be6611847a377029fe9d  /dev/mmcblk0p1
cb808e0e8fd5be6611847a377029fe9d  /dev/mmcblk1p1

Then I did shut down the system, remove the SD card and power it up again.

The problem is that I see no activity on the serial console with regards to u-boot booting from emmc. I would expect the same messages as I see when when pressing boot while powering on to boot directly from SD card. Not even the repeated "C" shows up once a second, when the sd card is not "propperly" formatted with a boot partition...

I know that there is no second partition for booting the OS, but u-boot should start independently of if there is a linux partition to boot or not - for all it matters: I should be able to boot the linux from the SD card using the uboot from the emmc.

What am I missing?

Thanks,
Martin

P.s: as an additional question: how can a power outage trigger such a situation in the first place? This does not give me a lot of confidence putting it into a robot

[David Marquart]

I had an issue like this recently, my uEnv.txt file kept getting corrupted for some reason.  Put the SD in your desktop and check it out.

[Martin Sperl]

Nothing - everything is fine!

Actually (as posted) the partitions have the SAME checksum (the one of the SD that boots, and the one on the eMMC which does not) - that said: being a "dd" copy they should be identical...

So it seems to me as if there is an issue with some sort of pre-boot loader loading from eMMC/SD, which does not access the eMMC for some reason, but hangs instead...

One observation that I see is that when I abort u-boot and switch to mmc 1 - it first complains about a timeout and then it seems to be able to access it...

Martin




On Tuesday, January 14, 2014 6:51:47 PM UTC+1, rh_ wrote:

On Mon, 13 Jan 2014 05:11:08 -0800 (PST)
Martin Sperl
<warriorsofth...@gmail.com>

wrote:

> Hi!
>
> A few weeks ago I had debian installed on my BeagleBoneBlack, booting
> from emmc, which worked fine.
>
> Now a few days ago I had to move the board to a different location -
> I just disconnected power.
> Since then the system is not booting from eMMC flash.

W95 FAT32 ( LBA )

There were some discussions about filesystem issues on this list.
And I recall discussion of similar symptoms.

What's fsck.vfat say?

[Martin Sperl]

Hi!

OK - I have now taken the minimal image of Debian from: http://www.armhf.com/index.php/boards/beaglebone-black/

And I have copied it to the emmc as well as to an SD card.

Again the same observation: the image boots from SD, but nothing happens on the eMMC.

Checksums of the boot partitions are:

root@debian-armhf:~# md5sum /dev/mmcblk*p1
af7a970e36b360f6b5b65aab04701b65  /dev/mmcblk0p1
af7a970e36b360f6b5b65aab04701b65  /dev/mmcblk1p1

So I wonder what happens BEFORE UBOOT starts (or MLO for that matter).
Is it possible that the initial pre-boot loader for the board has gotten corrupted for the eMMC case?

Thanks,
           Martin

[Robert Nelson]

Grab a usb-serial adapter, you question can usually be answered in one boot..

Regards,

--
Robert Nelson
http://www.rcn-ee.com/

[Martin Sperl]

On Friday, January 17, 2014 6:47:22 PM UTC+1, RobertCNelson wrote:

Grab a usb-serial adapter, you question can usually be answered in one boot..

I have been running _everything_ off  the serial since the beginning of this thread.

the Serial shows nothing what so ever if I do not press the "boot" button for SD and power the device
no "C" every second or so or anything else...

But with the "boot" button pressed during power up I get:

CCCCCCCC
U-Boot SPL 2013.10-00249-g15c5cdf (Nov 17 2013 - 16:35:11)
reading args
spl: error reading image args, err - -1
reading u-boot.img
reading u-boot.img


U-Boot 2013.10-00249-g15c5cdf (Nov 17 2013 - 16:35:11)

I2C:   ready
DRAM:  512 MiB
WARNING: Caches not enabled
NAND:  0 MiB
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Attempt to read outside the flash area
*** Warning - readenv() failed, using default environment

Net:   <ethaddr> not set. Validating first E-fuse MAC
cpsw, usb_ether
Hit any key to stop autoboot:  0 Enter code here...

the first "CCCCC" line is actually one "C" after the other with a spacing of about 1 second.
then uboot takes over.

Without pressing boot I get nothing at all on the serial line...

That is what surprises me so much and why I fear that the pre-bootloader is not working as expected...

Martin

[Robert Nelson]

The "CC" show you incorrectly flashed the bootloader (MLO) onto the eMMC..

[Martin Sperl]

On Friday, January 17, 2014 6:59:44 PM UTC+1, RobertCNelson wrote:

The "CC" show you incorrectly flashed the bootloader (MLO) onto the eMMC..

maybe I miss something about how to flash the MLO - so far I have always assumed that the MLO sits in the first partition of the emmc and SD card which gets loaded and executed and which takes the next steps...

And - as shown via md5sum those partitions are identical:

root@debian-armhf:~# md5sum /dev/mmcblk*p1
af7a970e36b360f6b5b65aab04701b65  /dev/mmcblk0p1
af7a970e36b360f6b5b65aab04701b65  /dev/

mmcblk1p1
root@debian-armhf:~#

So what is missing?

Thanks, Martin

P.s: I filled the eMMC like this: 

xzcat /tmp/debian-wheezy-7.2-armhf-3.8.13-bone30.img.xz > /dev/mmcblk1

same as I did to put the image on the SD-card from a CentOS box (but to /dev/sdc to write to the SD card).

[Robert Nelson]

> maybe I miss something about how to flash the MLO - so far I have always
> assumed that the MLO sits in the first partition of the emmc and SD card
> which gets loaded and executed and which takes the next steps...

Well the "correct" way to just flash the MLO is on a fresh drive is:

http://eewiki.net/display/linuxonarm/BeagleBone+Black#BeagleBoneBlack-SetupmicroSD/SDcard

> And - as shown via md5sum those partitions are identical:
> root@debian-armhf:~# md5sum /dev/mmcblk*p1
> af7a970e36b360f6b5b65aab04701b65 /dev/mmcblk0p1
> af7a970e36b360f6b5b65aab04701b65 /dev/
> mmcblk1p1
> root@debian-armhf:~#
>
> So what is missing?
>
> Thanks, Martin
>
> P.s: I filled the eMMC like this:
> xzcat /tmp/debian-wheezy-7.2-armhf-3.8.13-bone30.img.xz > /dev/mmcblk1
> same as I did to put the image on the SD-card from a CentOS box (but to
> /dev/sdc to write to the SD card).

Your going to have to ask the "armhf.com" guy directly..

I know this "eMMC" flasher works, as i'm having users beta test it
right now as the "angstrom replacement factory image"

http://rcn-ee.net/deb/testing/2014-01-16/BBB-eMMC-flasher-debian-7.3-2014-01-16-2gb.img.xz

Just, flash a 2GB or greater microSD

sudo dd if=./BBB-eMMC-flasher-debian-7.3-2014-01-16-2gb of=/dev/sdX

Stick in BBB, power up.. wait about 14 minutes for all the LEDS to be
on.. Remove microSD and power off..

[Martin Sperl]

Here to show that the content inside the first partition are identical:

root@debian-armhf:~# mount /dev/mmcblk1p1  /mnt
root@debian-armhf:~# md5sum /mnt/*
edcfd600554d044f4fab684ab1afd105  /mnt/MLO
c8dcb2985802b00b5c6f1979e58836aa  /mnt/u-boot.img
2ae0ff2eae5713b88f6a1f30aaebd4d7  /mnt/uEnv.txt
root@debian-armhf:~# umount  /mnt
root@debian-armhf:~# mount /dev/mmcblk0p1  /mnt
root@debian-armhf:~# md5sum /mnt/*
edcfd600554d044f4fab684ab1afd105  /mnt/MLO
c8dcb2985802b00b5c6f1979e58836aa  /mnt/u-boot.img
2ae0ff2eae5713b88f6a1f30aaebd4d7  /mnt/uEnv.txt
root@debian-armhf:~# umount  /mnt

so everything is identical

[Martin Sperl]

Thing is that I have tried the default Angstrom and other images as well - even those that flash the eMMC from SD automatically via emmc.sh.
None have been successfull - that is why I have taken the image that is supposedly also able to get installed directly to eMMC.

I will now give the image of yours a try - Downloading right now...

Martin

[Robert Nelson]

On Fri, Jan 17, 2014 at 12:36 PM, Martin Sperl
<warriorsofth...@gmail.com> wrote:
> Thing is that I have tried the default Angstrom and other images as well -
> even those that flash the eMMC from SD automatically via emmc.sh.
> None have been successfull - that is why I have taken the image that is
> supposedly also able to get installed directly to eMMC.

Really "none" ?...

Umm, how are you flashing it?

ALL capes removed, NO USB, (ethernet can be plugged in), a real 5 volt
DC power (2amps)..

[Martin Sperl]

On Friday, January 17, 2014 7:38:47 PM UTC+1, RobertCNelson wrote:

Really "none" ?...

Umm, how are you flashing it?

ALL capes removed, NO USB, (ethernet can be plugged in), a real 5 volt
DC power (2amps)..

 
No Capes, No Ethernet, Only Serial Console and 5V 2A USB Charger.

Only Power and serial connected and I did try it with your image - I was able to verify on the console that the image of yours is working propperly and creating the files - partitioning, rsync,... all working - in the end all LED turned on and everything.

Then disconnecting from Power and removing SD card.

Then powering it on - nothing happens on the serial line at all... And that is the experience with all the images I have tried---

That is why I am saying there is something VERY strange with my board - at least since I did disconnect it without shutting it down...

So I fear that the pre-bootloader (the one loading MLO) is somehow "defective" and works only for the SD.

Martin

P.s: my guess is that if I clone the system created by your image on the eMMC to a 2GB SD, then it will boot correctly from SD - I can give it a try tomorrow...

[evilwulfie]

you cannot use  USB power while flashing the board. you MUST use a supply plugged into the barrel connector

--
For more options, visit http://beagleboard.org/discuss
---
You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Martin Sperl]

On Saturday, January 18, 2014 8:06:23 AM UTC+1, Wulf Man wrote:

you cannot use  USB power while flashing the board. you MUST use a supply plugged into the barrel connector

Well - the last time I had flashed it before it worked with USB power from an USB Power supply.

I have now done the flash now from a 5V 3.5A power supply connected to the barrel connector - _exactly_ the same results.

Here the interresting portion of a pstree while flashing:

  â”├â”─sh,1025 /opt/scripts/boot/am335x_evm.sh
  â”│   â”└â”─bash,1161 /opt/scripts/tools/beaglebone-black-eMMC-flasher.sh
  â”│       â”└â”─rsync,2501 -aAXv /bin /boot /dev /etc /home /lib /lost+found ...
  â”│           â”└â”─rsync,2502 -aAXv /bin /boot /dev /etc /home /lib ...
  â”│               â”└â”─rsync,2503 -aAXv /bin /boot /dev /etc /home /lib ...

and the df at the time showing the linux partition on the sMMC being mounted:

root@beaglebone:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
rootfs           1582864 1218260    282532  82% /
udev               10240       0     10240   0% /dev
tmpfs             101836     624    101212   1% /run
/dev/mmcblk0p2   1582864 1218260    282532  82% /
tmpfs             254584       0    254584   0% /dev/shm
tmpfs             254584       0    254584   0% /sys/fs/cgroup
tmpfs               5120       0      5120   0% /run/lock
tmpfs             102400       0    102400   0% /run/user
/dev/mmcblk0p1     98094   74086     24008  76% /boot/uboot
/dev/mmcblk1p2   1715936  516260   1110844  32% /tmp/rootfs

you see the new root-fs is mounted and is getting filled up...

and after the "flashing" of the eMMC the partitioning looks like this (prior to rebooting):

root@beaglebone:~# fdisk -l /dev/mmcblk1

Disk /dev/mmcblk1: 1920 MB, 1920991232 bytes
4 heads, 16 sectors/track, 58624 cylinders, total 3751936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

        Device Boot      Start         End      Blocks   Id  System

/dev/mmcblk1p1   *        2048      198655       98304    e  W95 FAT16 (LBA)
/dev/mmcblk1p2          198656     3751935     1776640   83  Linux

The same effect:

• shutting down system
  
• disconnecting power
  
• removing SD card
  
• powering up (from barrel connector)

• nothing happens on the serial line

• disconnecting power
• powering up (from barrel connector) pressing the "boot" button
  
• "C" shows up every second - to be expected as there is no SD card installed to boot from
  

The power supply is NOT the reason either for flash not working - so what is the problem?

Is there a means to check the pre-bootloader used to load and boot MLO to see if it is corrupted?

Martin
 

[Martin Sperl]

I reran the flashing, but this time before disconnecting power I ran:
dd if=/dev/mmcblk1 of=/dev/sda bs=1M
to create a 1:1 copy of the files installed to eMMC on a 2G SD card.

Again the boot rom eMMC fails, but if I add the SD card and boot from it - uboot and later the OS comes up all right...

So the disk-image created on the emmc is working from SD.

So there must be some _other_ difference that inhibits the boot from eMMC - IMO during the pre-MLO phase...

Any Ideas?

Martin

[Martin Sperl]

Any ideas how to solve this issue?

Pressing the Boot button every time I am booting the board is not very helpful in an embedded application...

Martin

[dennis.c...@gmail.com]

The symptoms you describe seem to suggest that the TI ROM bootstrap is somehow hanging while trying to load MLO from the eMMC.  If it managed to successfully run MLO you should see some console output, while if the boot from the eMMC failed cleanly (i.e. didn't work but didn't hang) it would move on to try to boot the SD card, followed by "CCC".  I have no solution but maybe you could make some progress by figuring out how far into booting the eMMC it gets before it becomes unhappy and hangs.

Boot your SD card image and start by using fdisk to make the eMMC FAT partition inactive in the MBR (i.e. get rid of the "*" under "Boot" in the fdisk output).  I don't know how to do this with the Linux fdisk (I run NetBSD on my boards) but it should be able to do that somehow.  Reboot with the Boot button pushed and make sure the FAT partition is still inactive.  Now try to reboot without pushing the Boot button.  What should happen is that the ROM bootstrap should read the MBR from the eMMC, find there is no active partition, and immediately move on to try to boot the SD card (ending up at "CCC" if the SD card boot fails).  If this succeeds then you know that the ROM bootstrap managed to read the eMMC partition table successfully, since it failed cleanly; if the bootstrap hangs you'll know that the very first access to the eMMC is making the bootstrap unhappy.

If this does succeed in booting from the SD card try making the eMMC FAT partition "active" again but delete MLO from the partition.  This should also end up booting the SD card.  If it hangs instead you'll know the problem the bootstrap is having is with reading the FAT directory, if it succeeds then the issue must happen when it tries to load and/or run MLO.

If either of the above succeeds you'll at least have a way to boot from the SD card without holding the Boot button.  If either of them fails with a hang then you'll know that simple accesses to the eMMC by the ROM bootstrap are doing something bad, a symptom which suggests a hardware-ish cause to me.  The fact that the eMMC looks good from Linux but looks bad from the ROM bootstrap suggests that the problem must be effected by the differences between the Linux and the bootstrap environment.  For example, Linux runs with a higher processor core voltage and frequency than the bootstrap does (not that I have any idea why this would matter), and there are likely other differences I don't know about.

In any case, if you get through this you may have enough data to make a convincing case that your board, and not your software, is screwed up.

Dennis Ferguson

[Martin Sperl]

Well - I have done all the above and even done a dd=/dev/zero of=/dev/mmcblk1 bs=1M to erase the whole emmc.

None of these steps help in booting only from SD automatically, so I shall assume this is a HW issue and create an RMA...

Thanks, Martin

[joshua.v...@gmail.com]

I am having a simlar issue with my custom board based off the BBB.  I think the issue is with using MMC1 the TRM for the am335x hints that the ROM uses Raw Mode therefore i believe it will jump to some sectors before your partion which starts at 2048... I was able to get MLO to start but i had the eMMC formatted incorrectly so MLO crashed.  I formatted /dev/mmcblk0 as a vfat and this curpoted the other data on p1 I think...

So hope this could be cleared up on how to exactly format a eMMC that boots from MMC1 on a AM335x.