Board is hanging frequently

[Tim]

I just bought a Hawkboard from Special Computing and it hangs all the
time. I'm working off the eLinux guides:

http://elinux.org/Hawkboard
http://elinux.org/Hawkboard/BeginnersGuide

I have a serial console connected via a null modem cable and I'm
booting uBoot (default sw1: on off off off). I'm using tftp (server
tested from other systems and it's working reliably - no file
corruption or connection timeouts) to transfer images (the ones
mentioned in the guides -- checksums ok) and booting either the
provided Angstrom ramdisk image or the Ubuntu image from an SD card.

This is a rev A board. Here's the reported uBoot version - I have not
tried updating it yet:
U-Boot 2009.01-dirty (Nov 26 2009 - 02:15:00)

The board hangs:
in uBoot while transferring files via ftp
in uBoot while reading flash memory with nand read.e
in uBoot during the write portion of mtest
during Linux boot but at inconsistent points / stages
during network activity from Linux
after hotplugging any usb device in Linux
when trying to login via a usb keyboard (cold plug) while using a vga
monitor (hangs after a few key presses)
and once when I was logged into Linux but with no activity

I know I'm an early adopter but it seems it shouldn't be this
unstable. My guess is I either got a lemon, I need to update uBoot, or
the power supply is faulty. Does anyone have some insight into which
it might be or what I should try next?

I'm using an external power supply from Special Computing that I
bought with the board. It's a DVE 5v 2.5A switching power adapter.
They changed the plug but otherwise I don't have any reason to believe
it's bad. It provides 5v under load but I don't the equipment to do a
more thorough test. I did try running the board once from usb power
and it hung on nand read.e. Does anyone know if it should be a linear
power source instead? The manual doesn't specify and I haven't been
able to find one that sources enough amps. I'm working on getting an
alternate power supply. Turns out 5v, >2A, 5.5/2.5mm plug is a hard
combination to find and I haven't decided if it's better to hack one
together like special comp did or if I'm going to put a plug on
another type of supply (eg. pc power supply - though I understand
that's also a switching power supply).

I got it with their acrylic case so I haven't handled the board
directly. It's also relatively humid here and the board runs perfectly
when it does run so I don't think I zapped it.

Any suggestions? I can post example logs of times it froze if that
would help.

[J T Dsouza]

Hi all,
Same issues with me. Purchased a hawk board from IDA systems.
I have been struggling to get it to boot.
Most of the time it hangs while executing the kernel uImage at loading kernel. sometimes it will move a little ahead and throw up a whole lot of oops before hanging at udev.
On the rare occasion when it does boot the kernel, it will hang at mounting the nfs root.
I have tried usb (1.1, not the OTG) and mmc too with similiar results.
Out of the several hundred tries rootfs from USB has worked 2 or 3 times and Nfs 3 or 4 times. In both cases the board hung solid after a few minutes of doing nothing.

I am at my wits end to doing some systematic debugging and isolate these issues as software or hardware.
I am a bit scared to reflash a new uboot as i might end up with a brick. I am tempted to return the board.

Details:
Board version Rev A
uboot U-Boot 2009.01-dirty (Nov 26 2009 - 02:15:00)
images used
Linux-2.6.32-rc6-00079-g55996fd hangs on Starting kernel ..
Angstrom/2.6.32+2.6.33-rc4-r48+g hangs on Starting kernel .. but has booted occasionally but hangs at INIT: version 2.86  or udev
Maverick kernel locks up the board and one has to press reset at least twice to reboot.

In comparison my homebrew lpc2468 board worked like a charm.

[Nirmalya Lahiri]

Dsouza,  thanks to point it out... I also facing same problem but don't know how to solve...!!! --- নির্মাল্য লাহিড়ী [Nirmalya Lahiri] +৯১-৯৪৩৩১১৩৫৩৬ [+91-9433113536] --- On Fri, 10/15/10, J T Dsouza <jtd...@gmail.com> wrote:

[Caglar Akyuz]

On Friday 15 October 2010 09:29:30 pm J T Dsouza wrote:
> Hi all,

Hi,

> Same issues with me. Purchased a hawk board from IDA systems.
> I have been struggling to get it to boot.
> Most of the time it hangs while executing the kernel uImage at loading
> kernel. sometimes it will move a little ahead and throw up a whole lot of
> oops before hanging at udev.
> On the rare occasion when it does boot the kernel, it will hang at mounting
> the nfs root.
> I have tried usb (1.1, not the OTG) and mmc too with similiar results.
> Out of the several hundred tries rootfs from USB has worked 2 or 3 times
> and Nfs 3 or 4 times. In both cases the board hung solid after a few
> minutes of doing nothing.
>

Have you checked your Khasim's latest posts? It sounds like your problem is
related to it.

> I am at my wits end to doing some systematic debugging and isolate these
> issues as software or hardware.
> I am a bit scared to reflash a new uboot as i might end up with a brick. I
> am tempted to return the board.
>
> Details:
> Board version Rev A
> uboot U-Boot 2009.01-dirty (Nov 26 2009 - 02:15:00)
> images used
> Linux-2.6.32-rc6-00079-g55996fd hangs on Starting kernel ..
> Angstrom/2.6.32+2.6.33-rc4-r48+g hangs on Starting kernel .. but has booted
> occasionally but hangs at INIT: version 2.86 or udev
> Maverick kernel locks up the board and one has to press reset at least
> twice to reboot.
>
> In comparison my homebrew lpc2468 board worked like a charm.
>

Simplicity of lpc2468 makes it a wrong choice for a comparison to L-138, IMHO.

Regards,
Caglar

[Tim]

Hi Caglar,

Thanks for the heads up. The Angrstrom ramdisk does periodically hang
at "starting kernel" so I tried the core druid uImage with mixed
results. It still hangs sometimes at "starting kernel" but I haven't
been able to get it running either. It starts dumping this error
sequence shortly after reaching the login prompt:

...
Modules linked in:
Internal error: Oops - undefined instruction: 0 [#412] PREEMPT
last sysfs file: /sys/kernel/uevent_seqnum
Modules linked in:
Internal error: Oops - undefined instruction: 0 [#413] PREEMPT
last sysfs file: /sys/kernel/uevent_seqnum
Modules linked in:
Internal error: Oops - undefined instruction: 0 [#414] PREEMPT
last sysfs file: /sys/kernel/uevent_seqnum
Modules linked in:
Internal error: Oops - undefined instruction: 0 [#415] PREEMPT
last sysfs file: /sys/kernel/uevent_seqnum
...


Anyway, does your board(s) ever hang from uBoot (tftp xfer or nand
operations)? I'm pretty sure those stability issues are unrelated to
Linux. Do you think it could be a uBoot issue instead of hardware?

Thanks,
-Tim

[Tim]

Hey JT,

Does your board only freeze during Linux boot or also from uBoot?

-T

[JTD]

On Oct 17, 6:05 am, Tim <tim6...@gmail.com> wrote:
> Hey JT,
>
> Does your board only freeze during Linux boot or also from uBoot?
>

Rarely one has to press reset twice. Uboot fails to come up with a
single reset.
This has happened maybe 4 or 5 times totally, almost always when it
partially boots a kernel. So it's difficult to pinpoint as a uboot
problem. Also ever so often reset button fails and uboot continues to
work happily. Suspecting a faulty reset switch (flux creep etc.), i
checked the reset switch and it works perfectly. So it could be the
reset circuitry. But that can wait another day. Right now it's to
flash uboot from linux. I dont have doze at all.

[Caglar Akyuz]

On Sunday 17 October 2010 04:02:55 am Tim wrote:
> Hi Caglar,
>

Hi,

> Thanks for the heads up. The Angrstrom ramdisk does periodically hang
> at "starting kernel" so I tried the core druid uImage with mixed
> results. It still hangs sometimes at "starting kernel" but I haven't
> been able to get it running either. It starts dumping this error
> sequence shortly after reaching the login prompt:
>
> ...
> Modules linked in:
> Internal error: Oops - undefined instruction: 0 [#412] PREEMPT
> last sysfs file: /sys/kernel/uevent_seqnum
> Modules linked in:
> Internal error: Oops - undefined instruction: 0 [#413] PREEMPT
> last sysfs file: /sys/kernel/uevent_seqnum
> Modules linked in:
> Internal error: Oops - undefined instruction: 0 [#414] PREEMPT
> last sysfs file: /sys/kernel/uevent_seqnum
> Modules linked in:
> Internal error: Oops - undefined instruction: 0 [#415] PREEMPT
> last sysfs file: /sys/kernel/uevent_seqnum
> ...
>
>
> Anyway, does your board(s) ever hang from uBoot (tftp xfer or nand
> operations)? I'm pretty sure those stability issues are unrelated to
> Linux. Do you think it could be a uBoot issue instead of hardware?
>

Yes. Problem can also be seen at u-boot level, hanging tftp transfers mostly.
This is DDR2 timing issue. As per Khasim's mail, he is verifying new u-boot
and I guess he will upload new one soon. In the mean time, I will send it to
you off-list.

Regards,
Caglar

[J T Dsouza]

Ok reflashed uboot using linux only

connect with minicom
setenv serverip xx.yy.zz.aa
setenv ipaddr xx.yy.zz.ab

tftp  u-boot_uart_ais_v2.bin
nand erase
nand write.e 0xc0700000 0x20000 0x70000
reboot

U-Boot 2009.01-dirty (Nov 26 2009 - 02:15:00)

the board now boots a kernel consistently but fails mostly at udev. 1 in 10 failures at loading kernel.

[J T Dsouza]

Also grounding y3 xtal improves booting to near 100%. Also linux moves ahead of udev several times. Also the L138 is quite warm, but mountin a small cooling fan on top does not improve anything.

Looks like we have multiple issues. Ram timing, oscillator noise, ...

[Tim]

Thanks Caglar,

Looks promising. I haven't had a tftp hang with the new uBoot. I'm
still having some stability issues but it looks to be Linux specific
at this point. I'll post back if I see more uBoot problems but that
may have done the trick.

I took some live readings with a multimeter: it's pulling about 470mA
with ethernet connected and 410mA without. Didn't try with usb or sd.
Voltage is 5.8-5.11V dipping a bit after lockups but never below 5V.
Between that and the new uBoot fixing the tftp trouble I'm thinking
power issues haven't been a factor for me.

[JTD]

After plenty of tests. I can confirm that grounding y6,y3 and y4
crystal cans improves things substantially.
Full boot success is around 35%.

However the board hangs within a minute of doing nothing with no login

Is it noise or are the clocks changing?

Next I checked the temperature on the cpu. It hovers around 45 C. So I
tacked on a make shift heatsink. Now on time seems to have improved to
250 secs with ping running.. Full boot all the way to a console also
seems to have improved by 5%.

On Oct 18, 12:48 am, J T Dsouza <jtd1...@gmail.com> wrote:

[J T Dsouza]

Note: Attaching a heatsink is non trivial, as there are higher profile SMT caps (afaik) and u could short them with the heatsink body.

[J T Dsouza]

The reset tact switch has gone bad. I have to now use a resistor to short the switch pins for a reset.

[Caglar Akyuz]

On Monday 25 October 2010 01:21:15 pm JTD wrote:
> After plenty of tests. I can confirm that grounding y6,y3 and y4
> crystal cans improves things substantially.
> Full boot success is around 35%.
>
> However the board hangs within a minute of doing nothing with no login
>
> Is it noise or are the clocks changing?
>
> Next I checked the temperature on the cpu. It hovers around 45 C. So I
> tacked on a make shift heatsink. Now on time seems to have improved to
> 250 secs with ping running.. Full boot all the way to a console also
> seems to have improved by 5%.
>

Just to make sure that I get it correct, are you saying that grounding and
cooling improves things w.r.t. board locking ups?

Thanks,
Caglar

[JTD]

On Oct 25, 4:40 pm, Caglar Akyuz <caglarak...@gmail.com> wrote:
> On Monday 25 October 2010 01:21:15 pm JTD wrote:
>
> > After plenty of tests. I can confirm that grounding y6,y3 and y4
> > crystal cans improves things substantially.
> > Full boot success is around 35%.
>
> > However the board hangs within a minute of doing nothing with no login
>
> > Is it noise or are the clocks changing?
>
> > Next I checked the temperature on the cpu. It hovers around 45 C. So I
> > tacked on a make shift heatsink. Now on time seems to have improved to
> > 250 secs with ping running..  Full boot all the way to a console also
> > seems to have improved by 5%.
>
> Just to make sure that I get it correct, are you saying that grounding and
> cooling improves things w.r.t. board locking ups?
>

Indeed. Yes. Especially the xtal grounding. Last ping session 54
minutes, Locked up after i switched to the vga console (with minicom
running a ping). Hung after 40 secs.

IMO rule of thumb any component which you cant touch for more than 30
secs is too hot.

Uptime improved to > 1 hour, by repositioning the heat sink.

Right now the xtals are grounded with 8 inches of crocodile clip and
wire. To solder or not to solder that is the question - dont want to
lose my ability to return the board, as it does not seem to be merely
an EMC programming problem..

[Berni]

I have just got an hawkboard from IDA and i have some serious lockup
problems with it like a lot of others.

Doing ping many times in uboot will crash it,sometimes saveenv
command,sometimes nand write...
When i try to boot any of the example linux images i get stuck at
"Starting kernel..."
Also every time the board locks up something happens with the VGA
output, the image aether jumps
to a different location or gets artifacts.

The board has the seemingly suspicious DDR chip and i have tried 3
different uboots and they all worked the same.

What could the problem be here?

[Tim]

I haven't tried grounding the xtals, but I'm not having any thermal
problems. My board hangs without any chips even really warming up.

Have you tried to contact innovate solutions directly?