Stability Issues

[Nigel Horne]

I have found the stability of the C120 to be poor.  It keeps crashing with only the single LED on.  Last night I left it on doing nothing (except for what ever cron wanted to do) and this morning I couldn't SSH in to it so I had to power-cycle.  What soak testing as been done.  I hate to say this, but my Pi has never crashed.

-Nigel

[Nigel Horne]

If it helps I have this output in the error log:

Feb 12 08:20:57 ci20-2753 kernel: [ 5406.120357] BUG: sleeping function called from invalid context at mm/slub.c:847
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.128036] in_atomic(): 1, irqs_disabled(): 0, pid: 2483, name: memcached

From the kern.log:

Feb 12 06:52:04 ci20-2753 kernel: [   72.690701] BUG: sleeping function called from invalid context at mm/slub.c:847
Feb 12 06:52:04 ci20-2753 kernel: [   72.698298] in_atomic(): 1, irqs_disabled(): 0, pid: 2646, name: run-parts
Feb 12 06:52:04 ci20-2753 kernel: [   72.705435] Call Trace:
Feb 12 06:52:04 ci20-2753 kernel: [   72.707963] [<8066458c>] dump_stack+0x8/0x34
Feb 12 06:52:04 ci20-2753 kernel: [   72.712363] [<800eb578>] kmem_cache_alloc+0x150/0x25c
Feb 12 06:52:04 ci20-2753 kernel: [   72.717581] [<80348a50>] create_stat+0x30/0x13c
Feb 12 06:52:04 ci20-2753 kernel: [   72.722247] [<80348c30>] uid_stat_tcp_rcv+0x60/0x80
Feb 12 06:52:04 ci20-2753 kernel: [   72.727288] [<805299e0>] tcp_read_sock+0x1d0/0x294
Feb 12 06:52:04 ci20-2753 kernel: [   72.732226] [<805ec0c8>] xs_tcp_data_ready+0x84/0xcc
Feb 12 06:52:04 ci20-2753 kernel: [   72.737352] [<80533558>] tcp_rcv_established+0xc4/0x72c
Feb 12 06:52:04 ci20-2753 kernel: [   72.742734] [<8053b0e4>] tcp_v4_do_rcv+0x138/0x360
Feb 12 06:52:04 ci20-2753 kernel: [   72.747677] [<8053e134>] tcp_v4_rcv+0x884/0xa38
Feb 12 06:52:04 ci20-2753 kernel: [   72.752343] [<8051842c>] ip_local_deliver_finish+0xfc/0x310
Feb 12 06:52:04 ci20-2753 kernel: [   72.758101] [<804e0828>] __netif_receive_skb+0x380/0x514
Feb 12 06:52:04 ci20-2753 kernel: [   72.763584] [<804e3310>] process_backlog+0xfc/0x274
Feb 12 06:52:04 ci20-2753 kernel: [   72.768619] [<804e3fa8>] net_rx_action+0x130/0x338
Feb 12 06:52:04 ci20-2753 kernel: [   72.773557] [<80059254>] __do_softirq+0x118/0x2d0
Feb 12 06:52:04 ci20-2753 kernel: [   72.778411] [<8005978c>] do_softirq+0xa4/0xac
Feb 12 06:52:04 ci20-2753 kernel: [   72.782896] [<80059a8c>] irq_exit+0xb4/0xbc
Feb 12 06:52:04 ci20-2753 kernel: [   72.787215] [<80017e08>] plat_irq_dispatch+0x108/0x210
Feb 12 06:52:04 ci20-2753 kernel: [   72.792509] [<8001cb60>] ret_from_irq+0x0/0x4
Feb 12 06:52:04 ci20-2753 kernel: [   72.797005] [<8037a3d4>] jz4780_nand_read_buf+0xf4/0x278
Feb 12 06:52:04 ci20-2753 kernel: [   72.802480] [<8036fdb4>] nand_read_page_hwecc+0xc4/0x1b0
Feb 12 06:52:04 ci20-2753 kernel: [   72.807964] [<803732cc>] nand_do_read_ops+0x1b0/0x4c4
Feb 12 06:52:04 ci20-2753 kernel: [   72.813166] [<803739e8>] nand_read+0x54/0x80
Feb 12 06:52:04 ci20-2753 kernel: [   72.817573] [<8036a5c0>] part_read+0x60/0xb0
Feb 12 06:52:04 ci20-2753 kernel: [   72.821971] [<80367800>] mtd_read+0x80/0xd4
Feb 12 06:52:04 ci20-2753 kernel: [   72.826290] [<803851e4>] ubi_io_read+0x94/0x228
Feb 12 06:52:04 ci20-2753 kernel: [   72.830957] [<80382e90>] ubi_eba_read_leb+0x140/0x348
Feb 12 06:52:04 ci20-2753 kernel: [   72.836168] [<80381e18>] ubi_leb_read+0xd4/0x180
Feb 12 06:52:04 ci20-2753 kernel: [   72.840931] [<802197a8>] fallible_read_node+0x64/0x188
Feb 12 06:52:04 ci20-2753 kernel: [   72.846234] [<8021b750>] ubifs_tnc_locate+0x11c/0x254
Feb 12 06:52:04 ci20-2753 kernel: [   72.851437] [<802144cc>] ubifs_iget+0x7c/0x628
Feb 12 06:52:04 ci20-2753 kernel: [   72.856022] [<802111e4>] ubifs_lookup+0x10c/0x1c4
Feb 12 06:52:04 ci20-2753 kernel: [   72.860871] [<80103480>] d_alloc_and_lookup+0x4c/0x90
Feb 12 06:52:04 ci20-2753 kernel: [   72.866084] [<8010535c>] do_lookup+0x2ac/0x338
Feb 12 06:52:04 ci20-2753 kernel: [   72.870661] [<80106040>] path_lookupat+0x158/0x7d8
Feb 12 06:52:04 ci20-2753 kernel: [   72.875603] [<801066f0>] do_path_lookup+0x30/0xa0
Feb 12 06:52:04 ci20-2753 kernel: [   72.880448] [<80106844>] user_path_at+0x5c/0xa4
Feb 12 06:52:04 ci20-2753 kernel: [   72.885112] [<800fca9c>] vfs_fstatat+0x44/0x84
Feb 12 06:52:04 ci20-2753 kernel: [   72.889698] [<800fd04c>] sys_newstat+0x28/0x4c
Feb 12 06:52:04 ci20-2753 kernel: [   72.894278] [<80026d48>] stack_done+0x20/0x44
Feb 12 06:52:04 ci20-2753 kernel: [   72.898771]
Feb 12 06:55:52 ci20-2753 kernel: [  300.645420] JZ4780-EFUSE: vddq_set 0
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.120357] BUG: sleeping function called from invalid context at mm/slub.c:847
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.128036] in_atomic(): 1, irqs_disabled(): 0, pid: 2483, name: memcached
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.135170] Call Trace:
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.137774] [<8066458c>] dump_stack+0x8/0x34
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.142213] [<800eb578>] kmem_cache_alloc+0x150/0x25c
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.147512] [<80348a50>] create_stat+0x30/0x13c
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.152201] [<80348c30>] uid_stat_tcp_rcv+0x60/0x80
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.157255] [<805299e0>] tcp_read_sock+0x1d0/0x294
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.162195] [<805ec0c8>] xs_tcp_data_ready+0x84/0xcc
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.167323] [<80533980>] tcp_rcv_established+0x4ec/0x72c
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.172797] [<8053b0e4>] tcp_v4_do_rcv+0x138/0x360
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.177742] [<8053e134>] tcp_v4_rcv+0x884/0xa38
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.182407] [<8051842c>] ip_local_deliver_finish+0xfc/0x310
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.188166] [<804e0828>] __netif_receive_skb+0x380/0x514
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.193636] [<804e3310>] process_backlog+0xfc/0x274
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.198668] [<804e3fa8>] net_rx_action+0x130/0x338
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.203606] [<80059254>] __do_softirq+0x118/0x2d0
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.208461] [<8005978c>] do_softirq+0xa4/0xac
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.212947] [<80059a8c>] irq_exit+0xb4/0xbc
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.217270] [<80017e08>] plat_irq_dispatch+0x108/0x210
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.222565] [<8001cb60>] ret_from_irq+0x0/0x4
Feb 12 08:20:57 ci20-2753 kernel: [ 5406.227100]

Regards,

-Nigel

[Nigel Horne]

I'm going to disable memcached.  I doubt it's the cause, and it's like fixing a symptom not the problem, but I want to see if that helps.

N

[Nigel Horne]

Stopping memcached didn't help.  It's died again.  Uptime 3 hours this time :-(

-Nigel

[Zubair Kakakhel]

Hi Nigel,

Welcome to the forum.

Looks like you received a bad board.
We've shipped several hundred boards and found a few people have stability issues. (memtest failing). 4 people so far.

Don't worry. Please check this for RMA details.

http://www.elinux.org/CI20_Troubleshooting#Definitely_an_RMA_after_community_troubleshooting

I'll check the bad boards to try and see what has gone wrong.

Regards,
ZubairLK
________________________________________
From: mips-cre...@googlegroups.com [mips-cre...@googlegroups.com] on behalf of Nigel Horne [nigel...@gmail.com]
Sent: 12 February 2015 15:00
To: mips-cre...@googlegroups.com
Subject: Re: Stability Issues

Stopping memcached didn't help. It's died again. Uptime 3 hours this time :-(

-Nigel

--
You received this message because you are subscribed to the Google Groups "MIPS Creator CI20" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mips-creator-c...@googlegroups.com<mailto:mips-creator-c...@googlegroups.com>.
To post to this group, send email to mips-cre...@googlegroups.com<mailto:mips-cre...@googlegroups.com>.
To view this discussion on the web, visit https://groups.google.com/d/msgid/mips-creator-ci20/aebc760e-f7b0-45a5-92ea-c9d05f4b9f48%40googlegroups.com<https://groups.google.com/d/msgid/mips-creator-ci20/aebc760e-f7b0-45a5-92ea-c9d05f4b9f48%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

[Nigel Horne]

Thanks for your reply.  Here are the RMA answers.

What revision is the board (e.g Revision A green/ Revision B purple) CI20_Hardware#Board_Revisions_and_changes

Purple

 What power supply were you using? (e.g. Smartphone. Laptop usb. Adapter)

Mains via adapter

 What is the ambient room temperature? (Rough estimate)

About 21C.

 What were you doing when it died?

Sleeping one of the times ;-)  Other times it's been compiling.

 Have you tried re-flashing the NAND? Which image/Os?

No

 Have you tried booting from SD card?

No.

-Nigel

[Zubair Kakakhel]

Hi Nigel,

:)

I meant emailing Sales....@imgtec.com for RMA linking them to this post.

https://groups.google.com/forum/#!topic/mips-creator-ci20/B3F6bufovys

Regards,
ZubairLK
________________________________________
From: mips-cre...@googlegroups.com [mips-cre...@googlegroups.com] on behalf of Nigel Horne [nigel...@gmail.com]

Sent: 12 February 2015 20:03

To: mips-cre...@googlegroups.com
Subject: Re: Stability Issues

Thanks for your reply. Here are the RMA answers.

What revision is the board (e.g Revision A green/ Revision B purple) CI20_Hardware#Board_Revisions_and_changes<http://www.elinux.org/CI20_Hardware#Board_Revisions_and_changes>

Purple

What power supply were you using? (e.g. Smartphone. Laptop usb. Adapter)

Mains via adapter

What is the ambient room temperature? (Rough estimate)

About 21C.

What were you doing when it died?

Sleeping one of the times ;-) Other times it's been compiling.

Have you tried re-flashing the NAND? Which image/Os?

No

Have you tried booting from SD card?

No.

-Nigel

--
You received this message because you are subscribed to the Google Groups "MIPS Creator CI20" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mips-creator-c...@googlegroups.com<mailto:mips-creator-c...@googlegroups.com>.
To post to this group, send email to mips-cre...@googlegroups.com<mailto:mips-cre...@googlegroups.com>.

To view this discussion on the web, visit https://groups.google.com/d/msgid/mips-creator-ci20/5f2a3be3-0db7-4008-a733-285d5de11555%40googlegroups.com<https://groups.google.com/d/msgid/mips-creator-ci20/5f2a3be3-0db7-4008-a733-285d5de11555%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Michael Bass]

Hi Nigel,

If you think your IP address is stable under DHCP, you might think again.  I've had my CI20 re-register itself with DHCP, essentially obtaining a new IP address.  It was .79 then a few hours later it was .80.  The CI20 didn't reboot.  I don't know what triggers this.  It has gone over 12 hours without an IP address change.  Once it happened while I was logged in.  (Naturally, that ssh session wasn't responsive.)  It's a strange observation.

Mike

[Nigel Horne]

Hi Michael,

I've assigned it an IP address in the 192.168.x.x range and disabled Network Manager, so I would be annoyed if the IP address were to change.  But it's an interesting scenario.  I'll double check that dhclient isn't running and that nothing else is changing the address.  I could easily stash the output of ifconfig to a file every 5 minutes and check if it changes.

Thanks for the idea.

-Nigel

[Nigel Horne]

It just died again.  Similar place in the kernel, here is the stack trace.  This doesn't "feel" like a hardware problem to me, this "feel"s like a software and/or firmware problem.  Same kernel BUG message followed by a crash?  So I doubt that's a memory problem.

Feb 12 19:17:42 ci20-2753 kernel: [  510.884058] BUG: sleeping function called from invalid context at mm/slub.c:847
Feb 12 19:17:42 ci20-2753 kernel: [  510.891716] in_atomic(): 1, irqs_disabled(): 0, pid: 3259, name: mandb
Feb 12 19:17:42 ci20-2753 kernel: [  510.898494] Call Trace:
Feb 12 19:17:42 ci20-2753 kernel: [  510.901036] [<8066458c>] dump_stack+0x8/0x34
Feb 12 19:17:42 ci20-2753 kernel: [  510.905453] [<800eb578>] kmem_cache_alloc+0x150/0x25c
Feb 12 19:17:42 ci20-2753 kernel: [  510.910696] [<80348a50>] create_stat+0x30/0x13c
Feb 12 19:17:43 ci20-2753 kernel: [  510.915379] [<80348c30>] uid_stat_tcp_rcv+0x60/0x80
Feb 12 19:17:43 ci20-2753 kernel: [  510.920461] [<805299e0>] tcp_read_sock+0x1d0/0x294
Feb 12 19:17:43 ci20-2753 kernel: [  510.925432] [<805ec0c8>] xs_tcp_data_ready+0x84/0xcc
Feb 12 19:17:43 ci20-2753 kernel: [  510.930616] [<80533980>] tcp_rcv_established+0x4ec/0x72c
Feb 12 19:17:43 ci20-2753 kernel: [  510.936136] [<8053b0e4>] tcp_v4_do_rcv+0x138/0x360
Feb 12 19:17:43 ci20-2753 kernel: [  510.941094] [<8053e134>] tcp_v4_rcv+0x884/0xa38
Feb 12 19:17:43 ci20-2753 kernel: [  510.945806] [<8051842c>] ip_local_deliver_finish+0xfc/0x310
Feb 12 19:17:43 ci20-2753 kernel: [  510.951576] [<804e0828>] __netif_receive_skb+0x380/0x514
Feb 12 19:17:43 ci20-2753 kernel: [  510.957093] [<804e3310>] process_backlog+0xfc/0x274
Feb 12 19:17:43 ci20-2753 kernel: [  510.962142] [<804e3fa8>] net_rx_action+0x130/0x338
Feb 12 19:17:43 ci20-2753 kernel: [  510.967177] [<80059254>] __do_softirq+0x118/0x2d0
Feb 12 19:17:43 ci20-2753 kernel: [  510.972044] [<8005978c>] do_softirq+0xa4/0xac
Feb 12 19:17:43 ci20-2753 kernel: [  510.976554] [<80059a8c>] irq_exit+0xb4/0xbc
Feb 12 19:17:43 ci20-2753 kernel: [  510.980873] [<80017e08>] plat_irq_dispatch+0x108/0x210
Feb 12 19:17:43 ci20-2753 kernel: [  510.986270] [<8001cb60>] ret_from_irq+0x0/0x4
Feb 12 19:17:43 ci20-2753 kernel: [  510.990814] [<8037a3d4>] jz4780_nand_read_buf+0xf4/0x278
Feb 12 19:17:43 ci20-2753 kernel: [  510.996348] [<8036fd7c>] nand_read_page_hwecc+0x8c/0x1b0
Feb 12 19:17:43 ci20-2753 kernel: [  511.001853] [<803732cc>] nand_do_read_ops+0x1b0/0x4c4
Feb 12 19:17:43 ci20-2753 kernel: [  511.001869] [<803739e8>] nand_read+0x54/0x80
Feb 12 19:17:43 ci20-2753 kernel: [  511.001882] [<8036a5c0>] part_read+0x60/0xb0
Feb 12 19:17:43 ci20-2753 kernel: [  511.001893] [<80367800>] mtd_read+0x80/0xd4
Feb 12 19:17:43 ci20-2753 kernel: [  511.001906] [<803851e4>] ubi_io_read+0x94/0x228
Feb 12 19:17:43 ci20-2753 kernel: [  511.001917] [<80382e90>] ubi_eba_read_leb+0x140/0x348
Feb 12 19:17:43 ci20-2753 kernel: [  511.001930] [<80381e18>] ubi_leb_read+0xd4/0x180
Feb 12 19:17:43 ci20-2753 kernel: [  511.001950] [<802197a8>] fallible_read_node+0x64/0x188
Feb 12 19:17:43 ci20-2753 kernel: [  511.001963] [<8021b750>] ubifs_tnc_locate+0x11c/0x254
Feb 12 19:17:43 ci20-2753 kernel: [  511.001974] [<802144cc>] ubifs_iget+0x7c/0x628
Feb 12 19:17:43 ci20-2753 kernel: [  511.001985] [<802111e4>] ubifs_lookup+0x10c/0x1c4
Feb 12 19:17:43 ci20-2753 kernel: [  511.002001] [<80103480>] d_alloc_and_lookup+0x4c/0x90
Feb 12 19:17:43 ci20-2753 kernel: [  511.002014] [<8010535c>] do_lookup+0x2ac/0x338
Feb 12 19:17:43 ci20-2753 kernel: [  511.002026] [<80106040>] path_lookupat+0x158/0x7d8
Feb 12 19:17:43 ci20-2753 kernel: [  511.002037] [<801066f0>] do_path_lookup+0x30/0xa0
Feb 12 19:17:43 ci20-2753 kernel: [  511.002049] [<80106844>] user_path_at+0x5c/0xa4
Feb 12 19:17:43 ci20-2753 kernel: [  511.002060] [<800fca9c>] vfs_fstatat+0x44/0x84
Feb 12 19:17:43 ci20-2753 kernel: [  511.002071] [<800fd1e0>] sys_stat64+0x28/0x4c
Feb 12 19:17:43 ci20-2753 kernel: [  511.002088] [<80026d48>] stack_done+0x20/0x44
Feb 12 19:17:43 ci20-2753 kernel: [  511.002097]

-Nigel

[Nigel Horne]

I have updated to Debian 8.  There were a few errors, but nothing to mention.  I will see if that helps, though it's worth noting that you don't get an updated kernel you still have an old 3.0.8 revision, whereas Debian 8 on my Wintel PC is at 3.16.

-Nigel

[Nigel Horne]

I have an RMA so I will return ASAP.  It may take a few days so over the weekend I lent the board to a friend for extra diagnostics which may help.  He noted these alerts in the log when running a compilation which may help:

[ 3761.668824] do_page_fault() #2: sending SIGSEGV to conftest for invalid write access to
[ 3761.668835] 7f76fff8 (epc == 0040091c, ra == 00400940)
[ 3763.384110] do_page_fault() #2: sending SIGSEGV to conftest for invalid write access to
[ 3763.384121] 7fb3bfe8 (epc == 0040093c, ra == 00400968)
[17084.656177] nand_do_read_ops: return -EBADMSG!
[17084.660776] error -77 (ECC error) while reading 78 bytes from PEB 1812:2066288, read only 78 bytes, retry

He also volunteered how slow the CI20 is, even slower than a Pi B, so I know it's not just my opinion.  He also spotted the issue I mentioned on another thread that even when doing nothing the load average is still 2.  This points to a kernel tuning issue.

The box is packed and ready to go - all I need to do is find somewhere that's open when I'm around and which sells clear shipping tape.  Hence the delay in the return...

-Nigel

[Nigel Horne]

I've finally had the chance to organise a return of my C120 by UPS.  I'll keep you posted.  Whilst it is very disappointing it didn't "just work", I am gladdened that the company is helpful and doing its best to rectify.

Regards,

-Nigel

[fumeof...@gmail.com]

I've sent the board for RMA and I know that it's arrived since the UPS tracking says so.  I've heard nothing since then.  What happens next?

-Nigel

[Zubair Lutfullah Kakakhel]

Hi,

On 11/03/15 13:41, fumeof...@gmail.com wrote:
> I've sent the board for RMA and I know that it's arrived since the UPS tracking says so. I've heard nothing since then. What happens next?

I assume when it is processed they will send one out.

Probably worth pinging them on sales.support after a few days.

Thanks
ZubairLK

>
> -Nigel
>
> --
> You received this message because you are subscribed to the Google Groups "MIPS Creator CI20" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mips-creator-c...@googlegroups.com <mailto:mips-creator-c...@googlegroups.com>.
> To post to this group, send email to mips-cre...@googlegroups.com <mailto:mips-cre...@googlegroups.com>.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/mips-creator-ci20/6c8a1704-68d7-45e4-86a8-d4fcbf44707a%40googlegroups.com <https://groups.google.com/d/msgid/mips-creator-ci20/6c8a1704-68d7-45e4-86a8-d4fcbf44707a%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Nigel Horne]

I have done so.  Is it standard for a customer to need to drive it or have I fallen through the cracks?

-Nigel