GK802 server uptime

[Žydrūnas Urbonas]

Hi,

Anyone using GK802 as server and got it running for a long period?

I experience random restarts and freezes once a week when running motion (USB webcam capturing 720p and encoding to x264). No errors in syslog or dmesg prior the issue. And I wonder whether it could be hardware or kernel issue?

Kernel is from 4.1.0 branch

[Angus Gratton]

On Sun, Jan 26, 2014 at 08:46:49AM -0800, Žydrūnas Urbonas wrote:
> Anyone using GK802 as server and got it running for a long period?

I just moved house and took down my GK802, but it had about 180 days
uptime at that point. I've been using it as a personal server (VPS
replacement) since May (running email, some private web server uses,
files, etc.) and haven't had any instability issues.

Kernel is the one I built as part of the Debian installer I threw together,
3.0.35-48d9718-2-gk802. Older than the 4.1.0 branch I think.

The only USB device I have connected is a USB/Ethernet adapter for
network access. I'm not using wifi or bluetooth at all. I'm using both
SD card slots (I have a RAID1 array of MicroSD cards!)

> I experience random restarts and freezes once a week when running motion
> (USB webcam capturing 720p and encoding to x264). No errors in syslog or
> dmesg prior the issue. And I wonder whether it could be hardware or kernel
> issue?

Probably the best thing you can do is connect an external serial
console.

Pinout shown here:
https://github.com/imx6-dongle/wiki/wiki/GK802-hardware

Then you'll hopefully see some kernel panic output that helps you
track it down. If you're very keen you can turn on kdb and dig around
with the debugger some more after the crash.

Please let us know how you get on.

Good luck,


Angus

[Žydrūnas Urbonas]

Hi,

is there something that I need to enable or configure to get verbose output to console?

Currently I only get this short text:

U-Boot 2013.01-rc1-gd0486e7-dirty (May 04 2013 - 15:31:51)

CPU:   Freescale i.MX6Q rev1.2 at 792 MHz
Reset cause: WDOG
Board: MX6Q-gk802
DRAM:  1 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Hit any key to stop autoboot:  0
mmc0 is current device
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
mmc1 is current device
369 bytes read
## Executing script at 10008000
mmc1 is current device
3522732 bytes read
## Booting kernel from Legacy Image at 10800000 ...
   Image Name:   Linux-3.0.35
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3522668 Bytes = 3.4 MiB
   Load Address: 10008000
   Entry Point:  10008000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

Starting kernel ...

[Žydrūnas Urbonas]

Ok, looks like I'm on the right way.
At the UBoot prompt I issued a command:

setenv console 'ttymxc3,115200'

And now I'm not only see kernel output, but also get auto login into shell

Now will try to make it crash :)

[Žydrūnas Urbonas]

The device has just hung.
And nothing was written to console. Syslog has also no crash logged

Cacti load graph just before hang:

1.0-1.5 is quite normal for GK802 while capturing 720p stream from USB camera. But it never spikes over 2.0. This time there was some time while device had load more than 2.0

I wonder waht else option do I have to investigate this and try to debug the issue?

Thanks

[Žydrūnas Urbonas]

Tried disabling wireless (rmmod 8192cu) on stratup and use USB ethernet adapter instead.
Device has just unexpectedly rebooted after 1 day uptime without writing anything to serial console or syslog

Ok, will try now to get rid of USB hub and connect to camera MJPEG stream over wifi to see if it's USB host related

Will keep you posted

[Angus Gratton]

Hi Žydrūnas,

I'll be interested to hear how you go.

I'd also keep an eye on temperatures (can monitor via
/sys/class/thermal/thermal_zone0/temp) and your 5V power supply
quality as other possible causes of sudden lockups.

Good luck,

Angus

On Sun, Feb 02, 2014 at 12:50:34AM -0800, Žydrūnas Urbonas wrote:
> Tried disabling wireless (rmmod 8192cu) on stratup and use USB ethernet
> adapter instead.
> Device has just unexpectedly rebooted after 1 day uptime without writing
> anything to serial console or syslog
>
> Ok, will try now to get rid of USB hub and connect to camera MJPEG stream
> over wifi to see if it's USB host related
>
> Will keep you posted
>
> On Friday, January 31, 2014 1:06:05 PM UTC+2, Žydrūnas Urbonas wrote:
> >
> > The device has just hung.
> > And nothing was written to console. Syslog has also no crash logged
> >
> > Cacti load graph just before hang:
> >
> >

> > <https://lh5.googleusercontent.com/-j5aDddkM9_w/UuuCzKgRvsI/AAAAAAAACmU/uCS5g7-v5so/s1600/gk802_hung.png>

[Žydrūnas Urbonas]

I already prevented possible overheat by attaching a heatsink to the top surface (GK802 has metal housing which is very good) :)

But thanks for thermal sensor point. I will set up cacti to monitor this

[Žydrūnas Urbonas]

Device was running without any problems for almost two weeks. It was using USB ethernet adaptor instead of wireless and was capturing stream over the network. During the test temperature was 40-50 degrees.

Now I have connected USB web camera back and started stream capture over USB - GK802 hung in about few hours.
I could not notice what temperature was at the moment of hang as I use SNMP monitoring for this, so probably I need to set up some different monitoring (dump values into a file every 10 seconds).

Will perform some more tests, but so far I made no progress on identifying what causes those hangs/restarts when capturing from USB camera

[Žydrūnas Urbonas]

Well, not sure what I have changed, but so far my GK802 is running for 10 days where it should hang in 1-2 days

Only those messages in dmesg:

ehci_fsl_bus_suspend begins, DR
ehci_fsl_bus_suspend ends, DR
ehci_fsl_bus_resume begins, DR
ehci_fsl_bus_resume ends, DR
Invalid temperature, force it to 25C
ehci_fsl_bus_suspend begins, DR
ehci_fsl_bus_suspend ends, DR
ehci_fsl_bus_resume begins, DR
ehci_fsl_bus_resume ends, DR
ehci_fsl_bus_suspend begins, DR
ehci_fsl_bus_suspend ends, DR
ehci_fsl_bus_resume begins, DR
ehci_fsl_bus_resume ends, DR
INFO: rcu_bh_state detected stall on CPU 1 (t=0 jiffies)
Invalid temperature, force it to 25C
Invalid temperature, force it to 25C
Invalid temperature, force it to 25C
Invalid temperature, force it to 25C
INFO: rcu_bh_state detected stall on CPU 1 (t=0 jiffies)
Invalid temperature, force it to 25C
Invalid temperature, force it to 25C
INFO: rcu_bh_state detected stall on CPU 1 (t=0 jiffies)
INFO: rcu_bh_state detected stall on CPU 1 (t=0 jiffies)
Invalid temperature, force it to 25C
Invalid temperature, force it to 25C
INFO: rcu_bh_state detected stall on CPU 1 (t=0 jiffies)


Still monitoring