Panda instance non-responsive

19 views
Skip to first unread message

ethermeme

unread,
Apr 30, 2010, 12:19:54 PM4/30/10
to Panda
Hi Everyone,

I've been running a panda instance (ami-05d7336c) for a client over
the last 6 months. It seemed to be locked up and non-responsive this
morning, so I issued a reboot from the ec2 console, and the instance
rebooted, but it never did respond again. I wound up setting up a new
instance, but I'd like to know if anyone else has run into this
before?

I've attached the last console log I could get from the instance. I
truncated the top of it, it was all fsck's repair process.

Thank You,
Troy

-------

/dev/sda2: |========================================================-
99.6%
/dev/sda2: |========================================================|
100.0%

/dev/sda2: 12/44564480 files (8.3% non-contiguous), 1448314/89121408
blocks
fsck died with exit status 1
done.
Setting kernel variables...done.
Mounting local filesystems...kjournald starting. Commit interval 5
seconds

EXT3 FS on sda2, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

mount: none already mounted or /sys busy
mount: according to mtab, sysfs is already mounted on /sys

[31mfailed.
[39;49m
Activating swapfile swap...done.
Setting up networking....
Configuring network interfaces...Internet Systems Consortium DHCP
Client V3.0.4
Copyright 2004-2006 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth0/12:31:39:01:6c:f3
Sending on LPF/eth0/12:31:39:01:6c:f3
Sending on Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 5
DHCPOFFER from 169.254.1.0
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPACK from 169.254.1.0
bound to 10.255.115.1 -- renewal in 36793 seconds.
done.

INIT: Entering runlevel: 4

Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting nginx: nginx.

[33m*
[39;49m Not starting internet superserver: no services enabled.
Starting Postfix Mail Transport Agent: postfix.
Starting OpenBSD Secure Shell server: sshd.
Starting periodic command scheduler: crond.
Starting daemon monitor: monit.
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed

100 392 100 392 0 0 6653 0 --:--:-- --:--:--
--:--:-- 6653
100 392 100 392 0 0 6626 0 --:--:-- --:--:--
--:--:-- 0

Debian GNU/Linux 4.0 ec2-67-202-24-49.compute-1.amazonaws.com tty1

ec2-67-202-24-49.compute-1.amazonaws.com login:
INIT: Switching to runlevel: 6

INIT: Sending processes the TERM signal

Stopping daemon monitor: monit.
Stopping periodic command scheduler: crond.
Stopping nginx: Stopping internet superserver: inetd.
Stopping Postfix Mail Transport Agent: postfix.
Stopping rsync daemon: rsync.
Stopping OpenBSD Secure Shell server: sshd.
Saving the system clock..
Stopping kernel log daemon: klogd.
Stopping system log daemon: syslogd.
Asking all remaining processes to terminate...done.
Killing all remaining processes...done.
Deconfiguring network interfaces...There is already a pid file /var/
run/dhclient.eth0.pid with pid 858
removed stale PID file
Internet Systems Consortium DHCP Client V3.0.4
Copyright 2004-2006 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth0/12:31:39:01:6c:f3
Sending on LPF/eth0/12:31:39:01:6c:f3
Sending on Socket/fallback
DHCPRELEASE on eth0 to 169.254.1.0 port 67
done.
Cleaning up ifupdown....
Deactivating swap...done.
Unmounting local filesystems...done.
Will now restart.
md: stopping all md devices.

md: md0 switched to read-only mode.

Restarting system.

.

Linux version 2.6.16-xenU (bui...@xenbat.amazonsa) (gcc version 4.0.1
20050727 (Red Hat 4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007

BIOS-provided physical RAM map:

Xen: 0000000000000000 - 000000006cc00000 (usable)

1020MB HIGHMEM available.

727MB LOWMEM available.

NX (Execute Disable) protection: active

IRQ lockup detection disabled

Built 1 zonelists

Kernel command line: root=/dev/sda1 ro 4

Enabling fast FPU save and restore... done.

Enabling unmasked SIMD FPU exception support... done.

Initializing CPU#0

PID hash table entries: 4096 (order: 12, 65536 bytes)

Xen reported: 2327.498 MHz processor.

Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)

Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)

Software IO TLB disabled

vmalloc area: ee000000-f53fe000, maxmem 2d7fe000

Memory: 1759360k/1789952k available (1958k kernel code, 21308k
reserved, 620k data, 144k init, 1044488k highmem)

Checking if this processor honours the WP bit even in supervisor
mode... Ok.

Calibrating delay using timer specific routine.. 4659.57 BogoMIPS
(lpj=23297884)

Mount-cache hash table entries: 512

CPU: L1 I cache: 32K, L1 D cache: 32K

CPU: L2 cache: 4096K

CPU: L3 cache: 4096K

Checking 'hlt' instruction... OK.

Brought up 1 CPUs

migration_cost=0

Grant table initialized

NET: Registered protocol family 16

Enabling SMP...

Initializing CPU#1

migration_cost=15754

Brought up 2 CPUs

xen_mem: Initialising balloon driver.

highmem bounce pool size: 64 pages

VFS: Disk quotas dquot_6.5.1

Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)

Initializing Cryptographic API

io scheduler noop registered

io scheduler anticipatory registered (default)

io scheduler deadline registered

io scheduler cfq registered

i8042.c: No controller found.

RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize

Xen virtual console successfully installed as tty1

Event-channel device installed.

netfront: Initialising virtual ethernet driver.

Registering block device major 8

mice: PS/2 mouse device common for all mice

md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27

md: bitmap version 4.39

NET: Registered protocol family 2

IP route cache hash table entries: 65536 (order: 6, 262144 bytes)

TCP established hash table entries: 262144 (order: 9, 2097152 bytes)

TCP bind hash table entries: 65536 (order: 7, 524288 bytes)

TCP: Hash tables configured (established 262144 bind 65536)

TCP reno registered

TCP bic registered

NET: Registered protocol family 1

NET: Registered protocol family 17

NET: Registered protocol family 15

Using IPI No-Shortcut mode

md: Autodetecting RAID arrays.

md: autorun ...

md: ... autorun DONE.

kjournald starting. Commit interval 5 seconds

EXT3-fs: mounted filesystem with ordered data mode.

VFS: Mounted root (ext3 filesystem) readonly.

Freeing unused kernel memory: 144k freed


INIT: version 2.86 booting


[33m*
[39;49m Mount point '/lib/init/rw' does not exist. Skipping mount.
touch: cannot touch `/lib/init/rw/.ramfs': No such file or directory
Activating swap...Adding 917496k swap on /dev/sda3. Priority:-1
extents:1 across:917496k

done.
Checking root file system...fsck 1.40-WIP (14-Nov-2006)
/dev/sda1: clean, 80987/1313280 files, 365718/2621440 blocks
done.
EXT3 FS on sda1, internal journal

Setting the system clock..
Cleaning up ifupdown....
Loading kernel modules...done.
Loading device-mapper supportdm_mod: no version for "struct_module"
found: kernel tainted.

device-mapper: 4.5.0-ioctl (2005-10-04) initialised: dm-
de...@redhat.com

.
Checking file systems...fsck 1.40-WIP (14-Nov-2006)
/dev/sda2: clean, 12/44564480 files, 1448314/89121408 blocks
done.
Setting kernel variables...done.
Mounting local filesystems...kjournald starting. Commit interval 5
seconds

EXT3 FS on sda2, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

mount: none already mounted or /sys busy
mount: according to mtab, sysfs is already mounted on /sys

[31mfailed.
[39;49m
Activating swapfile swap...done.
Setting up networking....
Configuring network interfaces...Internet Systems Consortium DHCP
Client V3.0.4
Copyright 2004-2006 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth0/12:31:39:01:6c:f3
Sending on LPF/eth0/12:31:39:01:6c:f3
Sending on Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3
DHCPOFFER from 169.254.1.0
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPACK from 169.254.1.0
bound to 10.255.115.1 -- renewal in 41604 seconds.
done.

INIT: Entering runlevel: 4

Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting nginx: nginx.

[33m*
[39;49m Not starting internet superserver: no services enabled.
Starting Postfix Mail Transport Agent: postfix.
Starting OpenBSD Secure Shell server: sshd.
Starting periodic command scheduler: crond.
Starting daemon monitor: monit.
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed

100 392 100 392 0 0 49676 0 --:--:-- --:--:--
--:--:-- 49676
100 392 100 392 0 0 48353 0 --:--:-- --:--:--
--:--:-- 0

Debian GNU/Linux 4.0 ec2-67-202-24-49.compute-1.amazonaws.com tty1

ec2-67-202-24-49.compute-1.amazonaws.com login: fuse init (API version
7.8)

fuse distribution version: 2.7.3

Linux version 2.6.16-xenU (bui...@xenbat.amazonsa) (gcc version 4.0.1
20050727 (Red Hat 4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007

BIOS-provided physical RAM map:

Xen: 0000000000000000 - 000000006cc00000 (usable)

1020MB HIGHMEM available.

727MB LOWMEM available.

NX (Execute Disable) protection: active

IRQ lockup detection disabled

Built 1 zonelists

Kernel command line: root=/dev/sda1 ro 4

Enabling fast FPU save and restore... done.

Enabling unmasked SIMD FPU exception support... done.

Initializing CPU#0

PID hash table entries: 4096 (order: 12, 65536 bytes)

Xen reported: 2327.500 MHz processor.

Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)

Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)

Software IO TLB disabled

vmalloc area: ee000000-f53fe000, maxmem 2d7fe000

Memory: 1759360k/1789952k available (1958k kernel code, 21308k
reserved, 620k data, 144k init, 1044488k highmem)

Checking if this processor honours the WP bit even in supervisor
mode... Ok.

Calibrating delay using timer specific routine.. 4657.09 BogoMIPS
(lpj=23285493)

Mount-cache hash table entries: 512

CPU: L1 I cache: 32K, L1 D cache: 32K

CPU: L2 cache: 4096K

CPU: L3 cache: 4096K

Checking 'hlt' instruction... OK.

Brought up 1 CPUs

migration_cost=0

Grant table initialized

NET: Registered protocol family 16

Enabling SMP...

Initializing CPU#1

migration_cost=15383

Brought up 2 CPUs

xen_mem: Initialising balloon driver.

highmem bounce pool size: 64 pages

VFS: Disk quotas dquot_6.5.1

Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)

Initializing Cryptographic API

io scheduler noop registered

io scheduler anticipatory registered (default)

io scheduler deadline registered

io scheduler cfq registered

i8042.c: No controller found.

RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize

Xen virtual console successfully installed as tty1

Event-channel device installed.

netfront: Initialising virtual ethernet driver.

Registering block device major 8

mice: PS/2 mouse device common for all mice

md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27

md: bitmap version 4.39

NET: Registered protocol family 2

IP route cache hash table entries: 65536 (order: 6, 262144 bytes)

TCP established hash table entries: 262144 (order: 9, 2097152 bytes)

TCP bind hash table entries: 65536 (order: 7, 524288 bytes)

TCP: Hash tables configured (established 262144 bind 65536)

TCP reno registered

TCP bic registered

NET: Registered protocol family 1

NET: Registered protocol family 17

NET: Registered protocol family 15

Using IPI No-Shortcut mode

md: Autodetecting RAID arrays.

md: autorun ...

md: ... autorun DONE.

EXT3-fs: INFO: recovery required on readonly filesystem.

EXT3-fs: write access will be enabled during recovery.

kjournald starting. Commit interval 5 seconds

EXT3-fs: sda1: orphan cleanup on readonly fs

EXT3-fs: sda1: 1 orphan inode deleted

EXT3-fs: recovery complete.

EXT3-fs: mounted filesystem with ordered data mode.

VFS: Mounted root (ext3 filesystem) readonly.

Freeing unused kernel memory: 144k freed


INIT: version 2.86 booting


[33m*
[39;49m Mount point '/lib/init/rw' does not exist. Skipping mount.
touch: cannot touch `/lib/init/rw/.ramfs': No such file or directory
Activating swap...Adding 917496k swap on /dev/sda3. Priority:-1
extents:1 across:917496k

done.
Checking root file system...fsck 1.40-WIP (14-Nov-2006)
/dev/sda1: clean, 82352/1313280 files, 698892/2621440 blocks
done.
EXT3 FS on sda1, internal journal

Setting the system clock..
Cleaning up ifupdown....
Loading kernel modules...done.
Loading device-mapper supportdm_mod: no version for "struct_module"
found: kernel tainted.

device-mapper: 4.5.0-ioctl (2005-10-04) initialised: dm-
de...@redhat.com

.
Checking file systems...fsck 1.40-WIP (14-Nov-2006)
/dev/sda2: recovering journal
/dev/sda2: clean, 13/44564480 files, 1448315/89121408 blocks
done.
Setting kernel variables...done.
Mounting local filesystems...kjournald starting. Commit interval 5
seconds

EXT3 FS on sda2, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

mount: none already mounted or /sys busy
mount: according to mtab, sysfs is already mounted on /sys

[31mfailed.
[39;49m
Activating swapfile swap...done.
Setting up networking....
Configuring network interfaces...Internet Systems Consortium DHCP
Client V3.0.4
Copyright 2004-2006 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth0/12:31:39:01:6c:f3
Sending on LPF/eth0/12:31:39:01:6c:f3
Sending on Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4
DHCPOFFER from 169.254.1.0
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPACK from 169.254.1.0
bound to 10.255.115.1 -- renewal in 34844 seconds.
done.

INIT: Entering runlevel: 4

Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting nginx: nginx.

[33m*
[39;49m Not starting internet superserver: no services enabled.
Starting Postfix Mail Transport Agent: postfix.
Starting OpenBSD Secure Shell server: sshd.
Starting periodic command scheduler: crond.
Starting daemon monitor: monit.
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0
100 392 100 392 0 0 292 0 0:00:01 0:00:01
--:--:-- 1146

Debian GNU/Linux 4.0 ec2-67-202-24-49.compute-1.amazonaws.com tty1

ec2-67-202-24-49.compute-1.amazonaws.com login:

--
You received this message because you are subscribed to the Google Groups "Panda" group.
To post to this group, send email to panda...@googlegroups.com.
To unsubscribe from this group, send email to pandastream...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pandastream?hl=en.

Damien Tanner

unread,
Apr 30, 2010, 12:30:31 PM4/30/10
to panda...@googlegroups.com
Hi,

Does look like something out of the ordinary.

Thanks for the info though.

Damien Tanner Damien Tanner - Co-Founder and Director, Panda
+44 (0)20 7099 7486 - pandastream.com - @pandastream
Reply all
Reply to author
Forward
0 new messages