[Discuss] I think my server is running out of "something"
David Kramer
Home-built server/firewall/mail server/web server/MythTV back end/makes
coffee.
Motherboard: Abit IP35 Pro with Intel Core2Duo E6750 @ 2.66GHz
eth0 --> cablemodem
eth1 --> intranet
6 sata ports on motherboard (JBOD, not RAID)
1 sata drive for root
4 sata drives for MythTV recordings
1 sata DVD-RW
I decided to add another 1TB drive for MythTV recordings, so I bought a
WD Caviar Black sata drive. I plug it into the 6th and last sata port
and power back up, and the server won't boot. Unplug the new drive, and
it boots.
I got an idea of trying to disconnect the DVD-RW drive and plug the new
drive back into sata port 6, and the server boots. Spooky. So once
again I unhook the new drive, plug in the DVD-RW drive again, and it boots.
Then I discovered that eth1 was dark. No signal, lights aren't lighting
up. So my server can get to the internet, but the rest of the house was
SOL. I reboot, and then eth1 works, but eth0 is dark, so I can get to
my server from my other computers, but nothing can get out to the
internet. I reboot AGAIN and finally both ethernet jacks are live. I
backed away slowly thinking clean thoughts. Again, this is several
reboots with no changes to hardware getting different results.
Current status is that the system is back up with all the original hard
drives working, and both ethernet ports working, but my brand spankin
new hard drive is staring at me longingly waiting to be deployed.
I have a theory that my server is running out of "something"
(interrupts? DMAs?), and the luck of the draw is determining what
devices get what they need. I can't think of another scenario where
devices would randomly work or not at boot, and adding a device disables
others. What do you think?
What can I look at?
What can I try?
Thanks in advance.
Side note: I stick labels on all my drives with the install date.
Apparently some of my MythTV drives have been spinning almost
continuously since 2007. That is impressive. And scary.
_______________________________________________
Discuss mailing list
Dis...@blu.org
http://lists.blu.org/mailman/listinfo/discuss
John Abreau
to mash everything together onto a single machine?
In my home network, I have a CentOS server that just does firewall/DNS/
DHCP, a second server for web/mail/fileserver, and a third server for backups.
That's in addition to my desktops, laptops, and tablet.
I'm not running MythTV, but if I were to deploy MythTV, I'd use a dedicated
server that would do nothing but MythTV.
--
John Abreau / Executive Director, Boston Linux & Unix
OLD GnuPG KeyID: D5C7B5D9 / Email: abr...@gmail.com
OLD GnuPG FP: 72 FB 39 4F 3C 3B D6 5B E0 C8 5A 6E F1 2C BE 99
2011 PGP KeyID: 32A492D8 / Email: abr...@gmail.com
2011 PGP FP: 7834 AEC2 EFA3 565C A4B6 9BA4 0ACB AD85 32A4 92D8
David Kramer
> Is it completely impossible to deploy multiple servers instead of trying
> to mash everything together onto a single machine?
>
> In my home network, I have a CentOS server that just does firewall/DNS/
> DHCP, a second server for web/mail/fileserver, and a third server for backups.
> That's in addition to my desktops, laptops, and tablet.
>
> I'm not running MythTV, but if I were to deploy MythTV, I'd use a dedicated
> server that would do nothing but MythTV.
Of course it's possible to do so. But up until a few hours ago, it was
working completely fine, rarely going below 85% idle, and burning about
140 watts according to my Killowat. That's pretty damn awesome. Two
servers would mean almost twice the electricity draw with no real
benefit. Oh, and the server has an IR tranceiver on it for changing
channels on my cable box.
The MythTV capture card (PVR-350) does hardware MPEG encryption, so
MythTV isn't really dragging things down as much as you would think
processor-wise, though I'm sure it pounds the sata bus, though. I only
have one card and it's SD, so it's not like I'm trying to write out
multiple HD files at the same time. All the MythTV recordings go on
drives not used for anything else.
Postfix sees a lot of mail (I usually get around 200 spam messages a
day, most of which is caught by SpamAssassin). The web server is
hosting about 6 domains, some of which go to one of the two wiki engines
I have on my server, but maybe it handles 100 hits a day.
So why do I need multiple servers if my CPU cores are mostly idle?
John Abreau
same machine, it eventually gets too complicated to manage properly,
and ultimately it starts exhibiting bizarre glitches that are nearly
impossible to understand in enough detail to fix. From what you
describe in your original message, it sounds like your server may be
reaching that level of complexity.
The Efika MX Smarttop is supposed to draw only 5W. With the addition
of a low power USB drive, it may be able to handle your web and postfix
services, and it should still use a lot less than 140W.
--
John Abreau / Executive Director, Boston Linux & Unix
OLD GnuPG KeyID: D5C7B5D9 / Email: abr...@gmail.com
OLD GnuPG FP: 72 FB 39 4F 3C 3B D6 5B E0 C8 5A 6E F1 2C BE 99
2011 PGP KeyID: 32A492D8 / Email: abr...@gmail.com
2011 PGP FP: 7834 AEC2 EFA3 565C A4B6 9BA4 0ACB AD85 32A4 92D8
ma...@mohawksoft.com
your power supply?
Jerry Feldman
you might be marginally pulling too much power.
Boot issues:
I had a similar issue. last summer when I went on a cruise I took my HDs
out, and when I put them back the system dis not boot. I manually
reordered them and it booted fine.
BTW: JABR has a full rack at home :-)
--
Jerry Feldman <g...@blu.org>
Boston Linux and Unix
PGP key id:3BC1EB90
PGP Key fingerprint: 49E2 C52A FC5A A31F 8D66 C0AF 7CEA 30FC 3BC1 EB90
David Kramer
> Boot issues:
> I had a similar issue. last summer when I went on a cruise I took my HDs
> out, and when I put them back the system dis not boot. I manually
> reordered them and it booted fine.
It's a Silverstone Olympia OP640 650W, which should be plenty.
It's possible that just the SATA rail is maxed out. Maybe I should try
putting the drive in an external sata case for power but connect the
sata cable from the motherboard to it as an experiment.
Would anyone be willing to lend me one? I'll buy one if I have to but I
wouldn't have any use for it past this test.
> BTW: JABR has a full rack at home :-)
Let's keep the conversation clean ;)
Tom Metro
> Home-built server/firewall/mail server/web server/MythTV back end/makes
> coffee.
Others covered this...I'll only add that a router appliance would
provide a fairly cheap "upgrade", hardly add any additional power usage,
and improve your security, while offloading some of your server workload.
> 1 sata drive for root
> 4 sata drives for MythTV recordings
> 1 sata DVD-RW
>
> I decided to add another 1TB drive...the server won't boot.
What exactly do you see when it fails to boot?
> It's a Silverstone Olympia OP640 650W, which should be plenty.
> It's possible that just the SATA rail is maxed out.
I concur with the others that this sounds like a power supply issue,
though 6 hard drives should not be excessive for a 650W supply. (I have
as many or more drives on my MythTV server with something like a 450W
supply.) But if the supply is optimized for running high powered
graphics cards it may in fact be running out of current on the 12V line
used by the drives.
> Maybe I should try putting the drive in an external sata case for
> power...
That wold be a good test and possibly a permanent work around.
For a quicker test that requires no additional equipment, try powering
up the system with the new drive attached to the SATA poet, but the
power connector detached, wait for the kernel boot messages to appear,
and then power up the drive. Hard drives use a peak amount of power
during startup, and this will stager the load among your drives.
> Then I discovered that eth1 was dark.
> I reboot AGAIN and finally both ethernet jacks are live.
As others speculated, probably related to the power problems. What did
you see from 'ifconfig'? Were the "dark" ports listed, but inactive, or
absent entirely? You may have some udev problems with the hardware being
recognized consistently. But if this only happens with the new drive
attached, then unlikely to be anything but power.
> Apparently some of my MythTV drives have been spinning almost
> continuously since 2007. That is impressive. And scary.
I have a set of 4 Seagate 320 GB drives in a RAID 5 that have been
running inn my MythTV server since Fall 2006. :-)
-Tom
--
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/
Richard Pieri
>
> I concur with the others that this sounds like a power supply issue,
> though 6 hard drives should not be excessive for a 650W supply. (I have
> as many or more drives on my MythTV server with something like a 450W
> supply.)
I'm running 4x2TB SATA + 1x250GB SATA on my N40L with a 150W PSU.
I believe that Tom correct in his speculation about GPU drivers. That particular Silverstone model is designed to feed power to a high-end GPU, not to lots of disks. The quantity of disks combined with temperature-driven falloff could very well be leaving other parts of the system starved for power.
--Rich P.
David Kramer
> David Kramer wrote:
>> Home-built server/firewall/mail server/web server/MythTV back end/makes
>> coffee.
>
> Others covered this...I'll only add that a router appliance would
> provide a fairly cheap "upgrade", hardly add any additional power usage,
> and improve your security, while offloading some of your server workload.
I feel very strongly that this is not related to my problem, which
presents itself at boot time. I do actually have a WRT54G that I'm not
using because it would lock up when it saw too much load, but maybe it's
a software thing, and it would be fine with different software. I'll
bring it to the installfest and if anyone has time to help me with it,
then that's great, otherwise it will have to wait until I have MUCH more
free time.
If there is any real-world benefit to this at all, it will be a slight
drop in load by having the router block access to ports I don't have
open instead of my server having to handle them.
>> I decided to add another 1TB drive...the server won't boot.
>
> What exactly do you see when it fails to boot?
"DISK BOOT FAILURE, INSERT SYSTEM DISK AND PRESS ENTER"
>> It's a Silverstone Olympia OP640 650W, which should be plenty.
>> It's possible that just the SATA rail is maxed out.
>
> I concur with the others that this sounds like a power supply issue,
> though 6 hard drives should not be excessive for a 650W supply. (I have
> as many or more drives on my MythTV server with something like a 450W
> supply.) But if the supply is optimized for running high powered
> graphics cards it may in fact be running out of current on the 12V line
> used by the drives.
>
>
>> Maybe I should try putting the drive in an external sata case for
>> power...
>
> That wold be a good test and possibly a permanent work around.
I did one better. I borrowed a whole computer from a friend that has a
sata connector (yes, just one. Very strange) on the power supply, and
powered my new hard drive from that, and ran the sata cable from it to
my server. I got identical results: with the DVDRW hooked up, I got the
same disk boot failure message, and with it removed, it would boot up.
Sounds like the power supply is not the issue.
But now that I got to playing with it some, I noticed that every few
seconds or a couple of times a minute the system would seem to freeze
then continue. "top -c" showed very high idle time and low utilization,
and I saw nothing in /var/log/messages. During the freezes the mouse
would not move and the clock did not change (I leave seconds turned on
for EXACTLY this reason). When it would come out of a freeze, the mouse
would often be a bit jerky for a few seconds.
However, I was able to use gparted to partition the drive. The fact
that partitioning worked leads me to believe that the "freezes" might be
a straight UI thing. I would consider debugging this angle by running a
script that writes the time to a file in an endless loop (running in a
text mode tty, not the GUI).
After partitioning the drive, I updated /etc/fstab to mount it and
created the mount directory and made it 777. When I rebooted, I got a
message "/DATA4 NOT READY...S TO SKIP OR M TO MANUALLY CONFIGURE" or
something close to that. I issued the mount command, and it mounted
fine. Very strange. I redid this test and got the same results.
The /var/log/messages showed no problems with the drive (/dev/sde)
during boot:
sd 7:0:0:0: [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
sd 7:0:0:0: [sde] Write Protect is off
sd 7:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't
support DPO or FUA
sde:
sd 6:0:0:0: [sdd] Write Protect is off
sd 6:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't
support DPO or FUA
sdd: sdc1
sdd1
sd 5:0:0:0: [sdc] Attached SCSI disk
sd 6:0:0:0: [sdd] Attached SCSI disk
sde1
sd 7:0:0:0: [sde] Attached SCSI disk
When I manually mounted it, /var/log/messages showed
kjournald starting. Commit interval 5 seconds
EXT3 FS on sde1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Why it mounts on demand but not automatically is baffling.
The current state is that I unhooked that drive and hooked by DVDRW up
again. Now the system is running fine with no periodic freezes. Though
I admit I forgot to comment out the new line in /etc/fstab and had to
edit it and ^M.
Unless someone else has ideas, here are my next steps:
- Try a different sata cable (LONG shot. Brand new cable).
- See if there's anything else I can disable in the BIOS, like parallel
port, and remove any nonessential USB devices to see if my original
assumption of running out of IRQ/DMA is the problem.
- Put the new drive in my friend's machine and see if I can install
Ubuntu on it and boot off of it, proving the drive works fine on its own.
Any other ideas?
Peter Doherty
On Mar 21, 2012, at 12:47 AM, David Kramer wrote:
>
> After partitioning the drive, I updated /etc/fstab to mount it and
> created the mount directory and made it 777. When I rebooted, I got a
> message "/DATA4 NOT READY...S TO SKIP OR M TO MANUALLY CONFIGURE" or
> something close to that. I issued the mount command, and it mounted
> fine. Very strange. I redid this test and got the same results.
So I wonder if the problem could be that the BIOS has to detect all the drives during the POST sequence, and that the drives aren't all spinning up and registering with the BIOS fast enough. It's a long shot.
Check if your BIOS has options for staggered spin-up of the drives. Also try disabling the "quick boot" option that a lot of BIOS have.
Also see if there are updated BIOS for the motherboard, and check the change log for anything that sounds like it might be related to your problem.
The device ordering might be getting messed up. I'd start by walking though all your BIOS options and see if you find anything that sounds like it might help with HDD detection.
Of course, power problems do seem likely, but it doesn't quite add up. Does your PS have a breakdown of the max current output on each voltage line? 3.3v, 5v, 12v?
-peter