Accidently ran fsck on a mounted HDD and made problems! What to do?
Ant
I did something stupid earlier. I accidently ran fsck command on a
mounted HDD in Debian. I quickly ctrl-c'ed to get out of it. Now, I am
having HDD problems and I can't seem to recover/undo what I did. I
rebooted and my box didn't come back (can't check the box since I am at
work and doing it via SSH).
dmesg showed this after I before I rebooted it remotely with shutdown -r
now at: http://pastebin.ca/672751
Then, I decided to reboot and nothing came back (can't SSH back in). I
think it got stuck somewhere? :( If it indeed stuck, how do I fix this
after I get home to see the console?
Thank you in advance.
--
"He who cannot pick up an ant, and wants to pick up an elephant will
some day see his folly." --African
/\___/\
/ /\ /\ \ Ant @ http://antfarm.ma.cx (Personal Web Site)
| |o o| | Ant's Quality Foraged Links (AQFL): http://aqfl.net
\ _ / Please remove ANT if replying by e-mail.
( )
The Natural Philosopher
> Hello.
>
> I did something stupid earlier. I accidently ran fsck command on a
> mounted HDD in Debian. I quickly ctrl-c'ed to get out of it. Now, I am
> having HDD problems and I can't seem to recover/undo what I did. I
> rebooted and my box didn't come back (can't check the box since I am at
> work and doing it via SSH).
>
> dmesg showed this after I before I rebooted it remotely with shutdown -r
> now at: http://pastebin.ca/672751
>
> Then, I decided to reboot and nothing came back (can't SSH back in). I
> think it got stuck somewhere? :( If it indeed stuck, how do I fix this
> after I get home to see the console?
>
> Thank you in advance.
You should have let it run...
get a spare disk, boot from CD and dd the disk off...then fsck the old disk.
Chances are it will recover, or you wil be able to patch most of it.
Darren Salt
> I did something stupid earlier. I accidently ran fsck command on a mounted
> HDD in Debian. I quickly ctrl-c'ed to get out of it. Now, I am having HDD
> problems and I can't seem to recover/undo what I did. I rebooted and my box
> didn't come back (can't check the box since I am at work and doing it via
> SSH).
> dmesg showed this after I before I rebooted it remotely with shutdown -r
> now at: http://pastebin.ca/672751
[sample from that paste follows]
> EXT3-fs error (device hda1): ext3_free_blocks_sb: bit already cleared for block 228823
> Remounting filesystem read-only
> EXT3-fs error (device hda1): ext3_free_blocks_sb: bit already cleared for block 228824
> EXT3-fs error (device hda1): ext3_free_blocks_sb: bit already cleared for block 228825
[ditto for 228827 to 228830, 228832]
> EXT3-fs error (device hda1): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=2553887680, rec_len=0, name_len=0
> EXT3-fs error (device hda1): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=2553887680, rec_len=0, name_len=0
> EXT3-fs error (device hda1): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=2553887680, rec_len=0, name_len=0
> EXT3-fs error (device hda1): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=2553887680, rec_len=0, name_len=0
Ouch.
It looks like some objects were updated while fsck was running, and fsck has
consequently overwritten newer metadata with older. Your ^C may well have
prevented further damage...
> Then, I decided to reboot and nothing came back (can't SSH back in). I
> think it got stuck somewhere? :(
When fsck failed, I expect; if so, you should see a root password prompt on
the console.
> If it indeed stuck, how do I fix this after I get home to see the console?
Logging in & re-running fsck should be fine, but you should take that advice
about using a CD and making a copy (consider "belt & braces"). (The Debian
install CD will be fine for this.)
If you do log in and run fsck directly, you should find that the damaged
partition is either mounted read-only or not yet mounted.
--
| Darren Salt | linux or ds at | nr. Ashington, | Toon
| RISC OS, Linux | youmustbejoking,demon,co,uk | Northumberland | Army
| + Output *more* particulate pollutants. BUFFER AGAINST GLOBAL WARMING.
Nobody's perfect. Therefore, since I'm a nobody, it follows that I'm perfect.
Ant
> > HDD in Debian. I quickly ctrl-c'ed to get out of it. Now, I am having HDD
> > problems and I can't seem to recover/undo what I did. I rebooted and my box
> > didn't come back (can't check the box since I am at work and doing it via
> > SSH).
> > dmesg showed this after I before I rebooted it remotely with shutdown -r
> > now at: http://pastebin.ca/672751
> [sample from that paste follows]
> > EXT3-fs error (device hda1): ext3_free_blocks_sb: bit already cleared for block 228823
> > Remounting filesystem read-only
> > EXT3-fs error (device hda1): ext3_free_blocks_sb: bit already cleared for block 228824
> > EXT3-fs error (device hda1): ext3_free_blocks_sb: bit already cleared for block 228825
> [ditto for 228827 to 228830, 228832]
> > EXT3-fs error (device hda1): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=2553887680, rec_len=0, name_len=0
> > EXT3-fs error (device hda1): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=2553887680, rec_len=0, name_len=0
> > EXT3-fs error (device hda1): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=2553887680, rec_len=0, name_len=0
> > EXT3-fs error (device hda1): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=2553887680, rec_len=0, name_len=0
> Ouch.
> It looks like some objects were updated while fsck was running, and fsck has
> consequently overwritten newer metadata with older. Your ^C may well have
> prevented further damage...
Oh good.
> > Then, I decided to reboot and nothing came back (can't SSH back in). I
> > think it got stuck somewhere? :(
> When fsck failed, I expect; if so, you should see a root password prompt on
> the console.
OK. I hope it is there now.
> > If it indeed stuck, how do I fix this after I get home to see the console?
> Logging in & re-running fsck should be fine, but you should take that advice
> about using a CD and making a copy (consider "belt & braces"). (The Debian
> install CD will be fine for this.)
I have KNOPPIX v4.1 CD so I hope that's enough. I always have this CD
and the latest versions for emergencies for like this one!
> If you do log in and run fsck directly, you should find that the damaged
> partition is either mounted read-only or not yet mounted.
Do I just run fsck /dev/hda and it let run with defaults?
Ant
> Ant wrote:
> > Hello.
> >
> > I did something stupid earlier. I accidently ran fsck command on a
> > mounted HDD in Debian. I quickly ctrl-c'ed to get out of it. Now, I am
> > having HDD problems and I can't seem to recover/undo what I did. I
> > rebooted and my box didn't come back (can't check the box since I am at
> > work and doing it via SSH).
> >
> > dmesg showed this after I before I rebooted it remotely with shutdown -r
> > now at: http://pastebin.ca/672751
> >
> > Then, I decided to reboot and nothing came back (can't SSH back in). I
> > think it got stuck somewhere? :( If it indeed stuck, how do I fix this
> > after I get home to see the console?
> >
> > Thank you in advance.
> You should have let it run...
Leave it running while mounted? Isn't fscking while mounted bad?
> get a spare disk, boot from CD and dd the disk off...then fsck the old disk.
> Chances are it will recover, or you wil be able to patch most of it.
OK. I am going to use KNOPPIX 4.1 CD and fsck both /dev/hda and /dev/hdb
(two HDDs).
jellybean stonerfish
>
>> You should have let it run...
>
> Leave it running while mounted? Isn't fscking while mounted bad?
Perhaps stopping a disk fixing process is bad-der.
stonerfish
Darren Salt
>>> I did something stupid earlier. I accidently ran fsck command on a
>>> mounted HDD in Debian. I quickly ctrl-c'ed to get out of it. Now, I am
>>> having HDD problems and I can't seem to recover/undo what I did. I
>>> rebooted and my box didn't come back (can't check the box since I am at
>>> work and doing it via SSH).
>>> dmesg showed this after I before I rebooted it remotely with shutdown -r
>>> now at: http://pastebin.ca/672751
>> [sample from that paste follows]
>>> [...] ext3_free_blocks_sb: bit already cleared for block 228823
>>> Remounting filesystem read-only
[more "bit cleared" errors, and some "bad entry in directory" errors]
>> Ouch.
>> It looks like some objects were updated while fsck was running, and fsck
>> has consequently overwritten newer metadata with older. Your ^C may well
>> have prevented further damage...
> Oh good.
... and here's the bad news: (AIUI) fsck tries to write everything which it's
changed in one go when it's finished scanning the partition.
>>> Then, I decided to reboot and nothing came back (can't SSH back in). I
>>> think it got stuck somewhere? :(
[snip]
>>> If it indeed stuck, how do I fix this after I get home to see the
>>> console?
>> Logging in & re-running fsck should be fine, but you should take that
>> advice about using a CD and making a copy (consider "belt & braces").
>> (The Debian install CD will be fine for this.)
> I have KNOPPIX v4.1 CD so I hope that's enough. I always have this CD
> and the latest versions for emergencies for like this one!
If it's new enough or your hardware is old enough, it should be fine...
>> If you do log in and run fsck directly, you should find that the damaged
>> partition is either mounted read-only or not yet mounted.
> Do I just run fsck /dev/hda and it let run with defaults?
Yes (well, /dev/hda1), but expect some questions and expect to have to do
some cleanup work (as mentioned elsewhere in this thread) afterwards.
Again, consider making a copy of the unrepaired partition in case something
goes wrong or you need a reference point.
--
| Darren Salt | linux or ds at | nr. Ashington, | Toon
| RISC OS, Linux | youmustbejoking,demon,co,uk | Northumberland | Army
| + Output less CO2 => avoid massive flooding. TIME IS RUNNING OUT *FAST*.
I'd like to, but I'm observing National Apathy Week.
Ant
> >> Ouch.
> > Oh good.
Well, I don't really have the storage for it. I copied my data/work
datas though and i have a backup of /etc.
Anyways, bad news:
I came home. I saw a bunch of password errors for sulogin or something.
I couldn't type anything. I was able to press numlock to see light turn
on and off. I couldn't reboot with the three keys so I had to power
shutdown. I repowered back on and Grub said error code 2. It didn't say
much in Grub's Web site.
I booted up KNOPPIX v5.0 (not 5.1.1) and I couldn't mount the hda1 in
KDE: "mount: wrong fs type, bad option, bad superblock on /dev/hda1." I
ran fsck /dev/hda and got:
root@1[knoppix]# fsck /dev/hda
fsck 1.39-WIP (09-Apr-2006)
e2fsck 1.39-WIP (09-Apr-2006)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/hda
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate
superblock:
e2fsck -b 8193 <device>
I ran fdisk /dev/hda to see if I can read it at all:
Command (m for help): p
Disk /dev/hda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 1 36 289138+ 83 Linux
/dev/hda2 37 9729 77859022+ 5 Extended
/dev/hda5 37 1860 14651248+ 83 Linux
/dev/hda6 1861 2468 4883728+ 83 Linux
/dev/hda7 2469 2590 979933+ 83 Linux
/dev/hda8 2591 2712 979933+ 83 Linux
/dev/hda9 2713 3320 4883728+ 83 Linux
/dev/hda10 3443 3736 2361523+ 83 Linux
/dev/hda11 3737 9729 48138741 83 Linux
/dev/hda12 3321 3442 979933+ 83 Linux
Partition table entries are not in disk order
I am going to try e2fsck -b 8193 /dev/hda (heard ext3 is based on ext2
with journaling).
It looks like I am going to need to reinstall from scratch if it is not
recoverable. :(
Note: My HDDs are just IDE. Nothing fancy.
Ant
> > >> Ouch.
> > > Oh good.
> Anyways, bad news:
See http://pastebin.com/m5ca8c3f0 first.
I basically held down Y for like a few minutes during fsck /dev/hda1. :(
I tried to mount this partition with KDE, but it didn't see anything (uh
oh). All other partitions except the swap one seemed fine.
I am going to go reboot.
Ant
> > > >> Ouch.
> > > > Oh good.
> > Anyways, bad news:
> See http://pastebin.com/m5ca8c3f0 first.
Grub 1.5 now says error 15 meaning file not found according to Google
search results. :( Crap, hda1 is still empty. AUGH!
Back in KNOPPIX, I found out where all the datas went in hda1. They all
ended up in lost+found/ directory: http://pastebin.ca/673132 ...
I thought I could move these back, but I realized I am missing other
files like Kernel images. I think I am stuck and need to do a reinstall
soon. :(
Anton Ertl
>On Mon, 27 Aug 2007 18:21:26 -0500, Ant wrote:
>>> You should have let it run...
I would not recommend that.
>> Leave it running while mounted? Isn't fscking while mounted bad?
Yes.
>Perhaps stopping a disk fixing process is bad-der.
I don't think so. It happens to me frequently (although not on a
mounted FS), and the FSs are fine.
As long as the fsck has not changed anything (and AFAIK it always
writes messages when it changes things), he should be fine. Once the
fsck has started changing anything, I guess the best approach is to
hit the reset button, boot with a rescue disk, and fsck the file
system properly. And copying the partition to a spare disk can't
hurt, either.
Followups set to comp.os.linux.misc.
- anton
--
M. Anton Ertl Some things have to be seen to be believed
an...@mips.complang.tuwien.ac.at Most things have to be believed to be seen
http://www.complang.tuwien.ac.at/anton/home.html
Anton Ertl
>I booted up KNOPPIX v5.0 (not 5.1.1) and I couldn't mount the hda1 in
>KDE: "mount: wrong fs type, bad option, bad superblock on /dev/hda1." I
>ran fsck /dev/hda and got:
>
>root@1[knoppix]# fsck /dev/hda
If /dev/hda1 is the bad file system, you should fsck /dev/hda1. And
if fsck does not recognize the file system, use e2fsck.
The Natural Philosopher
You may bugger up files in use, but not static ones.
stopping halfway may bugger up the complete directory.
> stonerfish
Darren Salt
> In alt.os.linux.debian Ant <ANT...@zimage.com> wrote:
[huge snip]
>> See http://pastebin.com/m5ca8c3f0 first.
>> I basically held down Y for like a few minutes during fsck /dev/hda1. :(
>> I tried to mount this partition with KDE, but it didn't see anything (uh
>> oh). All other partitions except the swap one seemed fine.
>> I am going to go reboot.
> Grub 1.5 now says error 15 meaning file not found according to Google
> search results. :( Crap, hda1 is still empty. AUGH!
Well, at least you have /var - you can make use of dpkg's --get-selections
and --set-selections options.
> Back in KNOPPIX, I found out where all the datas went in hda1. They all
> ended up in lost+found/ directory: http://pastebin.ca/673132 ...
Anything which you've customised and can identify from that lot (fstab and
xorg.conf, for example), you should restore after reinstalling.
You may be able to do a minimal install in your root partition, after which
you can mount your existing /usr and /var then just force reinstallation of
(more or less) all packages. Afterwards, you can unmount /usr and /var then
remove the content which was hidden by the mount points.
(Untested. This advice is given without warranty or any guarantee of fitness
for purpose. May cause nasal demons. No sacrifice necessary.)
> I thought I could move these back, but I realized I am missing other
> files like Kernel images. I think I am stuck and need to do a reinstall
> soon. :(
Looks like it...
--
| Darren Salt | linux or ds at | nr. Ashington, | Toon
| RISC OS, Linux | youmustbejoking,demon,co,uk | Northumberland | Army
| + Burn less waste. Use less packaging. Waste less. USE FEWER RESOURCES.
I'm a figment of my imagination.
Ant
> I demand that Ant may or may not have written...
> > In alt.os.linux.debian Ant <ANT...@zimage.com> wrote:
> [huge snip]
> >> See http://pastebin.com/m5ca8c3f0 first.
> >> I basically held down Y for like a few minutes during fsck /dev/hda1. :(
> >> I tried to mount this partition with KDE, but it didn't see anything (uh
> >> oh). All other partitions except the swap one seemed fine.
> >> I am going to go reboot.
> > Grub 1.5 now says error 15 meaning file not found according to Google
> > search results. :( Crap, hda1 is still empty. AUGH!
> Well, at least you have /var - you can make use of dpkg's --get-selections
> and --set-selections options.
Well, I do run apt-get clean once in a while so thoe packages get nuked.
:( It's already too late since I already cleaned out /var and /tmp
directories/partitions.
> > Back in KNOPPIX, I found out where all the datas went in hda1. They all
> > ended up in lost+found/ directory: http://pastebin.ca/673132 ...
> Anything which you've customised and can identify from that lot (fstab and
> xorg.conf, for example), you should restore after reinstalling.
Yeah, I used the one from my backup though. Those were like a month old
so it wasn't too bad.
> You may be able to do a minimal install in your root partition, after which
> you can mount your existing /usr and /var then just force reinstallation of
> (more or less) all packages. Afterwards, you can unmount /usr and /var then
> remove the content which was hidden by the mount points.
That is what I did (mounting). ;)
> (Untested. This advice is given without warranty or any guarantee of fitness
> for purpose. May cause nasal demons. No sacrifice necessary.)
> > I thought I could move these back, but I realized I am missing other
> > files like Kernel images. I think I am stuck and need to do a reinstall
> > soon. :(
> Looks like it...
Yep. Done. Just need to reinstall and reconfigure things.
M...@privacy.net
If you have a backup, do a restore. Problem solved.
If you don't have a backup, you are a moron and deserve
to lose all of your data. Consider it the price you pay
for being an idiot, and start doing regualr backups.
I hope this helps...