I have just joined this group because I felt I own Carlo this
feedback. On Friday, the 9th, one of our employees deleted all her
files from her share in the server (her home directory). She informed
me 7 hours later, after writing lots of stuff back there, with lots of
disk activity in the server due to the other users. Then, and only
then, I did shut down the server. There is no backup (strangely
enough, our CEO regards backup procedures as a nuisance, thus he is
delaying the budget to buy a backup solution – yes, I know, he is an
a**hole). I spent the weekend searching for tools and around 2:00 am
of Monday the 12th (yesterday), I found ext3grep. All other solutions
failed, mostly Linux-based Live CDs, due to a known issue with the
server’s board Intel DG965WH, which can boot from a CD but fails to
recognize any IDE devices after boot, thus leaving me whit “a very
limited shell” (always the same message – most distributions were
Knoppix-based). My system is Ubuntu 7.10, and I have a Linux
(software) RAID5 with 4 Maxtor SATA2 200 GB disks. I am not a Linux
expert, I keep learning on-the-fly, and I have installed some stuff I
though I would never be able even to understand. But I am not,
definitively, a programmer, I do not program even shell scripts. I
just do not enjoy it. I first read the documentation about ext3grep
(amazing stuff, even though it is too technical for my poor intellect,
nevertheless I was finally able to understand something about
filesystems). I download the source. I installed Ubuntu into a spare
80 GB Maxtor SATA2 disk. I could not dd my RAID since I have no other
spare disk (thanks again, CEO) and we are up to 200 GB of data now, so
I had to work with the actual filesystem – it was really the only
choice, or I could say "this is not my problem" and go home to stuff
meself with some beer. I COULD leave things like that (I am not paid
to perform magic nor miracles - I am not even fairly paid, to tell the
truth), but all things considered (the CEO, the lack of budget for
training and spare parts and wathnot), I decided it was morally wrong
to throw our employee off the sledge for the wolfs in order to save my
skin. It is better to gang up against the CEO. I then proceeded to
compile the source (some minor problems there – “unknown depmode none”
and “fail sanity check” –, due to my ignorance about c and gcc),
mounted the array as read-only, with the server isolated in another
subnetwork and in single-user mode (never done that before, a few
false starts). Fase1 and Fase2 worked fine, and the only problem I
really had was with --after and --before, it seemed to ignore the
timestamp values and recovered *everything*. I used 12103380000 and
1210341600 as timestamp, it is the interval of time I considered (she
said the problem occurred between 8:00 am and 9:00 am), thus these
values should give me a margin to work. Also, the new Ubuntu
installation had wrong locales, everything was “POSIX”. My actual
server has “es_CO.UTF-8” (Spanish, Colombia), and date returns me “mar
may 13 18:42:21 COT 2008” (“mar” stands for the weekday in Spanish,
Tuesday – Martes, and “COT” is Colombian Time). So, ext3grep started
recovering *EVERYTHING*. My first try was aborted after the entire
disk got filled. I deleted everything recovered, and started again,
this time I had to constantly stop (crtl-z) the process, delete the
garbage, df –k to be sure, fg the process, wait a couple of minutes,
stop, and so on and so forth, up to the point it started to recover
the girl’s directory. At that point, I just left the process running
until the whole directory was restored and the process finished
normally, because her directory was the last one. This accomplished, I
copied the results to a temporary location, took out the temp system
disk, put back the original system disk, fired the server again in
multiuser mode to normal operation and copied the recovered files to
the girls’ directory (I did not think about waiting for her to confirm
the results, I was not going thru all this again and could not afford
to have the server inactive another day). The girls said the most
important things were recovered, so no-one got fired. I have no clue
why ext3grep ignored the timestamps. I though it would be easier to
recover her entire directory, since it has been created less than a
month ago, so it would be no problem. But I also could neither
understand how to guess her directory’s block from Fase1 file or Fase2
file, nor which option is to be used in this case, because, as I said
before, I am not a Linux expert, nor programmer, so I could not
understand the principles explained in the operational part of Carlo’s
document. It assumes knowledge of other tools and techniques which I
do not possess. Nevertheless, the simple fact that ext3grep exists and
works is all that matters. For those with better knowledge, it must be
a relief to know someone happened to find an answer for all their
problems. Also, it is still alpha, so the little “peccadilloes” are
forgivable. All in all, I worked 72 hour non-stop, with a little more
than coffee and cigarettes to feed me, doing something I had never
done before with tools and techniques I could barely understand, with
less than the adequate resources and tools, and it worked better than
I could ever expect in my wildest dreams. It was not my responsibility
to recover those files (although I guess all sysadmins think
everything that happen within our domains is our responsibility), and
absolutely no-one apart from the girl thanked me. Given the
environmental conditions (no tools, no money, delayed shutdown, lack
of knowledge), I was not supposed to be able to recover anything. But
hadn’t I tried, I would never know, would not have learnt new stuff
(and the first thing I learnt was that you cannot undelete ext3 files
and then again, you can), the girl would be looking for a new job now,
and I would never had this case to report. Really, nothing that
happened during the process is important, what is important is that an
alpha tool went “where no man has gone before” - agin!. I must say
this is an important tool, I really hope it keeps being developed and
improved. And before Carlo asks if I will donate, let me tell I live
in Colombian time, with Colombian money, do not expect a million
dollars postal order coz I can barely pay my bills – I am poor, in
other words, but I intend to do so, as soon as I can put my hands into
some coins. We all spend a lot of money in useless sh*t most of the
time (like food, rent, medicine or clothes - what is it that is so
important about clothes, anyway? Or food, for all the matter? But
don't touch my new iPod!), so I suppose everybody can spare a couple
of coins, and if we all do this we ensure continuity of open source.
And feed the programmer, after all.
On Tue, May 13, 2008 at 05:59:45PM -0700, lobo_loco wrote: > the girl’s directory. At that point, I just left the process running > until the whole directory was restored and the process finished > normally, because her directory was the last one.
Murphy! You little bastard you!
> of knowledge), I was not supposed to be able to recover anything. But > hadn’t I tried, I would never know, would not have learnt new stuff > (and the first thing I learnt was that you cannot undelete ext3 files > and then again, you can), the girl would be looking for a new job now, > and I would never had this case to report.
It is very special to me that my tool actually caused some poor girl to keep her job! Wow.
> improved. And before Carlo asks if I will donate, let me tell I live > in Colombian time, with Colombian money, do not expect a million > dollars postal order coz I can barely pay my bills – I am poor, in > other words, but I intend to do so, as soon as I can put my hands into > some coins. We all spend a lot of money in useless sh*t most of the > time (like food, rent, medicine or clothes - what is it that is so > important about clothes, anyway? Or food, for all the matter? But > don't touch my new iPod!), so I suppose everybody can spare a couple > of coins, and if we all do this we ensure continuity of open source. > And feed the programmer, after all.
You should DEFINITELY not donate anything out of your own pocket! You already did enough, putting 72 hours of your time into helping this girl. Sure-- if you could (without much effort) have the company that you work for 'donate' something-- then I think that would be ok; because the software did save them some grief, and it IS their fault for not having backups. But I wouldn't even want you to go through any trouble for that.
Thank you for the story, and for saving the girls job! :)
PS ext3grep was written, originally, for my particular case where I deleted the whole partition... so, it recovers the whole partition. There is no way to tell it to just recover only a single directory. It would be easy to write that, but I'm waiting for Kare to do that :p Or maybe for someone to pay me 100 euro with the special request to write a patch for that :p
