Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

root kit cleanup?

5 views
Skip to first unread message

Johnny Walker

unread,
Jun 14, 2001, 10:50:19 PM6/14/01
to
Hi,

I've had an experience with someone installing a rootkit on one of my
systems, and was wondering how to clean up a few files;

Messiah:/bin# rm ps
rm: cannot unlink `ps': Operation not permitted

Messiah:/bin# ln -f ps
ln: `ps' and `./ps' are the same file

Messiah:/bin# mv ps /dev/null
mv: cannot unlink `ps': Operation not permitted
mv: cannot remove `ps': Operation not permitted

Any ideas as to how I can undo what's been done?

The system is Slackware Linux 7.0

Thanks.

Johnny


jeff

unread,
Jun 15, 2001, 12:16:05 AM6/15/01
to
On the glorious day of Fri, 15 Jun 2001 02:50:19 GMT, "Johnny Walker"
<yee...@yahoo.com> felt the need to utter

Well, if at all possible, the best solution (well, most secure) is to
just do a reinstall instead of hoping you have got all the files the
sh*thead touched.
perhaps try `man chattr` for your unlinking problems...

Luke Vogel

unread,
Jun 15, 2001, 1:57:34 AM6/15/01
to

re-install may not always cleanup the files. A format of the hard disk
is recomended practice.

and yes, 'man chattr' and 'man lsattr' should fix your imediate problem.
--
Regards
Luke
------
Q: What does FAQ stand for?
A: We are Frequently Asked this Question, and we have no idea.
------
PLEASE NOTE: Spamgard (tm) installed.
mailto:lukeN...@bell-bird.com.au (remove NOSPAM ... obviously:)
------

pe...@icke-reklam.ipsec.nu.invalid

unread,
Jun 15, 2001, 4:10:18 AM6/15/01
to
Johnny Walker <yee...@yahoo.com> wrote:
> Hi,

> I've had an experience with someone installing a rootkit on one of my
> systems, and was wondering how to clean up a few files;

> Messiah:/bin# rm ps
> rm: cannot unlink `ps': Operation not permitted

> Messiah:/bin# ln -f ps
> ln: `ps' and `./ps' are the same file

> Messiah:/bin# mv ps /dev/null
> mv: cannot unlink `ps': Operation not permitted
> mv: cannot remove `ps': Operation not permitted

> Any ideas as to how I can undo what's been done?

Reinstall from distribution CD's ! Unless you have an exact
list of what's changed, simply replacing some files won't
guarrantee that all backdoors are replaced.

And since you did not know about the linux 'chattr' command
it seems unlikley that you have a database over original
files & signatures.


Do reinstall !!


> The system is Slackware Linux 7.0

> Thanks.

> Johnny

--
Peter Håkanson
IPSec Sverige (At the Riverside of Gothenburg, home of Volvo)
Sorry about my e-mail address, but i'm trying to keep spam out.
Remove "icke-reklam"and "invalid" and it works.

Michael Scheidell

unread,
Jun 15, 2001, 10:40:09 AM6/15/01
to

"Johnny Walker" <yee...@yahoo.com> wrote in message
news:LJeW6.237370$eK2.50...@news4.rdc1.on.home.com...

> Hi,
>
> I've had an experience with someone installing a rootkit on one of my
> systems, and was wondering how to clean up a few files;

reboot from a floppy or cd rom.
reformat and reinstall your os.

even if you have a tripwire md5 hash for all of your files, there could
still be 'things' left in the /tmp dirs and password files.

Also, change ALL your passwords .

the hacker now has your password and shadow files, as well as a good packet
sniff of all your network traffic, including, but not limited to email
accounts on web based systems, and passwords for your routers and switches

check the cert web page below for detailed info if you really want to spend
the time to clean up your system.
I suggest you find a security professional in the area to assist you, unless
you want to worry about 'knats' being left over.

--
Michael Scheidell
Florida Datamation, Inc.
sche...@fdma.com / 1+(561) 368-9561
Internet Security and Consulting
See updated IT Security News at http://www.fdma.com/
After system Compromise : http://www.cert.org/tech_tips/

Bruce Ediger

unread,
Jun 15, 2001, 1:15:46 PM6/15/01
to
In article <9gcfta$3sd$4...@nyheter.crt.se>,
<pe...@icke-reklam.ipsec.nu.invalid> wrote:

>Johnny Walker <yee...@yahoo.com> wrote:
>> I've had an experience with someone installing a rootkit on one of my
>> systems, and was wondering how to clean up a few files;
...

>> Any ideas as to how I can undo what's been done?
>
>Reinstall from distribution CD's ! Unless you have an exact
>list of what's changed, simply replacing some files won't
>guarrantee that all backdoors are replaced.

Oh, bunk. If some kid walking down the alley behind your house opens
your back gate and enters the back yard, do you demolish the whole house
and build a new one?

No.

And that's just what you "security experts" tell people to do. Security
is an economic thing, not a black and white, boolean absolute. It's most
certainly possible to recover from getting rootkitted, it's merely
difficult. It's certainly possible that the cheapest or easiest way to
recover is to backup your data, and reinstall the OS and system stuff,
but it's also certainly possible that's not the cheapest or easiest.

Saying "reinstall from scratch because you can't be absolutely certain
that no backdoors exist in a cracked system" is just plain dopey. Sure,
you can't be 100% certain no back doors exist in a recovered system. You
can't be 100% sure that no back doors exist in a freshly installed system,
either: http://www.acm.org/classics/sep95/ The laws of thermodynamics don't
allow you to be 100% certain that all the air molecules won't pool on the
other side of the room from you, either.

Besides the absolute uncertainty of knowing if one more back door got
installed, is anyone aware of a rootkit that has "layers" of backdoors?

That said, unless you study the rootkit source code carefully, you'll have
a hard time undoing what a rootkit does. It's a lot of work.

Kevan Benson

unread,
Jun 15, 2001, 2:04:01 PM6/15/01
to
Bruce Ediger wrote:

> Oh, bunk. If some kid walking down the alley behind your house opens
> your back gate and enters the back yard, do you demolish the whole house
> and build a new one?
>
> No.
>
> And that's just what you "security experts" tell people to do. Security
> is an economic thing, not a black and white, boolean absolute. It's most
> certainly possible to recover from getting rootkitted, it's merely
> difficult. It's certainly possible that the cheapest or easiest way to
> recover is to backup your data, and reinstall the OS and system stuff,
> but it's also certainly possible that's not the cheapest or easiest.
>
> Saying "reinstall from scratch because you can't be absolutely certain
> that no backdoors exist in a cracked system" is just plain dopey. Sure,
> you can't be 100% certain no back doors exist in a recovered system. You
> can't be 100% sure that no back doors exist in a freshly installed system,
> either: http://www.acm.org/classics/sep95/ The laws of thermodynamics
> don't allow you to be 100% certain that all the air molecules won't pool
> on the other side of the room from you, either.
>
> Besides the absolute uncertainty of knowing if one more back door got
> installed, is anyone aware of a rootkit that has "layers" of backdoors?
>
> That said, unless you study the rootkit source code carefully, you'll have
> a hard time undoing what a rootkit does. It's a lot of work.
>

With that in mind, here are some methods which may help clean the system
without a format (although I do recommend a format if it is fairly easy to
replace the files)

You need access to a fairly unmodified (absolutely unmodified would be
better) Slackware 7.0. Get md5sums of all (or almost all) the files. You
need to take this list to the exploited machine and compare it to a md5sum
list of files on that machine. Any file that does not match needs to be
replaced with an original (unless it's a package you upgraded, in which
case just delete and reinstall that package). This is a poor man's
tripwire, and shouldn't be to hard if you can do a little scripting (bash,
perl python, who cares). Next time, make sure to run something like
tripwire at least, if not tripwire itself and most of this can be avoided.

Depending on how far you go with this, you can revert your distribution to
stock. Nothing makes you feel as warm and fuzzy inside when you have just
installed and secured a system though, probably the ONLY time I ever feel
any system I administer is truly secure.

--
- Kevan Benson
- Sonic.net, Inc.
- (707)522-1000 x219

Christer Palm

unread,
Jun 15, 2001, 3:26:15 PM6/15/01
to
Why all these recommendations about reformatting?!?
Hard disks should not be reformatted - this can definitely cause
trouble!

Michael Scheidell wrote:
>
> reboot from a floppy or cd rom.
> reformat and reinstall your os.
>

--
Christer Palm

Michael Scheidell

unread,
Jun 15, 2001, 5:05:19 PM6/15/01
to
"Christer Palm" <pa...@nogui.se> wrote in message
news:3B2A7D91...@nogui.se...

> Why all these recommendations about reformatting?!?
> Hard disks should not be reformatted - this can definitely cause
> trouble!

??eh?


e t e r s h e r w o o d @home.com Pete Sherwood

unread,
Jun 15, 2001, 5:22:17 PM6/15/01
to
"Michael Scheidell" <m...@privacy.net> wrote in message
news:9gdtah$12rs$1...@caerulus.cerintha.com...

Exactly!

Bill Unruh

unread,
Jun 15, 2001, 6:16:54 PM6/15/01
to
In <LJeW6.237370$eK2.50...@news4.rdc1.on.home.com> "Johnny Walker" <yee...@yahoo.com> writes:

>Hi,

>I've had an experience with someone installing a rootkit on one of my
>systems, and was wondering how to clean up a few files;

>Messiah:/bin# rm ps
>rm: cannot unlink `ps': Operation not permitted

chattr -i ps
But that is not your biggest problem. You have to find all changed
files. The best is to reinstall, and then sweep for suid files.
If you have used rpm, use
rpm -Va |grep '^..5'>/tmp/verify
to see all files which have changed since installation.
(of course your rpm database may have been changed)

Christer Palm

unread,
Jun 15, 2001, 8:38:06 PM6/15/01
to

A re-format may wipe the drives bad-sector table and rewrite the sector
headers which may negatively impact the drives reliability. Note that a
format in MS-DOS terms has nothing to do with formatting, and doesn't
make any sense in a UNIX environment anyway.

--
Christer Palm

Bryan D Howard

unread,
Jun 15, 2001, 9:04:09 PM6/15/01
to
Christer Palm <pa...@nogui.se> writes:
> A re-format may wipe the drives bad-sector table and rewrite the sector
> headers which may negatively impact the drives reliability. Note that a
> format in MS-DOS terms has nothing to do with formatting, and doesn't
> make any sense in a UNIX environment anyway.

Yes it's all a matter of different terminology. It's really too bad
that MS got the terminology so wrong, but I guess that's because they
originally invented everything (retroactively and poorly).

Anyway, what MS folks call format, Unix folks call mkfs. *We* all
know that format means something else.

{Bryan}

Michael Scheidell

unread,
Jun 16, 2001, 11:45:11 AM6/16/01
to

"Bryan D Howard" <br...@overtone.jetcafe.org> wrote in message
news:m23d91i...@overtone.jetcafe.org...

I guess it all depends on what your definition of is is.

Colin McKinnon

unread,
Jun 18, 2001, 5:26:50 AM6/18/01
to
Bruce Ediger <ebal...@qwest.net> wrote in message
news:6prW6.326$xJ.1...@news.uswest.net...

> Oh, bunk. If some kid walking down the alley behind your house opens
> your back gate and enters the back yard, do you demolish the whole house
> and build a new one?
>
> No.
>
> And that's just what you "security experts" tell people to do. Security
> is an economic thing, not a black and white, boolean absolute. It's most
> certainly possible to recover from getting rootkitted, it's merely
> difficult. It's certainly possible that the cheapest or easiest way to
> recover is to backup your data, and reinstall the OS and system stuff,
> but it's also certainly possible that's not the cheapest or easiest.

Re-installing from scratch / restoring from backup is not the only
solution - I'd agree, but the only acceptable alternative is to verify your
existing installation. If you already know how to do this properly, you can
afford to disregard the advice to re-install from scratch. If you're not
properly prepared for a system compromise or you're not 100% sure that you
can establish exactly what the attacker has done to your system then
anything else is plain stupid.

Advocating that someone disregard good advice without qualifying the context
is, IMHO, a little irresponsible too.

Colin


Juergen P. Meier

unread,
Jun 19, 2001, 12:30:26 AM6/19/01
to
Bruce Ediger <ebal...@qwest.net> wrote:
>In article <9gcfta$3sd$4...@nyheter.crt.se>,
> <pe...@icke-reklam.ipsec.nu.invalid> wrote:
>>Johnny Walker <yee...@yahoo.com> wrote:
>>> I've had an experience with someone installing a rootkit on one of my
>>> systems, and was wondering how to clean up a few files;
> ...
>>> Any ideas as to how I can undo what's been done?
>>
>>Reinstall from distribution CD's ! Unless you have an exact
>>list of what's changed, simply replacing some files won't
>>guarrantee that all backdoors are replaced.
>
>Oh, bunk. If some kid walking down the alley behind your house opens
>your back gate and enters the back yard, do you demolish the whole house
>and build a new one?
>
>No.
>
>And that's just what you "security experts" tell people to do. Security
>is an economic thing, not a black and white, boolean absolute. It's most
>certainly possible to recover from getting rootkitted, it's merely
>difficult. It's certainly possible that the cheapest or easiest way to
>recover is to backup your data, and reinstall the OS and system stuff,
>but it's also certainly possible that's not the cheapest or easiest.

Guess what the US Department of Foreign Affairs did with the US
Ambassy Building in Moscow several Years ago.

Did they bother to "clean it up" ?

Its not your backyard-burglar who's breaking into your house, its
rather a high-tech espionage organisation that bugs your house inside
out.

Do you think you could cleanup a house bugged by some organisation
like the ex-KGB or the NSA/CIA? Do _you_ really think you could
find and remove all bugs and backdoors ?

If you have to make an analogy, please bother to make it a good one.

juergen

--
J...@lrz.fh-muenchen.de - "This World is about to be Destroyed!"

Juergen P. Meier

unread,
Jun 19, 2001, 12:34:36 AM6/19/01
to
Juergen P. Meier <J...@lrz.fh-muenchen.de> wrote:
>Guess what the US Department of Foreign Affairs did with the US
>Ambassy Building in Moscow several Years ago.
>
>Did they bother to "clean it up" ?

Actually, they did.

But did they let some amateur do this? no. They had
highly trained experts for doing this job.

what do we learn from this analogy?

Marc SCHAEFER

unread,
Jun 19, 2001, 9:27:49 AM6/19/01
to
Luke Vogel <lu...@bell-bird.com.au> wrote:
> re-install may not always cleanup the files. A format of the hard disk
> is recomended practice.

Sometimes you also need to update the BIOS. Or, even, revert to a previous
Intel processor microcode

:)

(not yet true, might become).

David Ressman

unread,
Jun 25, 2001, 1:06:18 AM6/25/01
to
Bruce Ediger <ebal...@qwest.net> wrote:
> Besides the absolute uncertainty of knowing if one more back door got
> installed, is anyone aware of a rootkit that has "layers" of backdoors?

I was hit by one last week. We had an Indy running IRIX 6.2 (almost
completely unpatched) that got broken into. As I was poking around doing
the post mortem, I found a very noticeable set of backdoors just because
they were datestamped at the time of the breakin. After I cleaned up
that kit, I noticed a discrepancy between the information "ps" was giving
me and the info that a clean copy of "ps" I had copied over from another
system was giving me.

/usr/lib/ias/scheme (/bin/login), telnetd, ps, and a couple others had
been replaced with replacements that were identical in size and mtime
to the binaries that shipped with IRIX. The only way I caught those was
was because the SCCS information supplied by "what".

Did they also make another rootkit with fake SCCS information in the
binaries? Maybe, maybe not. I reinstalled.

Just because you haven't seen one doesn't mean they don't exist. If you
just clean up the obvious backdoors and don't use tripwire or aide, how
can you be sure you don't have any on your systems right now?

David

Bruce Ediger

unread,
Jun 27, 2001, 4:26:50 PM6/27/01
to
In article <9h6gsa$14r$1...@bob.news.rcn.net>,

David Ressman <phu...@enteract.com> wrote:
>Bruce Ediger <ebal...@qwest.net> wrote:
>> Besides the absolute uncertainty of knowing if one more back door got
>> installed, is anyone aware of a rootkit that has "layers" of backdoors?
>
>I was hit by one last week. We had an Indy running IRIX 6.2 (almost
...

>Did they also make another rootkit with fake SCCS information in the
>binaries? Maybe, maybe not. I reinstalled.

Very interesting. Thanks for the report. I haven't heard of anything
like this before.

>Just because you haven't seen one doesn't mean they don't exist. If you
>just clean up the obvious backdoors and don't use tripwire or aide, how
>can you be sure you don't have any on your systems right now?

I found the rootkit distribution, I read the "setup" script in it, and
confirmed everything the "setup" script did, reversing the installation
of trojaned executables.

If anything had seemed suspicious about the setup script, or anything
didn't jibe, I probably would have re-installed the OS. But everything
matched up, so I didn't. After that, I put the machine behind a firewall.
That doesn't entirely let it off the hook, but at least backdoor sshd's
aren't accesible on port 25000 or 62464, and odd RPC services can't be
called from outside the firewall.

Eric Myers

unread,
Jun 27, 2001, 10:52:47 PM6/27/01
to
In article <ekr_6.2497$T_2.4...@news.uswest.net>,

Bruce Ediger <bed...@qwest.net> wrote:
>I found the rootkit distribution, I read the "setup" script in it, and
>confirmed everything the "setup" script did, reversing the installation
>of trojaned executables.
>
>If anything had seemed suspicious about the setup script, or anything
>didn't jibe, I probably would have re-installed the OS. But everything
>matched up, so I didn't.

You found ONE rootkit, but how do you know there are not more?
I just went over a machine that was used to scan a network in Japan
(they complained) and I quickly found the rootkit in a directory
named ".. " (note the space). But after looking a bit more I found
at least one more rootkit, with stuff in /usr/man/man1/.../ and
/usr/X11R6/lib/X11/videonet. I think I found it all, but how do I
really know? There might be more. Re-install from scratch!

--
Eric Myers <my...@umich.edu>
High Energy Theoretical Physics Tel: 734-763-4325
Department of Physics Fax: 734-763-2213
University of Michigan, Ann Arbor http://www.umich.edu/~myers

Peter F. Curran

unread,
Jun 28, 2001, 9:18:43 AM6/28/01
to
In article <ekr_6.2497$T_2.4...@news.uswest.net>,
bed...@qwest.net (Bruce Ediger) writes:
[snip]

>If anything had seemed suspicious about the setup script, or anything
>didn't jibe, I probably would have re-installed the OS. But everything
>matched up, so I didn't. After that, I put the machine behind a firewall.
>That doesn't entirely let it off the hook, but at least backdoor sshd's
>aren't accesible on port 25000 or 62464, and odd RPC services can't be
>called from outside the firewall.

Reinstall at first opportunity. Lots of nasty things can be done with
simple text editing tools, even to user accounts. Consider that 'su' or
even 'ls' might be aliased there, running some nasty person's most evil
script on YOUR machine. The truth is that even reinstalling the
operating system won't necessarily give you a "clean" machine unless you
also wipe all applications and user data.

--
Peter F Curran
Rensselaer Polytechnic Institute


"If you paid for your operating system, you probably
paid too much for your operating system."
**** USE EMAIL ADDRESS IN ORG LINE TO REPLY ****

inigma_x

unread,
Jun 30, 2001, 6:45:04 AM6/30/01
to
You should also worry about loadable kernel modules. A evil kernel
module can hide files, processes, open ports, ect...
0 new messages