Advice Needed On Recent Rootings
Jayson Vantuyl
person (Romanian we think) is specifically targeting us and our
customers (looks like he hit a machine and found connections from others
in their logs, went from there).
We have no idea how he's getting in, but we've got his rootkit fairly
nailed down (he uses a few slightly different ones).
We've caught a few systems as he was breaking in (we have
.bash_history files and the site he downloads his rootkits from).
The part that bothers me is that all of these systems were updated to
the newest versions on debian.security.org (if apt-get was doing its
job) and firewalled down to just the ports we needed (22, 25, 53, 80).
My boss is thinking they might have some sort of crack for OpenSSH (only
service I can say all of these have in common) and he's considering
trying a switch to the nonfree one just to see if it helps.
While I don't like this (OpenSSH is open and it should be that way), has
anyone else had this kind of experience? Is there some big hack I
should know about?
I've checked CERT and the SANS list. Both of them were helpful, but
most of the answers said "run the newest version of X", which I have
assumed apt-get fixed (in stable at least). I mean, some versions were
older, but I had heard most of them had backported fixes. Is this
happening to anyone else?
I'm at a complete loss as to how to explain this one, help would be
appreciated. The only comforting thought is that I can't imaging Redhat
would have done any better.
Jayson Vantuyl
Computing Edge, Inc.
--
To UNSUBSCRIBE, email to debian-secu...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Noah Meyerhans
> We have no idea how he's getting in, but we've got his rootkit fairly
> nailed down (he uses a few slightly different ones).
If you believe he'll be back, it might be worth it to set up a honeypot
and a box running tcpdump and capturing all the traffic to honeypot.
Set the honeypot up with the same services you run on your production
machines, and make sure that no services at all (not even ssh) are
runnign on the tcpdump system. At least that way, when the box gets
cracked, you'll be able to see what ports the guy was talking to when he
broke in. It also might be useful as evidence in case you ever decide
to try and prosecute him.
I assume the cracked boxes were running woody? What are the actual
services running on the various open ports? What versions of the
packages?
I don't know of any exploits for the version of OpenSSH included on
security.debian.org for woody. It would certainly be interesting to
find out that there is one in the wild!
noah
--
_______________________________________________________
| Web: http://web.morgul.net/~frodo/
| PGP Public Key: http://web.morgul.net/~frodo/mail.html
Ed McMan
Jayson> We've had a number of hacked boxen recently. It appears a certain
Jayson> person (Romanian we think) is specifically targeting us and our
Jayson> customers (looks like he hit a machine and found connections from others
Jayson> in their logs, went from there).
That's pretty unsettling..
Have you tried running snort? If its a known vulnerability it should
be able to pick it up (don't use Debian's.. it's very out of date).
You might want to try scanning your boxes with nessus too (kind of
unlikely that it would find anything, but... (don't use debian version
again)).
Have all of the hacked boxes been running a while without a reboot?
Someone discussed that programs running from updated libraries
would still be vulnerable until they were restarted. For instance, if
you havn't restarted ssh or apache (if you're using ssl) since openssl
was upgraded, an openssl exploit would still work.
------------------------------------------------------
| Eddie J Schwartz <EdMcMan@[despammed.com|m00.net]> |
| AIM: Uncaring Eyes ICQ: 35576339 YHOO: edmcman2 |
| "We Trills have an expression -- at forty, you |
| think you know everything. At four hundred you |
| realize you know nothing." - Dax, Startrek DS9 |
------------------------------------------------------
Marc-Christian Petersen
Hi Jayson,
> We've had a number of hacked boxen recently. It appears a certain
> person (Romanian we think) is specifically targeting us and our
> customers (looks like he hit a machine and found connections from others
> in their logs, went from there).
I have two boxen running connected to the internet, one is Debian Kernel Image
+ all latest available security fixes for debian, the other one is almost the
same but with 2.4.20-wolk4.1s enabled all grsecurity stuff.
Both machines are connected for a long time now, both on the same ip subnet
and I've announced a hackcontest privately to some people some time ago (the
machines intention is for hacking ;).
The first, debian kernel image machine, was hacked 37 times in 1 year, the
other one was hacked 0 times, looking into the logs I see _tons_ of "PaX:
from <IP> terminating $foobar".
So the way to go is absolutely grsecurity if you want to be very safe even
against exploits and security holes in userspace applications which are not
known yet.
> The part that bothers me is that all of these systems were updated to
> the newest versions on debian.security.org (if apt-get was doing its
> job) and firewalled down to just the ports we needed (22, 25, 53, 80).
what mailserver do you run on 25? what type of webserver (if so on port 80)
and what nameserver? Bind? ;)
> While I don't like this (OpenSSH is open and it should be that way), has
> anyone else had this kind of experience? Is there some big hack I
> should know about?
No public exploits are known for the most recent OpenSSH version v3.6.1p2,
which does _not_ mean there are no exploits.
> I've checked CERT and the SANS list. Both of them were helpful, but
> most of the answers said "run the newest version of X", which I have
> assumed apt-get fixed (in stable at least). I mean, some versions were
> older, but I had heard most of them had backported fixes. Is this
> happening to anyone else?
yes, with the machine/software packages w/o grsecurity/PaX support.
Personally I don't trust those so called "security updates". I always compile
relevant software for myself from the servers programs homepage.
Don't get me wrong. I don't say that the security updates are not safe. It is
just my personal choice of doing it on my own!!
--
ciao, Marc
J M Cerqueira Esteves
> We've had a number of hacked boxen recently. It appears a certain
> person (Romanian we think) is specifically targeting us and our
> customers (looks like he hit a machine and found connections from others
> in their logs, went from there).
> The part that bothers me is that all of these systems were updated to
> the newest versions on debian.security.org (if apt-get was doing its
> job) and firewalled down to just the ports we needed (22, 25, 53, 80).
Did those machines have kernels with the ptrace vulnerability?
Did they have interactive users logging in from less secure machines?
Recently, a friend of mine also had some compromised Debian machines,
attacked from Romania and elsewhere. His personal machine was
successfully attacked twice:
We didn't make a careful analysis, but it is possible that a Samba
vulnerability was exploited for the first attack, since he had not made
security updates for a few months.
After that he made a fresh install and took proper care of updates from
debian.security.org, but... after a few days his machine was compromised
again; this time we believe the attack was performed through another
machine, running some Mandrake release which was lacking security
updates. The main user of this Mandrake system is also a regular user
on my friend's machine and somehow the intruder has taken advantage
of that for reaching Debian as the "normal" user. The only known
vulnerability on my friend's machine at the time was... the kernel
with the ptrace vulnerability, which the intruder may have exploited
(in fact we found an exploit for that among the intruder's files in the
Mandrake machine).
Best regards
J Esteves
Bradley Alexander
service and _completely_ rebuild them. Fdisk and format the drives and
only run services which you want on them. The more extra stuff you put
in there, the more the chance of missing something. I would also
consider running iptables to make sure the ports you aren't watching are
blocked. (I have had excellent luck with gShield,
http://muse.linuxmafia.org). I have gShield set up to block most every
port, and have had a very very good experience with it.
Scan the boxes with nmap and nessus before putting them online (both
with and without the firewall rules in place). I gave a presentation on
it to my Linux user group called Reasonably Secure Builds, which is at
http://www.tux.org/~storm. The presentation is about 2 years old, but
still has some good information.
Specific port issues:
22 - There are no known exploits for OpenSSH to my knowledge. That said,
it is always possible that there is something in the latest version. To
be safe, if you can, avoid version 1 protocol at all costs. ssh v1 is
broken and vulnerable. Use only version 2 if you can.
25 - It is entirely possible this is how the attacker got in. If you can
avoid ftp (by using scp/sftp), do so. This will close 25% of your known
open ports. And anonymous ftp is especially vulnerable.
53 - BIND has been a source of vulnerabilities. Make sure you are
running the latest version of BIND. BIND 9.2.2 is probably fairly safe.
Avoid BIND 4 and if you have to use BIND 8, make sure it is the latest
(8.3.4). But if you are running BIND 8, you could probably upgrade to 9.
(this, of course, on the assumption that you are running BIND in the
first place. I can't speak to other name server packages.)
80 - make sure you are running the latest and greatest version of
Apache.
And remember, you can never prove you were _not_ 0wned, you can only
prove that you were.
HTH,
--Brad
============================================================================
Bradley M. Alexander |
gTLD SysAdmin, Security Engineer | storm [at] tux.org
Debian/GNU Linux Developer | storm [at] debian.org
============================================================================
Key fingerprints:
DSA 0x54434E65: 37F6 BCA6 621D 920C E02E E3C8 73B2 C019 5443 4E65
RSA 0xC3BCBA91: 3F 0E 26 C1 90 14 AD 0A C8 9C F0 93 75 A0 01 34
============================================================================
Of all of the things I have lost, I think I miss my mind the most.
Noah Meyerhans
> avoid ftp (by using scp/sftp), do so. This will close 25% of your known
> open ports. And anonymous ftp is especially vulnerable.
If you want to sound credible you should probably at least know what
listens on port 25. (Hint: it's not FTP)
David Ramsden
> > We've had a number of hacked boxen recently. It appears a certain
> > person (Romanian we think) is specifically targeting us and our
> > customers (looks like he hit a machine and found connections from others
> > in their logs, went from there).
> >
> > We have no idea how he's getting in, but we've got his rootkit fairly
> > nailed down (he uses a few slightly different ones).
> >
> > We've caught a few systems as he was breaking in (we have
> > .bash_history files and the site he downloads his rootkits from).
> >
> > The part that bothers me is that all of these systems were updated to
> > the newest versions on debian.security.org (if apt-get was doing its
> > job) and firewalled down to just the ports we needed (22, 25, 53, 80).
> >
I've found that when running a system were the users can put up their
web pages.. most insecure.
It's virtually impossible to know what each user is running under their
web space.. An exploitable version of PHPNuke for example, leading to
the web server privs. and from there, who knows.
So if you can't think of any service that may have been exploited due to
being up to date with security.debian.org maybe think about what users
are running under their webspace.
That's a bit of a stab in the dark but something I feel admins.
overlook (ntoe to self: look at running Apache in chroot jail :-p).
So maybe they gained access to a system via something like the above,
then found out a common username/password (root, for example) and is
able to login to the other machines via SSH - No need to exploit.
Some things to think about possibly.
Good luck!
David.
--
.''`. David Ramsden <da...@hexstream.eu.org>
: :' : http://portal.hexstream.eu.org/
`. `'` PGP key ID: 507B379B on wwwkeys.pgp.net
`- Debian - when you have better things to do than to fix a system.
Bradley Alexander
on the phone while replying, but thats no excuse).
Port 25 is email/smtp. For this, I would recommend postfix. I know
Debian ships with Exim, but for any configurations beyond basic email, I
have had abyssmal luck getting Exim to do it. I prefer Postfix over
Sendmail (or qmail or whatever), but this is my personal opinion.
On Sun, 2003-05-25 at 15:44, Noah Meyerhans wrote:
> > 25 - It is entirely possible this is how the attacker got in. If you can
> > avoid ftp (by using scp/sftp), do so. This will close 25% of your known
> > open ports. And anonymous ftp is especially vulnerable.
>
> If you want to sound credible you should probably at least know what
> listens on port 25. (Hint: it's not FTP)
>
> noah
--
--Brad
============================================================================
Bradley M. Alexander |
gTLD SysAdmin, Security Engineer | storm [at] tux.org
Debian/GNU Linux Developer | storm [at] debian.org
============================================================================
Key fingerprints:
DSA 0x54434E65: 37F6 BCA6 621D 920C E02E E3C8 73B2 C019 5443 4E65
RSA 0xC3BCBA91: 3F 0E 26 C1 90 14 AD 0A C8 9C F0 93 75 A0 01 34
============================================================================
Criminals love gun control - it makes their jobs safer.
David B Harris
Jayson Vantuyl <kag...@souja.net> wrote:
> We have no idea how he's getting in, but we've got his rootkit fairly
> nailed down (he uses a few slightly different ones).
Good god man! Include them in your post. There may be a new, unknown
vulnerability. Not to mention that people will be able to tell you
exactly what the rootkits do.
Javier Fernández-Sanguino Peña
> Sunday, May 25, 2003, 2:04:30 PM, Jayson Vantuyl (Jayson) wrote:
>
> Jayson> We've had a number of hacked boxen recently. It appears a certain
> Jayson> person (Romanian we think) is specifically targeting us and our
> Jayson> customers (looks like he hit a machine and found connections from others
> Jayson> in their logs, went from there).
>
> That's pretty unsettling..
>
> Have you tried running snort? If its a known vulnerability it should
> be able to pick it up (don't use Debian's.. it's very out of date).
Err...
packages.debian.org/snort
shows unstable/testing provide 2.0.0, which is quite recent. It can easily
be backported to stable.
> You might want to try scanning your boxes with nessus too (kind of
> unlikely that it would find anything, but... (don't use debian version
> again)).
Why not?
http://packages.debian.org/nessus
Shows unstable/testing provide 2.0.5 (2.0.6 is out and will be in the
archive soon). Backporting it is really easy and you can find backported
packages (for older versions) at http://people.debian.org/~jfs/nessus/
So, maybe you meant do not use versions in _stable_ (see #183524)
Regards
Javi
Javier Fernández-Sanguino Peña
(..)
>
> That's a bit of a stab in the dark but something I feel admins.
> overlook (ntoe to self: look at running Apache in chroot jail :-p).
Maybe this helps:
http://www.debian.org/doc/manuals/securing-debian-howto/ap-chroot-apache-env.en.html
Regards
Javi
Javier Fernández-Sanguino Peña
Maybe following the steps described in "Chapter 10 - After the compromise
(incident response)" [1] of the Securing Debian Manual is best.
I think he might get also good answers if he posts this information to the
security-incidents mailing list [2] (maybe with a cross-post to this list
too)
Regards
JAvi
[1]
http://www.debian.org/doc/manuals/securing-debian-howto/ch-after-compromise.en.html
[2]
http://securityfocus.com/archive/75
Jayson Vantuyl
> Here's one major thing to consider. If all of your servers within your
> network are nearly the same, security wise, then you should consider
> that ALL of them are hacked. Until you've rebuilt every single one with
> trustable sources, your network is not safe. While you may not realize
> it, this Evil entity could still be gathering information on all your
> new systems, right as you put them online, which would really suck.
That's the disturbing part. They aren't on the same network (unless you
count the Internet) and they are *NOT* the same. One wasn't running
Apache, all were running SSH, they were running different mail servers
(one postfix, two sendmail, a few courier-mta even!). The kernels range
from 2.4.17-2.4.20 (depending on maintenance contracts :). SSH is the
only commonality I can see and that disturbs me. I've got a pretty good
handle on this guy's rootkits.
He appears to modify the kernel image in memory via /dev/kmem (a
next-generation LKM attack). I've considered removing /dev/kmem (does
anything use it?) but I don't know about any side effects (and it
doesn't prevent him mknod'ing it). It appears he actually has some sort
of kernel-level TTY logger *AND* a kernel-hack to hide files and
processes. The only comfort in this is that some of our kernels are
apparently so exotic that his meddling crashes the machine during the
break-in (instead of leaving a more compromized system). In general,
all of the rootkits are the same flavor (and seem unrelated to the LKM
stuff).
He uses a number of rootkits, but they all seem to be littered with his
handle (Kapitan). At first I thought it was the guy who made the
rootkit, but later he appears to have customer configured it to e-mail
his e-mail address at yahoo (also includes kapitan). It's also obvious
that he's aspiring to script-kiddie-dom. Later hacks show progressively
more hacking of the same rootkit to strip off some other poor sap's name
and plaster his everywhere.
> If you're THAT infested, you NEED to clean house. Take a weekend, or a
> couple days, call in all the technical people who can build systems and
> order in for Pizza. Take the entire network offline and rebuild it.
> Until you can track all of the machine that are hacked or ensure that
> they are all, in fact, clean, you can assume no level of safety.
Uhhh, that's me. Trust me when I say I'm as technical as it gets (short
of the Gods like Linus). It's not a single machine, it's a whole bunch
of them. It's not a password problem either. He seems to have hacked
multiple of them within an hour of each other (his rootkit files aren't
very clever about covering up mtime). I just can't tell how he got in.
I've got some process accouting logs to go through, but they're ...
verbose.
> I hope that some folks will assist you in finding what hole has been
> exploited on your network, but as for right now, you need to seriously
> consider whether boxes that you think are clean, are in fact, clean.
Got a good handle on that. I was primarly trying to gauge if this is an
epidemic or something I've done. Right now, it looks like an Apache
hole (there are logs of odd Apache requests before the crack and a few
machines that weren't cracked show web hits from the other machines),
but that could be wrong.
This guy's methods are crude once he gets in (hell, the only
applications I've even seen are sniffers and an IRC relay: psybnc).
Doesn't seem much more than your standard punk, script-kiddie. But he's
got a *VERY* slick way of getting in. Not sure how...
Thanks,
Jayson
Jayson Vantuyl
> If you believe he'll be back, it might be worth it to set up a honeypot
> and a box running tcpdump and capturing all the traffic to honeypot.
> Set the honeypot up with the same services you run on your production
> machines, and make sure that no services at all (not even ssh) are
> runnign on the tcpdump system. At least that way, when the box gets
> cracked, you'll be able to see what ports the guy was talking to when he
> broke in. It also might be useful as evidence in case you ever decide
> to try and prosecute him.
machine recording a tcpdump over a serial console. I've also been
looking into how to use some LKM-style tricks (read source mods) to hide
the sniffing process and a userspace solution to hide the promiscuous
mode IF.
> I assume the cracked boxes were running woody? What are the actual
> services running on the various open ports? What versions of the
> packages?
the outside):
ssh
smtp
domain
www
pop3s
imaps
>From the inside add:
netbios-ssn/netbios-dgm/netbios-ns
imap
pop3
This has been a hit on about seven different machines with vastly
different configurations (some missing everything but SSH) and all
firewalled down to the minimum.
> I don't know of any exploits for the version of OpenSSH included on
> security.debian.org for woody. It would certainly be interesting to
> find out that there is one in the wild!
Indeed. When I find out more I'll send to the list.
Jayson
Jayson Vantuyl
>
> Jayson> We've had a number of hacked boxen recently. It appears a certain
> Jayson> person (Romanian we think) is specifically targeting us and our
> Jayson> customers (looks like he hit a machine and found connections from others
> Jayson> in their logs, went from there).
>
> That's pretty unsettling..
town (~150k). While that sounds like a lot, we're in backwoods
Missouri--where the businesses are low-tech and the owners are really,
really cheap (average income per capita is like $15k/yr with 80% of the
town living paycheck to paycheck). That makes the computer community
pitifully small--and widespread Linux-related badness overly ugly.
It's got us spooked enough to get law-enforcement involved.
> Have you tried running snort? If its a known vulnerability it should
> be able to pick it up (don't use Debian's.. it's very out of date).
> You might want to try scanning your boxes with nessus too (kind of
> unlikely that it would find anything, but... (don't use debian version
> again)).
Not yet. I'll try a non-packaged version.
> Have all of the hacked boxes been running a while without a reboot?
> Someone discussed that programs running from updated libraries
> would still be vulnerable until they were restarted. For instance, if
> you havn't restarted ssh or apache (if you're using ssl) since openssl
> was upgraded, an openssl exploit would still work.
Most were around 30 days (although I shed a tear rebooting the two that
were over 400 days!).
Jayson
Jayson Vantuyl
> web pages.. most insecure.
> It's virtually impossible to know what each user is running under their
> web space.. An exploitable version of PHPNuke for example, leading to
> the web server privs. and from there, who knows.
nothing but Firewall, Mail, and Non-User webservice (standard
ultra-small business, all-eggs-in-one-basket setup, we advise against it
but provide it if asked).
> So if you can't think of any service that may have been exploited due to
> being up to date with security.debian.org maybe think about what users
> are running under their webspace.
Acknowledged, though.
> That's a bit of a stab in the dark but something I feel admins.
> overlook (ntoe to self: look at running Apache in chroot jail :-p).
> So maybe they gained access to a system via something like the above,
> then found out a common username/password (root, for example) and is
> able to login to the other machines via SSH - No need to exploit.
Thankfully, we don't have root passwords. In our space, we find root to
more of a concept than a user, so we disable the password and set up a
group that can su to root. That way we have a good handle on things.
Root never logs in, so we know somethings up if we see that. Also, if
it is hacked from a su-able user, we get a log of that too. I highly
recommend this set up (although it wasn't enough in our case :).
> Some things to think about possibly.
> Good luck!
Thanks.
Jayson
Marcel Weber
> This has been a hit on about seven different machines with vastly
> different configurations (some missing everything but SSH) and all
> firewalled down to the minimum.
>
I did not reread the whole thread, so sorry if I'm asking silly
questions, but perhaps it's not a security issue, but a policy issue:
- Have you ever checked your password policies? Are there weak passwords
around that the hacker might use to log in? Or has he / she in any way
managed to get a password in some way?
- Have all passwords of user accounts been changed since the break ins?
- You say, that you're running imap on the server. Can the imap users
log onto the machine or are these accounts completely seperated from the
system accounts? If no, it might be, that the hacker is sniffing the
imap passwords and using them to log onto your machines.
And last but not least: Is the firewall seperated from the servers or
running ON the servers. It's a good advice to lock down the machines
locally using iptables, but I think that doesn't save you a dedicated
firewall. Might be a Debian GNU/Linux or BSD box or even something
commercial.
Regards
Marcel
Olaf Dietsche
> Thankfully, we don't have root passwords. In our space, we find root to
> more of a concept than a user, so we disable the password and set up a
> group that can su to root. That way we have a good handle on things.
> Root never logs in, so we know somethings up if we see that. Also, if
> it is hacked from a su-able user, we get a log of that too. I highly
> recommend this set up (although it wasn't enough in our case :).
Just curious, how do you su to root, if root's password is disabled?
Do you have a modified su replacement?
Regards, Olaf.
Jason Lunz
> of them. It's not a password problem either. He seems to have hacked
> multiple of them within an hour of each other (his rootkit files
> aren't very clever about covering up mtime). I just can't tell how he
> got in.
Maybe he didn't use the same method for all of them. With the tty
sniffer, he could have sniffed passwords from the first box he cracked
if he was lucky enough to catch an admin su'ing. Do the timestamps
support that theory? (This is why ssh keys are good -- no secret of any
kind ever exists on the server, so even if it's compromised the attacker
can't sniff a password or secret key and use that to get into other
machines).
Also, how many people ssh into these machines? He could have control of
the desktop machine of someone who has user access, and then use local
holes to gain root once logged in as that user.
Jason
Noah Meyerhans
> Just curious, how do you su to root, if root's password is disabled?
> Do you have a modified su replacement?
su uses PAM. So it doesn't need to use root entry in /etc/passwd. It
could do something insane like consult a RADIUS server or something.
Whatever you can dream up, really.
noah
Jayson Vantuyl
> Maybe he didn't use the same method for all of them. With the tty
> sniffer, he could have sniffed passwords from the first box he cracked
> if he was lucky enough to catch an admin su'ing. Do the timestamps
> support that theory? (This is why ssh keys are good -- no secret of any
> kind ever exists on the server, so even if it's compromised the attacker
> can't sniff a password or secret key and use that to get into other
> machines).
involved.
Question: Can one use a key *AND* a password? That would make me
really happy. I just don't like getting ahold of a file or a password
being enough...
> Also, how many people ssh into these machines? He could have control of
> the desktop machine of someone who has user access, and then use local
> holes to gain root once logged in as that user.
Server machines, no real desktop users. One of these was a firewall
that pretty much only had SSH listening. *IF* it was hacked directly
(rather than being compromised with a sniff'd password), then we've got
something to target. The timestamps don't support much of anything,
since we don't really have many logs left (he's stupid, but not
st00p1d). Our logging infrastructure is . . . improving. Also, we're
implementing grsecurity. I've been very impressed so far (and suspect
2.0 will be even better when it's stable).
Jayson
Jayson Vantuyl
> Do you have a modified su replacement?
If you examine /etc/pam.d/su there should be a commented line using
pam_wheel.so. There are two there, one only allows su from a group, the
other allows that group to su with no password. I use the latter
extensively.
Jayson
Jayson Vantuyl
> I did not reread the whole thread, so sorry if I'm asking silly
> questions, but perhaps it's not a security issue, but a policy issue:
mentioned completely apply. Officially we are consultants. We have a
policy of "we tell you what not to do, then you tell us to do it". It's
often quite ironic. In all of these situations we had mentioned that
there were drawbacks to lax security. After so many arguements of "no
one would want to hack us", and "it makes my job harder" we just say
"duly noted, our hands are clean". We are billing by the hour for the
clean up. I've almost begun to get a perverse satisfaction out of it,
although it's really cutting into my time on real projects.
The only reason I posted is that this guy seemed to take seven different
and varied machines in a very short time frame--with very little skill
evident in his methods. They are all fairly distant (both in IP range
and administrative oversight). It's just odd to me that so many went so
fast, so I thought I'd chime in to make sure that this wasn't
widespread.
Jayson
Peter Solodov
> On Wed, May 28, 2003 at 02:06:21PM +0200, Olaf Dietsche wrote:
>> Just curious, how do you su to root, if root's password is
>> disabled? Do you have a modified su replacement?
>
> One of the few really nice things to come out of RedHat is PAM.
It's probably a good idea to mention that PAM came out of Sun :-)
- Peter
--
Peter Solodov | Concordia University
http://alcor.concordia.ca/~peter | Montreal, QC, Canada
Jason Lunz
> Question: Can one use a key *AND* a password? That would make me
> really happy. I just don't like getting ahold of a file or a password
> being enough...
That's how it's done, by default. The key is encrypted on-disk, and is
only decrypted when it's used. Just getting the key doesn't do an
attacker any good without its password.
The advantage here is that all secrets are client-side. The _local_ ssh
process reads the keyfile and the password, uses the decrypted key to
decrypt a randomly-generated challenge sent by the server, and forgets
the key again. The server gets proof that the client knows the key
without ever having access to any secret, even momentarily.
So if you're using keys properly, only each admin's personal workstation
ever carries secrets. Using sudo or su, every machine involved is
vulnerable to sniffing at some point.
> Server machines, no real desktop users. One of these was a firewall
> that pretty much only had SSH listening. *IF* it was hacked directly
> (rather than being compromised with a sniff'd password), then we've got
> something to target.
Was it patched against the recent kernel root hole? On any unpatched
2.4.20 (or older) linux machine, user access is pretty much trivially
equivalent to root.
Jason
Michael Rowe
> Question: Can one use a key *AND* a password? That would make me
> really happy. I just don't like getting ahold of a file or a password
> being enough...
You can put a passphrase on the key, if that's what you mean:
ssh-keygen -p
You then have to enter the passphrase to use the key, so an attacker
getting hold of the key file can't use it without the passphrase. Of
course, this can still be sniffed if they are on the box.
--
Michael Rowe <mr...@mojain.com>
IM - mr...@jabber.org Prof - ACM, IEEE, Computer Soc.
Web - http://www.mojain.com/ Vice - Barley malt, brewed or
Key - http://mojain.com/keys/mrowe.asc distilled (hold the ice)
Marc F. Neininger
> Server machines, no real desktop users. One of these was a firewall
> that pretty much only had SSH listening. *IF* it was hacked directly
> (rather than being compromised with a sniff'd password), then we've got
> something to target. The timestamps don't support much of anything,
> since we don't really have many logs left (he's stupid, but not
> st00p1d). Our logging infrastructure is . . . improving. Also, we're
> implementing grsecurity. I've been very impressed so far (and suspect
> 2.0 will be even better when it's stable).
How about an iptables skript that writes its logs to another machine?
And at the same time the skript should filter traffic from one server to
another...
Another point I would check is if all your _desktop_ computers are
yours. He might have hacked one just to get better access to your server
machines.
Just my 2€
Marc
Jayson Vantuyl
> On Thu, 29 May 2003, Jayson Vantuyl wrote:
> > On Wed, May 28, 2003 at 02:06:21PM +0200, Olaf Dietsche wrote:
> >> Just curious, how do you su to root, if root's password is
> >> disabled? Do you have a modified su replacement?
> >
> > One of the few really nice things to come out of RedHat is PAM.
>
> It's probably a good idea to mention that PAM came out of Sun :-)
>
> - Peter
is making it a reality on Linux.
J
Phillip Hofmeister
> He appears to modify the kernel image in memory via /dev/kmem (a
> next-generation LKM attack). I've considered removing /dev/kmem (does
> anything use it?) but I don't know about any side effects (and it
> doesn't prevent him mknod'ing it). It appears he actually has some sort
> of kernel-level TTY logger *AND* a kernel-hack to hide files and
> processes. The only comfort in this is that some of our kernels are
> apparently so exotic that his meddling crashes the machine during the
> break-in (instead of leaving a more compromized system). In general,
> all of the rootkits are the same flavor (and seem unrelated to the LKM
> stuff).
Assuming he has rooted the box removing /dev/kmem won't do any good as
he can merely recreate it using mknod(1).
--
Phillip Hofmeister
PGP/GPG Key:
http://www.zionlth.org/~plhofmei/
wget -O - http://www.zionlth.org/~plhofmei/key.txt | gpg --import
--
Excuse #163: RPC_PMAP_FAILURE
Adam Majer
> He appears to modify the kernel image in memory via /dev/kmem (a
> next-generation LKM attack). I've considered removing /dev/kmem (does
> anything use it?) but I don't know about any side effects (and it
> doesn't prevent him mknod'ing it). It appears he actually has some sort
(link below helps here :)
> of kernel-level TTY logger *AND* a kernel-hack to hide files and
> processes. The only comfort in this is that some of our kernels are
> apparently so exotic that his meddling crashes the machine during the
> break-in (instead of leaving a more compromized system). In general,
> all of the rootkits are the same flavor (and seem unrelated to the LKM
> stuff).
> Uhhh, that's me. Trust me when I say I'm as technical as it gets (short
> of the Gods like Linus). It's not a single machine, it's a whole bunch
> of them. It's not a password problem either. He seems to have hacked
> multiple of them within an hour of each other (his rootkit files aren't
> very clever about covering up mtime). I just can't tell how he got in.
> I've got some process accouting logs to go through, but they're ...
> verbose.
There are many ways of getting into the system. I was running the same
thing and got cracked. Now, grsecurity is the only way to fly :)
Go there, DL the patches and _use_ them. It's a real pain to set up
really tight ACLs, but then even root cannot do anything. So cracking
SSH or something might get him access to the box, but not access to
the entire machine.
Futhermore, if you need to run SSH, don't allow everyone to access
it - only allow access to it from known IP ranges. Why? Because
sshd has to be run SETUID and SETGID. Even with grsecurity this allows
the attacker to delete user data.
With something like sendmail or apache, it only needs to see a very
limited part of the file system, so even braking these will not do
any real damage.
Also, grsecurity has stack randomization, etc... so buffer overflows
are a guessing game at best.
- Adam
PS. Any exploits against grsecurity? :)
Phillip Hofmeister
> With something like sendmail or apache, it only needs to see a very
> limited part of the file system, so even braking these will not do
> any real damage.
Don't get too over confident about chrooting Apache. One Apache process
runs as root. This means if there is an exploit that sends arbitrary
code across the shared scoreboard it could be ran as root and break out
of the jail.
However, for the most part, chrooting is a valid countermeasure/method
to compartmentalize. It is a shame that no distribution comes with
packages natively created with/for chrooting.
--
Phillip Hofmeister
PGP/GPG Key:
http://www.zionlth.org/~plhofmei/
wget -O - http://www.zionlth.org/~plhofmei/key.txt | gpg --import
--
Excuse #134: Backbone Scoliosis
Mark Ferlatte
> However, for the most part, chrooting is a valid countermeasure/method
> to compartmentalize. It is a shame that no distribution comes with
> packages natively created with/for chrooting.
Also, Debian's Bind 9 package is pretty trivial to chroot (although it doesn't
by default). Debian's postfix package does chroot by default, although you
tend to have to turn it off if you want to use things like postfix-tls or SASL.
M
Adam Majer
> On Mon, 02 Jun 2003 at 03:38:21PM -0500, Adam Majer wrote:
> > With something like sendmail or apache, it only needs to see a very
> > limited part of the file system, so even braking these will not do
> > any real damage.
>
> Don't get too over confident about chrooting Apache. One Apache process
> runs as root. This means if there is an exploit that sends arbitrary
> code across the shared scoreboard it could be ran as root and break out
> of the jail.
First, sorry for my very late reply :) I'm just reading the
messages here now...
Anyway, I wasn't talking about chroot. I was talking about
grsecurity and ACLs (I think). Then you specify what each
process is allowed to do and see (even root cannot get passed that).
You can make Apache see only the directories that you want it
to see. You can also specify that Apache cannot initiate a connection
(except to trusted nameserver for instance) and it can only listen on port 80.
With other features of grsecurity like stack randomization, Apache
becomes pretty much explot-proof...
- Adam
Javier Fernández-Sanguino Peña
> Phillip Hofmeister said on Tue, Jun 03, 2003 at 10:02:09AM -0400:
> > However, for the most part, chrooting is a valid countermeasure/method
> > to compartmentalize. It is a shame that no distribution comes with
> > packages natively created with/for chrooting.
>
> I believe that OpenBSD does.
>
Yes it does. Although I don't believe that the way to go is chrooting since
it makes it very difficult to ease upgrades.
> Also, Debian's Bind 9 package is pretty trivial to chroot (although it doesn't
> by default). Debian's postfix package does chroot by default, although you
> tend to have to turn it off if you want to use things like postfix-tls or SASL.
There are a number of patches in the BTS to make bind work in a chroot
environment out of the box, using bind's own chroot functionality. In any
case, there are also a number of packages to provide an easy way to setup
chroot/restricted environments (makejail and compartment come to mind).
In any case I don't think that chrooting is the way to go here, it was
built to be used as a testing/programing tool, not really a security tool.
There are number of (Linux) patches to provide full compartimentalization
of processes in the system which might be the way to go. Just my 2c.
Regards
Javi