We have no idea how he's getting in, but we've got his rootkit fairly
nailed down (he uses a few slightly different ones).
We've caught a few systems as he was breaking in (we have
.bash_history files and the site he downloads his rootkits from).
The part that bothers me is that all of these systems were updated to
the newest versions on debian.security.org (if apt-get was doing its
job) and firewalled down to just the ports we needed (22, 25, 53, 80).
My boss is thinking they might have some sort of crack for OpenSSH (only
service I can say all of these have in common) and he's considering
trying a switch to the nonfree one just to see if it helps.
While I don't like this (OpenSSH is open and it should be that way), has
anyone else had this kind of experience? Is there some big hack I
should know about?
I've checked CERT and the SANS list. Both of them were helpful, but
most of the answers said "run the newest version of X", which I have
assumed apt-get fixed (in stable at least). I mean, some versions were
older, but I had heard most of them had backported fixes. Is this
happening to anyone else?
I'm at a complete loss as to how to explain this one, help would be
appreciated. The only comforting thought is that I can't imaging Redhat
would have done any better.
Jayson Vantuyl
Computing Edge, Inc.
--
To UNSUBSCRIBE, email to debian-secu...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
If you believe he'll be back, it might be worth it to set up a honeypot
and a box running tcpdump and capturing all the traffic to honeypot.
Set the honeypot up with the same services you run on your production
machines, and make sure that no services at all (not even ssh) are
runnign on the tcpdump system. At least that way, when the box gets
cracked, you'll be able to see what ports the guy was talking to when he
broke in. It also might be useful as evidence in case you ever decide
to try and prosecute him.
I assume the cracked boxes were running woody? What are the actual
services running on the various open ports? What versions of the
packages?
I don't know of any exploits for the version of OpenSSH included on
security.debian.org for woody. It would certainly be interesting to
find out that there is one in the wild!
noah
--
_______________________________________________________
| Web: http://web.morgul.net/~frodo/
| PGP Public Key: http://web.morgul.net/~frodo/mail.html
Jayson> We've had a number of hacked boxen recently. It appears a certain
Jayson> person (Romanian we think) is specifically targeting us and our
Jayson> customers (looks like he hit a machine and found connections from others
Jayson> in their logs, went from there).
That's pretty unsettling..
Have you tried running snort? If its a known vulnerability it should
be able to pick it up (don't use Debian's.. it's very out of date).
You might want to try scanning your boxes with nessus too (kind of
unlikely that it would find anything, but... (don't use debian version
again)).
Have all of the hacked boxes been running a while without a reboot?
Someone discussed that programs running from updated libraries
would still be vulnerable until they were restarted. For instance, if
you havn't restarted ssh or apache (if you're using ssl) since openssl
was upgraded, an openssl exploit would still work.
------------------------------------------------------
| Eddie J Schwartz <EdMcMan@[despammed.com|m00.net]> |
| AIM: Uncaring Eyes ICQ: 35576339 YHOO: edmcman2 |
| "We Trills have an expression -- at forty, you |
| think you know everything. At four hundred you |
| realize you know nothing." - Dax, Startrek DS9 |
------------------------------------------------------
Hi Jayson,
> We've had a number of hacked boxen recently. It appears a certain
> person (Romanian we think) is specifically targeting us and our
> customers (looks like he hit a machine and found connections from others
> in their logs, went from there).
I have two boxen running connected to the internet, one is Debian Kernel Image
+ all latest available security fixes for debian, the other one is almost the
same but with 2.4.20-wolk4.1s enabled all grsecurity stuff.
Both machines are connected for a long time now, both on the same ip subnet
and I've announced a hackcontest privately to some people some time ago (the
machines intention is for hacking ;).
The first, debian kernel image machine, was hacked 37 times in 1 year, the
other one was hacked 0 times, looking into the logs I see _tons_ of "PaX:
from <IP> terminating $foobar".
So the way to go is absolutely grsecurity if you want to be very safe even
against exploits and security holes in userspace applications which are not
known yet.
> The part that bothers me is that all of these systems were updated to
> the newest versions on debian.security.org (if apt-get was doing its
> job) and firewalled down to just the ports we needed (22, 25, 53, 80).
what mailserver do you run on 25? what type of webserver (if so on port 80)
and what nameserver? Bind? ;)
> While I don't like this (OpenSSH is open and it should be that way), has
> anyone else had this kind of experience? Is there some big hack I
> should know about?
No public exploits are known for the most recent OpenSSH version v3.6.1p2,
which does _not_ mean there are no exploits.
> I've checked CERT and the SANS list. Both of them were helpful, but
> most of the answers said "run the newest version of X", which I have
> assumed apt-get fixed (in stable at least). I mean, some versions were
> older, but I had heard most of them had backported fixes. Is this
> happening to anyone else?
yes, with the machine/software packages w/o grsecurity/PaX support.
Personally I don't trust those so called "security updates". I always compile
relevant software for myself from the servers programs homepage.
Don't get me wrong. I don't say that the security updates are not safe. It is
just my personal choice of doing it on my own!!
--
ciao, Marc
> The part that bothers me is that all of these systems were updated to
> the newest versions on debian.security.org (if apt-get was doing its
> job) and firewalled down to just the ports we needed (22, 25, 53, 80).
Did those machines have kernels with the ptrace vulnerability?
Did they have interactive users logging in from less secure machines?
Recently, a friend of mine also had some compromised Debian machines,
attacked from Romania and elsewhere. His personal machine was
successfully attacked twice:
We didn't make a careful analysis, but it is possible that a Samba
vulnerability was exploited for the first attack, since he had not made
security updates for a few months.
After that he made a fresh install and took proper care of updates from
debian.security.org, but... after a few days his machine was compromised
again; this time we believe the attack was performed through another
machine, running some Mandrake release which was lacking security
updates. The main user of this Mandrake system is also a regular user
on my friend's machine and somehow the intruder has taken advantage
of that for reaching Debian as the "normal" user. The only known
vulnerability on my friend's machine at the time was... the kernel
with the ptrace vulnerability, which the intruder may have exploited
(in fact we found an exploit for that among the intruder's files in the
Mandrake machine).
Best regards
J Esteves
Scan the boxes with nmap and nessus before putting them online (both
with and without the firewall rules in place). I gave a presentation on
it to my Linux user group called Reasonably Secure Builds, which is at
http://www.tux.org/~storm. The presentation is about 2 years old, but
still has some good information.
Specific port issues:
22 - There are no known exploits for OpenSSH to my knowledge. That said,
it is always possible that there is something in the latest version. To
be safe, if you can, avoid version 1 protocol at all costs. ssh v1 is
broken and vulnerable. Use only version 2 if you can.
25 - It is entirely possible this is how the attacker got in. If you can
avoid ftp (by using scp/sftp), do so. This will close 25% of your known
open ports. And anonymous ftp is especially vulnerable.
53 - BIND has been a source of vulnerabilities. Make sure you are
running the latest version of BIND. BIND 9.2.2 is probably fairly safe.
Avoid BIND 4 and if you have to use BIND 8, make sure it is the latest
(8.3.4). But if you are running BIND 8, you could probably upgrade to 9.
(this, of course, on the assumption that you are running BIND in the
first place. I can't speak to other name server packages.)
80 - make sure you are running the latest and greatest version of
Apache.
And remember, you can never prove you were _not_ 0wned, you can only
prove that you were.
HTH,
--Brad
============================================================================
Bradley M. Alexander |
gTLD SysAdmin, Security Engineer | storm [at] tux.org
Debian/GNU Linux Developer | storm [at] debian.org
============================================================================
Key fingerprints:
DSA 0x54434E65: 37F6 BCA6 621D 920C E02E E3C8 73B2 C019 5443 4E65
RSA 0xC3BCBA91: 3F 0E 26 C1 90 14 AD 0A C8 9C F0 93 75 A0 01 34
============================================================================
Of all of the things I have lost, I think I miss my mind the most.
If you want to sound credible you should probably at least know what
listens on port 25. (Hint: it's not FTP)
I've found that when running a system were the users can put up their
web pages.. most insecure.
It's virtually impossible to know what each user is running under their
web space.. An exploitable version of PHPNuke for example, leading to
the web server privs. and from there, who knows.
So if you can't think of any service that may have been exploited due to
being up to date with security.debian.org maybe think about what users
are running under their webspace.
That's a bit of a stab in the dark but something I feel admins.
overlook (ntoe to self: look at running Apache in chroot jail :-p).
So maybe they gained access to a system via something like the above,
then found out a common username/password (root, for example) and is
able to login to the other machines via SSH - No need to exploit.
Some things to think about possibly.
Good luck!
David.
--
.''`. David Ramsden <da...@hexstream.eu.org>
: :' : http://portal.hexstream.eu.org/
`. `'` PGP key ID: 507B379B on wwwkeys.pgp.net
`- Debian - when you have better things to do than to fix a system.
Port 25 is email/smtp. For this, I would recommend postfix. I know
Debian ships with Exim, but for any configurations beyond basic email, I
have had abyssmal luck getting Exim to do it. I prefer Postfix over
Sendmail (or qmail or whatever), but this is my personal opinion.
On Sun, 2003-05-25 at 15:44, Noah Meyerhans wrote:
> > 25 - It is entirely possible this is how the attacker got in. If you can
> > avoid ftp (by using scp/sftp), do so. This will close 25% of your known
> > open ports. And anonymous ftp is especially vulnerable.
>
> If you want to sound credible you should probably at least know what
> listens on port 25. (Hint: it's not FTP)
>
> noah
--
--Brad
============================================================================
Bradley M. Alexander |
gTLD SysAdmin, Security Engineer | storm [at] tux.org
Debian/GNU Linux Developer | storm [at] debian.org
============================================================================
Key fingerprints:
DSA 0x54434E65: 37F6 BCA6 621D 920C E02E E3C8 73B2 C019 5443 4E65
RSA 0xC3BCBA91: 3F 0E 26 C1 90 14 AD 0A C8 9C F0 93 75 A0 01 34
============================================================================
Criminals love gun control - it makes their jobs safer.
Good god man! Include them in your post. There may be a new, unknown
vulnerability. Not to mention that people will be able to tell you
exactly what the rootkits do.
Err...
packages.debian.org/snort
shows unstable/testing provide 2.0.0, which is quite recent. It can easily
be backported to stable.
> You might want to try scanning your boxes with nessus too (kind of
> unlikely that it would find anything, but... (don't use debian version
> again)).
Why not?
http://packages.debian.org/nessus
Shows unstable/testing provide 2.0.5 (2.0.6 is out and will be in the
archive soon). Backporting it is really easy and you can find backported
packages (for older versions) at http://people.debian.org/~jfs/nessus/
So, maybe you meant do not use versions in _stable_ (see #183524)
Regards
Javi
Maybe this helps:
http://www.debian.org/doc/manuals/securing-debian-howto/ap-chroot-apache-env.en.html
Regards
Javi
Maybe following the steps described in "Chapter 10 - After the compromise
(incident response)" [1] of the Securing Debian Manual is best.
I think he might get also good answers if he posts this information to the
security-incidents mailing list [2] (maybe with a cross-post to this list
too)
Regards
JAvi
[1]
http://www.debian.org/doc/manuals/securing-debian-howto/ch-after-compromise.en.html
[2]
http://securityfocus.com/archive/75
He appears to modify the kernel image in memory via /dev/kmem (a
next-generation LKM attack). I've considered removing /dev/kmem (does
anything use it?) but I don't know about any side effects (and it
doesn't prevent him mknod'ing it). It appears he actually has some sort
of kernel-level TTY logger *AND* a kernel-hack to hide files and
processes. The only comfort in this is that some of our kernels are
apparently so exotic that his meddling crashes the machine during the
break-in (instead of leaving a more compromized system). In general,
all of the rootkits are the same flavor (and seem unrelated to the LKM
stuff).
He uses a number of rootkits, but they all seem to be littered with his
handle (Kapitan). At first I thought it was the guy who made the
rootkit, but later he appears to have customer configured it to e-mail
his e-mail address at yahoo (also includes kapitan). It's also obvious
that he's aspiring to script-kiddie-dom. Later hacks show progressively
more hacking of the same rootkit to strip off some other poor sap's name
and plaster his everywhere.
> If you're THAT infested, you NEED to clean house. Take a weekend, or a
> couple days, call in all the technical people who can build systems and
> order in for Pizza. Take the entire network offline and rebuild it.
> Until you can track all of the machine that are hacked or ensure that
> they are all, in fact, clean, you can assume no level of safety.
Uhhh, that's me. Trust me when I say I'm as technical as it gets (short
of the Gods like Linus). It's not a single machine, it's a whole bunch
of them. It's not a password problem either. He seems to have hacked
multiple of them within an hour of each other (his rootkit files aren't
very clever about covering up mtime). I just can't tell how he got in.
I've got some process accouting logs to go through, but they're ...
verbose.
> I hope that some folks will assist you in finding what hole has been
> exploited on your network, but as for right now, you need to seriously
> consider whether boxes that you think are clean, are in fact, clean.
Got a good handle on that. I was primarly trying to gauge if this is an
epidemic or something I've done. Right now, it looks like an Apache
hole (there are logs of odd Apache requests before the crack and a few
machines that weren't cracked show web hits from the other machines),
but that could be wrong.
This guy's methods are crude once he gets in (hell, the only
applications I've even seen are sniffers and an IRC relay: psybnc).
Doesn't seem much more than your standard punk, script-kiddie. But he's
got a *VERY* slick way of getting in. Not sure how...
Thanks,
Jayson
ssh
smtp
domain
www
pop3s
imaps
>From the inside add:
netbios-ssn/netbios-dgm/netbios-ns
imap
pop3
This has been a hit on about seven different machines with vastly
different configurations (some missing everything but SSH) and all
firewalled down to the minimum.
> I don't know of any exploits for the version of OpenSSH included on
> security.debian.org for woody. It would certainly be interesting to
> find out that there is one in the wild!
Indeed. When I find out more I'll send to the list.
Jayson
> Have you tried running snort? If its a known vulnerability it should
> be able to pick it up (don't use Debian's.. it's very out of date).
> You might want to try scanning your boxes with nessus too (kind of
> unlikely that it would find anything, but... (don't use debian version
> again)).
Not yet. I'll try a non-packaged version.
> Have all of the hacked boxes been running a while without a reboot?
> Someone discussed that programs running from updated libraries
> would still be vulnerable until they were restarted. For instance, if
> you havn't restarted ssh or apache (if you're using ssl) since openssl
> was upgraded, an openssl exploit would still work.
Most were around 30 days (although I shed a tear rebooting the two that
were over 400 days!).
Jayson
> So if you can't think of any service that may have been exploited due to
> being up to date with security.debian.org maybe think about what users
> are running under their webspace.
Acknowledged, though.
> That's a bit of a stab in the dark but something I feel admins.
> overlook (ntoe to self: look at running Apache in chroot jail :-p).
> So maybe they gained access to a system via something like the above,
> then found out a common username/password (root, for example) and is
> able to login to the other machines via SSH - No need to exploit.
Thankfully, we don't have root passwords. In our space, we find root to
more of a concept than a user, so we disable the password and set up a
group that can su to root. That way we have a good handle on things.
Root never logs in, so we know somethings up if we see that. Also, if
it is hacked from a su-able user, we get a log of that too. I highly
recommend this set up (although it wasn't enough in our case :).
> Some things to think about possibly.
> Good luck!
Thanks.
Jayson
> This has been a hit on about seven different machines with vastly
> different configurations (some missing everything but SSH) and all
> firewalled down to the minimum.
>
I did not reread the whole thread, so sorry if I'm asking silly
questions, but perhaps it's not a security issue, but a policy issue:
- Have you ever checked your password policies? Are there weak passwords
around that the hacker might use to log in? Or has he / she in any way
managed to get a password in some way?
- Have all passwords of user accounts been changed since the break ins?
- You say, that you're running imap on the server. Can the imap users
log onto the machine or are these accounts completely seperated from the
system accounts? If no, it might be, that the hacker is sniffing the
imap passwords and using them to log onto your machines.
And last but not least: Is the firewall seperated from the servers or
running ON the servers. It's a good advice to lock down the machines
locally using iptables, but I think that doesn't save you a dedicated
firewall. Might be a Debian GNU/Linux or BSD box or even something
commercial.
Regards
Marcel
> Thankfully, we don't have root passwords. In our space, we find root to
> more of a concept than a user, so we disable the password and set up a
> group that can su to root. That way we have a good handle on things.
> Root never logs in, so we know somethings up if we see that. Also, if
> it is hacked from a su-able user, we get a log of that too. I highly
> recommend this set up (although it wasn't enough in our case :).
Just curious, how do you su to root, if root's password is disabled?
Do you have a modified su replacement?
Regards, Olaf.
Maybe he didn't use the same method for all of them. With the tty
sniffer, he could have sniffed passwords from the first box he cracked
if he was lucky enough to catch an admin su'ing. Do the timestamps
support that theory? (This is why ssh keys are good -- no secret of any
kind ever exists on the server, so even if it's compromised the attacker
can't sniff a password or secret key and use that to get into other
machines).
Also, how many people ssh into these machines? He could have control of
the desktop machine of someone who has user access, and then use local
holes to gain root once logged in as that user.
Jason
su uses PAM. So it doesn't need to use root entry in /etc/passwd. It
could do something insane like consult a RADIUS server or something.
Whatever you can dream up, really.
noah
Question: Can one use a key *AND* a password? That would make me
really happy. I just don't like getting ahold of a file or a password
being enough...
> Also, how many people ssh into these machines? He could have control of
> the desktop machine of someone who has user access, and then use local
> holes to gain root once logged in as that user.
Server machines, no real desktop users. One of these was a firewall
that pretty much only had SSH listening. *IF* it was hacked directly
(rather than being compromised with a sniff'd password), then we've got
something to target. The timestamps don't support much of anything,
since we don't really have many logs left (he's stupid, but not
st00p1d). Our logging infrastructure is . . . improving. Also, we're
implementing grsecurity. I've been very impressed so far (and suspect
2.0 will be even better when it's stable).
Jayson
If you examine /etc/pam.d/su there should be a commented line using
pam_wheel.so. There are two there, one only allows su from a group, the
other allows that group to su with no password. I use the latter
extensively.
Jayson
The only reason I posted is that this guy seemed to take seven different
and varied machines in a very short time frame--with very little skill
evident in his methods. They are all fairly distant (both in IP range
and administrative oversight). It's just odd to me that so many went so
fast, so I thought I'd chime in to make sure that this wasn't
widespread.
Jayson
It's probably a good idea to mention that PAM came out of Sun :-)
- Peter
--
Peter Solodov | Concordia University
http://alcor.concordia.ca/~peter | Montreal, QC, Canada
That's how it's done, by default. The key is encrypted on-disk, and is
only decrypted when it's used. Just getting the key doesn't do an
attacker any good without its password.
The advantage here is that all secrets are client-side. The _local_ ssh
process reads the keyfile and the password, uses the decrypted key to
decrypt a randomly-generated challenge sent by the server, and forgets
the key again. The server gets proof that the client knows the key
without ever having access to any secret, even momentarily.
So if you're using keys properly, only each admin's personal workstation
ever carries secrets. Using sudo or su, every machine involved is
vulnerable to sniffing at some point.
> Server machines, no real desktop users. One of these was a firewall
> that pretty much only had SSH listening. *IF* it was hacked directly
> (rather than being compromised with a sniff'd password), then we've got
> something to target.
Was it patched against the recent kernel root hole? On any unpatched
2.4.20 (or older) linux machine, user access is pretty much trivially
equivalent to root.
Jason
You can put a passphrase on the key, if that's what you mean:
ssh-keygen -p
You then have to enter the passphrase to use the key, so an attacker
getting hold of the key file can't use it without the passphrase. Of
course, this can still be sniffed if they are on the box.
--
Michael Rowe <mr...@mojain.com>
IM - mr...@jabber.org Prof - ACM, IEEE, Computer Soc.
Web - http://www.mojain.com/ Vice - Barley malt, brewed or
Key - http://mojain.com/keys/mrowe.asc distilled (hold the ice)
> Server machines, no real desktop users. One of these was a firewall
> that pretty much only had SSH listening. *IF* it was hacked directly
> (rather than being compromised with a sniff'd password), then we've got
> something to target. The timestamps don't support much of anything,
> since we don't really have many logs left (he's stupid, but not
> st00p1d). Our logging infrastructure is . . . improving. Also, we're
> implementing grsecurity. I've been very impressed so far (and suspect
> 2.0 will be even better when it's stable).
How about an iptables skript that writes its logs to another machine?
And at the same time the skript should filter traffic from one server to
another...
Another point I would check is if all your _desktop_ computers are
yours. He might have hacked one just to get better access to your server
machines.
Just my 2€
Marc
J
Assuming he has rooted the box removing /dev/kmem won't do any good as
he can merely recreate it using mknod(1).
--
Phillip Hofmeister
PGP/GPG Key:
http://www.zionlth.org/~plhofmei/
wget -O - http://www.zionlth.org/~plhofmei/key.txt | gpg --import
--
Excuse #163: RPC_PMAP_FAILURE
(link below helps here :)
> of kernel-level TTY logger *AND* a kernel-hack to hide files and
> processes. The only comfort in this is that some of our kernels are
> apparently so exotic that his meddling crashes the machine during the
> break-in (instead of leaving a more compromized system). In general,
> all of the rootkits are the same flavor (and seem unrelated to the LKM
> stuff).
> Uhhh, that's me. Trust me when I say I'm as technical as it gets (short
> of the Gods like Linus). It's not a single machine, it's a whole bunch
> of them. It's not a password problem either. He seems to have hacked
> multiple of them within an hour of each other (his rootkit files aren't
> very clever about covering up mtime). I just can't tell how he got in.
> I've got some process accouting logs to go through, but they're ...
> verbose.
There are many ways of getting into the system. I was running the same
thing and got cracked. Now, grsecurity is the only way to fly :)
Go there, DL the patches and _use_ them. It's a real pain to set up
really tight ACLs, but then even root cannot do anything. So cracking
SSH or something might get him access to the box, but not access to
the entire machine.
Futhermore, if you need to run SSH, don't allow everyone to access
it - only allow access to it from known IP ranges. Why? Because
sshd has to be run SETUID and SETGID. Even with grsecurity this allows
the attacker to delete user data.
With something like sendmail or apache, it only needs to see a very
limited part of the file system, so even braking these will not do
any real damage.
Also, grsecurity has stack randomization, etc... so buffer overflows
are a guessing game at best.
- Adam
PS. Any exploits against grsecurity? :)
Don't get too over confident about chrooting Apache. One Apache process
runs as root. This means if there is an exploit that sends arbitrary
code across the shared scoreboard it could be ran as root and break out
of the jail.
However, for the most part, chrooting is a valid countermeasure/method
to compartmentalize. It is a shame that no distribution comes with
packages natively created with/for chrooting.
--
Phillip Hofmeister
PGP/GPG Key:
http://www.zionlth.org/~plhofmei/
wget -O - http://www.zionlth.org/~plhofmei/key.txt | gpg --import
--
Excuse #134: Backbone Scoliosis
Also, Debian's Bind 9 package is pretty trivial to chroot (although it doesn't
by default). Debian's postfix package does chroot by default, although you
tend to have to turn it off if you want to use things like postfix-tls or SASL.
M
First, sorry for my very late reply :) I'm just reading the
messages here now...
Anyway, I wasn't talking about chroot. I was talking about
grsecurity and ACLs (I think). Then you specify what each
process is allowed to do and see (even root cannot get passed that).
You can make Apache see only the directories that you want it
to see. You can also specify that Apache cannot initiate a connection
(except to trusted nameserver for instance) and it can only listen on port 80.
With other features of grsecurity like stack randomization, Apache
becomes pretty much explot-proof...
- Adam
Yes it does. Although I don't believe that the way to go is chrooting since
it makes it very difficult to ease upgrades.
> Also, Debian's Bind 9 package is pretty trivial to chroot (although it doesn't
> by default). Debian's postfix package does chroot by default, although you
> tend to have to turn it off if you want to use things like postfix-tls or SASL.
There are a number of patches in the BTS to make bind work in a chroot
environment out of the box, using bind's own chroot functionality. In any
case, there are also a number of packages to provide an easy way to setup
chroot/restricted environments (makejail and compartment come to mind).
In any case I don't think that chrooting is the way to go here, it was
built to be used as a testing/programing tool, not really a security tool.
There are number of (Linux) patches to provide full compartimentalization
of processes in the system which might be the way to go. Just my 2c.
Regards
Javi