SSH intrusion in Fedora 16

[Marko Rauhamaa]

I strongly suspect my home server has been broken into. I noticed the
intrusion by sheer luck by wondering why the fan was on and the CPU
utilization was at 100%. The "top" utility showed that "cc1" processes
were coming and going. I tried to identify the source with "ps" which
revealed:

$ ps ax | grep cc1
31054 pts/11 R+ 0:00 /usr/libexec/gcc/x86_64-redhat-linux/4.6\
.2/cc1 -quiet -I . -I . -D SSHDIR="/etc/ssh" -D _PATH_SSH_PROGRAM="\
/usr/local/bin/ssh" -D _PATH_SSH_ASKPASS_DEFAULT="/usr/local/libexe\
c/ssh-askpass" -D _PATH_SFTP_SERVER="/usr/local/libexec/sftp-server\
" -D _PATH_SSH_KEY_SIGN="/usr/local/libexec/ssh-keysign" -D _PATH_S\
SH_PKCS11_HELPER="/usr/local/libexec/ssh-pkcs11-helper" -D _PATH_SS\
H_PIDDIR="/var/run" -D _PATH_PRIVSEP_CHROOT_DIR="/var/empty" -D SSH\
_RAND_HELPER="/usr/local/libexec/ssh-rand-helper" -D HAVE_CONFIG_H \
loginrec.c -quiet -dumpbase loginrec.c -mtune=generic -march=x86-64\
-auxbase loginrec -g -O2 -Wall -Wpointer-arith -Wuninitialized -Ws\
ign-compare -Wno-pointer-sign -Wformat-security -fno-strict-aliasin\
g -fno-builtin-memset -fstack-protector-all -o /tmp/cci7t8b5.s
$ ps -ef | grep cc1
root 31098 31097 0 10:10 pts/11 00:00:00 /usr/libexec/gcc/x8\
6_64-redhat-linux/4.6.2/cc1 -quiet -I . -I . -D SSHDIR="/etc/ssh" -\
D _PATH_SSH_PROGRAM="/usr/local/bin/ssh" -D _PATH_SSH_ASKPASS_DEFAU\
LT="/usr/local/libexec/ssh-askpass" -D _PATH_SFTP_SERVER="/usr/loca\
l/libexec/sftp-server" -D _PATH_SSH_KEY_SIGN="/usr/local/libexec/ss\
h-keysign" -D _PATH_SSH_PKCS11_HELPER="/usr/local/libexec/ssh-pkcs1\
1-helper" -D _PATH_SSH_PIDDIR="/var/run" -D _PATH_PRIVSEP_CHROOT_DI\
R="/var/empty" -D SSH_RAND_HELPER="/usr/local/libexec/ssh-rand-help\
er" -D HAVE_CONFIG_H ssh-keygen.c -quiet -dumpbase ssh-keygen.c -mt\
une=generic -march=x86-64 -auxbase ssh-keygen -g -O2 -Wall -Wpointe\
r-arith -Wuninitialized -Wsign-compare -Wno-pointer-sign -Wformat-s\
ecurity -fno-strict-aliasing -fno-builtin-memset -fstack-protector-\
all -o /tmp/ccmOShia.s

Unfortunately the onslaught was over before I could trace the process
ids to the ancestor process.

I then checked the sanity of my SSH installation:

$ sudo rpm -qf /usr/bin/ssh
openssh-clients-5.8p2-16.fc16.1.x86_64
$ sudo rpm -V openssh-clients
5S.T..... c /etc/ssh/ssh_config
5S....... /usr/bin/ssh
5S.T..... /usr/bin/ssh-add
$ sudo rpm -qf /usr/sbin/sshd
openssh-server-5.8p2-16.fc16.1.x86_64
$ sudo rpm -V openssh-server
5S.T..... c /etc/ssh/sshd_config
5S.T..... /usr/sbin/sshd

I also downloaded openssh-clients-5.8p2-16.fc16.1.x86_64.rpm from the
net and compared files:

$ diff /etc/ssh/ssh_config etc/ssh/ssh_config
26a27,30
> # GSSAPIAuthentication no
> # GSSAPIDelegateCredentials no
> # GSSAPIKeyExchange no
> # GSSAPITrustDNS no
46a51
> GSSAPIAuthentication yes
$ ls -l /usr/bin/ssh usr/bin/ssh
-rwxr-xr-x 1 root root 392464 Jul 25 10:47 /usr/bin/ssh
-rwxr-xr-x 1 marko marko 434288 Jul 25 10:47 usr/bin/ssh

Notice how the time stamps are identical but the sizes are significantly
different.

Complete rpm -Va findings attached.

I have no idea how the intrusion took place. The only possibilities that
come to mind are the SSH server, which is open to the world, or
something hidden in a Fedora rpm.


Marko

[Bit Twister]

On Tue, 20 Dec 2011 14:33:23 +0200, Marko Rauhamaa wrote:
>
> Complete rpm -Va findings attached.
>
> I have no idea how the intrusion took place. The only possibilities that
> come to mind are the SSH server, which is open to the world,

That would be my guess

> or something hidden in a Fedora rpm.

I'll still go with a password attack at the server.

My recommendation, get backup procedure hard copy,
backup any needed user stuff and what not,
verify data is on backup media and is accessible.
Get restore procedure hard copy.

format the drive, disconnect cable from the Internet, clean install with
different passwords.

change/set PermitRootLogin without-password or PermitRootLogin no
in /etc/ssh/sshd_config

restart sshd. Connect Internet cable, setup your repositories, get
updates.

Install/run something like aide, osiris, ossec-hids, samhain,
tripwire, snare, integrit, .....

Reboot, restore from backup, tell users to change passwords, tell them why,

Why the above? you ask.

---------------- standard hacked/cracked text follows: ----------------------

Think about what happens when the cracker installs a rootkit.

The kit intalls programs which will not report the files/activities/ip
address which the cracker requires to run a stealth operation from
your computer.

An example, cat, grep and ls are modified to skip any line containing
the string crack_kit and/or an ip address(s).

Now doing a cat /etc/passwd or cat /var/log/messages would not show the
account crack_kit in /etc/passwd or logins by the account crack_kit in
/var/log/messages.

grep -v good_ips_here /var/log/secure would not show the cracker's ip address.

Or a script example to find a new .crack_kit directory:
cd /home
ls -a1 > /tmp/today
diff /tmp/today /tmp/yesterday > /tmp/results
if [ $? ] ; then
mail -s "ALERT: check /home " root@localhost < /tmp/results
fi
mv /tmp/today /tmp/yesterday


See your machine looks ok, nothing strange.

Remember, the better cracker has installed ALL the programs you use to
find the cracker's tracks, but the programs are hiding the tracks you
are trying to find.

For a good crack to go unnoticed, the cracker stops all monitoring
programs tripwire, swatch,...., logging programs sysloger,...
installs the rootkit, removes tracks from current logs, removes any
mail in root's mailbox, enable all the previous disabled
monitors/loggers.

Now if you had the logs being updated or mirrored on a different
machine with programs monitoring those, then you have a chance.

To check your machine you could load the binaries used to look for
cracks on a cd, and use those to audit the system.
Now /cdrom/bin/cat /etc/passwd would show the crack_kit account.

To check your installed rpms
/cdrom/bin/rpm -Va | grep '..5' > /tmp/verify.log
diff /tmp/verify.log /floppy/verify.log

You could do /cdrom/bin/md5sums on all the */bin/* files and store the
results on cd and compare results to find new/modified bins.

Of course, all the baseline results have to be agains the box before
is is ever connected to the network. :)

[Bit Twister]

On 21 Dec 2011 11:58:43 GMT, all mail refused wrote:
> PasswordAuthentication no in /etc/ssh/sshd_config

That is a higher security level, but a bit restrictive.
I can agree "root" could ssh into system as a user, then get root access.
See Note:

With PermitRootLogin without-password you can still ssh root@target to
get in if root@target already has your public key.

Note:
Comes in handy if system is dinked up and users can not log in.
Also if you need to scp/rsync files with root privs from one system
to target.

[Marko Rauhamaa]

all mail refused <elvis...@notatla.org.uk>:

> PasswordAuthentication no in /etc/ssh/sshd_config

Sure, but:

* That's not the default in Fedora 16.

* The default firewall in Fedora 16 leaves sshd open to the world.

* If you get in as a regular user, you can sudo in using that password.

* I don't know the intruder came in using a password -- or through SSH
for that matter. I think the passwords were pretty strong and secret.


Marko