Shellshock 1

[Phuong Fulsom]

Ihave some Linux servers, which from what I can tell are not vulnerable to the shellshock attack vector, but I am curious what the attack looks like in the logs. What does a successful attack look like in the appache2 logs?What would a successful attack look like in the system log?

A well configured web-server won't be able to overwrite its logs, and in all but the tiniest environments, you should be using centralized logging to protect you from losing these kinds of log entries.

If you're successfully exploited, you may see lots of errors in the error_log which show failed attempts to access resources, execute programs or delete files. If they successfully escalate privileges, and you don't have off-site logging, then you may see no evidence.

If the User-agent header is used to perform the attack, it will be quite visible in the log access log, since that header is logged. But there are other headers which can be used to perform the attack, which are not logged by default.

If an attempted attack failed due to bash having been updated to the intermediate version that fixed the bug but still allowed functions to be specified through environment, you would see warnings in the error log. They may look like this:

I saw the CISO of Mandiant speak recently, he said when he is asked if there was always a pattern of behaviour that distinguishes an attacker (malware) from valid activity, he replies that if the attacker behaved exactly the same as valid worker we should thank them for doing work for free for us. Clearly the attacker will behave differently in some way from a valid worker. This implies that if we monitor behaviour/activity on a system, we should be able to detect if any malicous activity has taken place. Unfortunately system logs are likley to be tampered with by an advanced attacker so as well as shipping out to syslog and all the usual methods, you must monitor the whole system behaviour in some way to look for anomolies rather than just scrape one log. Sorry if this is an inconvenient answer but APTs are rather inconvenient... I appreciate this post does not actually answer the original posted question and is therfore not 100% helpful, but I do want people to remember not to focus too closely on a specific log entry and do not forget to monitor/analyse general behaviour as well

The scanner comes back with: "Site appears vulnerable to the 'shellshock' vulnerability ( -bin/cvename.cgi?name=CVE-2014-6271)."

I realize I'm talking about a four year old vulnerability but it's one that still exists and it's a rabbit hole I wanted to jump into. I've come across this vulnerability a few times in the past and I've either used Metasploit or 34900.py ("Apache mod_cgi - 'Shellshock' Remote Command Injection") to get my shell. I seem to recall having an issue with one or both at some point and I moved on to another avenue because my search results yielded bits and pieces but nothing that I could wrap my hands around.

Stumbling upon this vulnerability recently, I paused to dig into it with the intention of getting a better understanding for manual exploitation.

Excellent -- we have a shell!

It wasn't really that hard to get this working, I just needed to play with the syntax. In my searching, I saw examples of using wget or curl to pull in other files but I never understood why the need to add extra steps when you can get the shell directly.

So maybe you're thinking what's the big deal? Why did I need to go through this exercise?

Sometimes I rely on tools and it's a crutch. Sometimes I understand the mechanics and the tool is just easier / quicker. In this case, it was most definitely a crutch for a lack of knowledge and here's where this would have helped me out.

A while ago, I wrote up Vulnhub SickOS 1.1 Walkthrough and I actually noted the server was vulnerable to Shellshock. In the writeup, I walk through the process of exploiting the CMS which gets me a low privilege shell but now let me take you through the express lane.

We know we have a Squid proxy running on our target. Let's use Curl to hit the CGI script through the proxy:

curl -x :3128 -L -bin/status

You may have some luck checking the logs of your attack vectors, but given that so many services are vulnerable and not all of them log every access, it's likely not possible to conclusively find an attack.

Common rootkits install themselves in /root or / or /tmp or one of the binary paths but really they could be anywhere. They might have a name similar to a real service or something "important" looking like "IPTables" or "kernel-bin" but they could also be random strings of characters or the same name as a genuine binary (just in a different path). You can spot a really obvious rootkit loading in /etc/rc.local or making connections via netstat -neopa. Look for suspicious process names in top -c.

A less common and much more difficult to find rootkit replaces a library or loads itself as a shim library and intercepts system calls. This is almost impossible to find unless you strace/ltrace every single thing running on your system and compare the behaviour with the expected behaviour of a known-good system or source code.

Since there are several attack vectors for Shellshock, some of them yet being unknown for general public or caused by a custom CGI script, there is no definite way to tell if you are compromised or not.

If you have proper server health monitoring (such as Zabbix) up and running, it can help you finding out security breaches, too. You can also compare the MD5/SHA sums of system files to a known good backup.

I just had the pleasure to clean up a compromised older Plesk system.The first thing that gave it away were numerous processes that were started listening to a number of ports and others trying to download code from the original scanning server.

Following the logs I found out the ultimate hole was the a cgi_wrapper script, something that was supposed to protect and shield the system is what actually tore the hole into the protection.Here are some of the log lines from probes and the successful attack:

These are the lines from the access_log, as this is just a sample, note the 200 on two of the lines while the other ones fail with 404. You do not have to worry about the lines that have a 404 since these did not succeed, the ones with 200 however did. The pattern on these attacks here is always the same: 1. find a vulnerable cgi script use shellshock exploit to download and execute a perl script, delete the perl script again. The perl script will actually download some source files (tgz) compile them and run them, from what I have seen they include at least a backdoor and a automatic update mechanism, plus what looks like exploits to try and gain elevated execution privileges. All of the initial scripts are actually executed as the user provided by the wrapper while later services are started with a PPID of 1 (started from root process)).

After a while I noticed ssh connections from various places like China that usually do not visit our server that much. I patched bash as a emergency measure (would have been nice to have patched sources available from the FSF Website and not just the really OLD sources and patch-files (one of which did not apply correctly at first).System is scheduled for a complete wipe now, so if anyone is looking for something else about the attack, you can ask, but do it soon.

Try to get a clean static build of rpm and run command rpm --verify --all. It will tell you which files belonging to a package have been modified. But since you may run it on a compromised system, you may not trust completely the result. Then you may simply do a rpm -qa to get the list of packages, recreate another system with the same packages versions and then a find / -type f xargs -r -n 100 md5sum sort on both system and see what differs.Also if you manage your system properly (meaning not installing anything manually outside of /opt or /usr/local/bin or another unmanaged place), you can search for all files in your system which does not belong to a package, with find / -type f -exec rpm -qf \;. It should show errors for unknown files. I let you to not show the positives as an exercise ;-)

To do the same periodically with cryptographic proof, there is a tool called Tripwire which you may still find as free version. It is old but does its job. A newer alternative is AIDE, but it was not using crypto when i looked at it years ago.

Obviously these tools should have been installed and configured before the system is compromised, and these tools can also be targeted if your system is successfully hacked to root access. Besides, these tools can be very intensive and slow down your system.

Red Hat has been made aware of a vulnerability affecting all versions of the bash package as shipped with Red Hat products. This vulnerability CVE-2014-6271 could allow for arbitrary code execution. Certain services and applications allow remote unauthenticated attackers to provide environment variables, allowing them to exploit this issue.

Red Hat has become aware that the patch for CVE-2014-6271 is incomplete. An attacker can provide specially-crafted environment variables containing arbitrary commands that will be executed on vulnerable systems under certain conditions. The new issue has been assigned CVE-2014-7169.

Updated bash packages that address CVE-2014-7169 are now available for Red Hat Enterprise Linux 4, 5, 6, and 7, Red Hat Enterprise Linux 5.6 Long Life, Red Hat Enterprise Linux 5.9 Extended Update Support, Red Hat Enterprise Linux 6.2 Advanced Update Support, and Red Hat Enterprise Linux 6.4 Extended Update Support, and Shift_JIS for Red Hat Enterprise Linux 5 and 6. See also Resolution for Bash Code Injection Vulnerability via Specially Crafted Environment Variables (CVE-2014-6271, CVE-2014-7169) in Red Hat Enterprise Linux.

3a8082e126