What to do when your website is hacked/exploited

So your website has been "hacked"? You load your website in your browser, and are redirected to some spammers website, or maybe you google'd yourself (naughty), and found a few thousand indexed pages for knock off prada gear? Ok, so how do you fix this, and more importantly, how do you learn how they did it so you can defend against it later.

Secure the scene

The first thing I do is take a snapshot of the hacked web site. I want the entire webroot, and the access and error logs so that I can review them.

cd /var/www/vhosts/mysite.com
tar -zcf 20150101-siteexploit-evidence.tar httpdocs logs/access_log* logs/error_log*

At this point I want to either shutdown the web server, or just restore the site and get back to the investigation. I would shut down if I felt the exploit was serious enough to warrant more serious action. Otherwise, just restore from backup, or checkout your repo, whatever you gotta do.

Here is where the fun begins. Try to figure out when, and how, your site was exploited. First things first, take an inventory of the scene.

Looking For Clues to how the site was hacked

Run the last command for suspicious logins. "last" will show a listing of last logged in users. Maybe someone SSH'd into your machine (hopefully not!) or logged in via FTP. If you see a suspicious IP in last then chances are someone has initiated the attack by simply uploading a file through FTP or SCP.

Check for modified files. If you are seeing files modified recently that you did not edit yourself they are suspect and probably contain malicious code. Check the modification timestamp on those files. That date will be useful when looking through the access logs. If you know that you have not made changes to your site in {n} days use that as the basis for your search:

find . -type f -mtime -{n} -print > modified_files.txt

Replace {n} with an actual number.

Scan all files in the webroot for common exploit signatures. If you find a lot of eval(gzinflate then you can almost be certain that it is malicious code. A good rule of thumb is eval is "evil", except when it's not (protip: it usually is).

grep -ir 'gzinflate\(base64_decode' *.php > suspect_files.txt

Here is a bash script that checks for common exploits:

#!/bin/bash
 
pattern='r0nin|m0rtix|upl0ad|r57shell|c99shell|shellbot|phpshell|void\.ru|phpremoteview|directmail|bash_history|\.ru/|brute *force|multiviews|cwings|vandal|bitchx|eggdrop|guardservices|psybnc|dalnet|undernet|vulnscan|spymeta|raslan58|Webshell|get_pass|PhpConfigSpy|SubhashDasyam|(eval.*(base64_decode|gzinflate|\$_))|\$[0O]{4,}|FilesMan|JGF1dGhfc|IIIl|die\(PHP_OS|posix_getpwuid|Array\(base64_decode|document\.write\("\\u00|sh(3(ll|11))|earnmoneydo'
searchpath=/var/www/vhosts
grep $pattern $searchpath -roE --include=*.php* | sort

That will check all .php files under /var/www/vhosts for common php shells and exploits.

Collect Evidence

Ok, so you have a date, maybe a date range, and a list of potentially suspect files. With that information it's time to start looking at logs. The goal is to be able to determine when, and potentially how, the attacker exploited the site.

There are 5 common vulnerabilities attackers exploit.

1. Remote code execution
2. SQL injection
3. Format string vulnerabilities
4. Cross Site Scripting (XSS)
5. Username enumeration

All web developers should be familiar with the OWASP top 10 too. Anyways, you want to look for signs of those types of attacks. I scan my access logs and typically check for the following things:

• Check for POST requests to suspicious locations, such as a file upload location, or other areas that are not accessed directly (like an includes directory, or theme directory).
• Check for GET or POST requests with a remote URL as a parameter, such as /page/?url=http://atackersite.com/malicious_script
• Check for GET or POST requests to the files you found when scanning the site for malicious code
• Check for very odd query parameters in GET and POST requests
• You should know your site, check for requests that are just out of the ordinary

When you find suspicious entries in your logs, take note of the IP addresses. Scan your logs for all entries for that IP, you might be able to find when the exploit occurred. If you do, you then know the requests that might have been the culprit, or at least have a date range for when the exploit took place. Check the logs around that date range. It could be that the attacker initiated the exploit through a proxy, and finished the work through another one.

Take Action

I recently had a clients website get exploited. The attacker modified dozens of PHP scripts, uploaded about 30 different PHP shells, and just generally made a mess of things. Most of the modified files had the same modification timestamp, so using that I checked the logs and found that someone had been crafting POSTs to a file upload plugin for a wysiwyg editor. Looking into that plugin, I found that they were able to upload a PHP shell. I quickly removed the plugin, restored the modified files, and removed all of the uploaded PHP shells.

The exploit showed me that the security of the website and server were severely lacking. Some  fundamental security precautions were not implemented. Such as allowing script execution in a user upload directory. That should just never happen. I setup some rules to disable script execution in specific directories, disable direct access to other directories, secured file and directory permissions.

I also setup fail2ban on the clients server. Fail2ban is great for monitoring logs and looking for certain signatures, and then banning the offending IP. You can define other actions to take, such as emailing you or something, but I just want to block their access to the server. So, ban via iptables. I noticed that this particular attacker was sending a lot of POST requests at the same time, so I setup a fail2ban filter to handle a postflood. You will probably want to setup filters for dealing with failed log in attempts to the server, and to address any of the other suspicious activity you found in your logs.

You will definitely want to update your CMS and update any 3rd party dependencies. Doing that alone will go a long way towards keeping your site exploit free.

Wrap up

Security is hard. It's a never ending battle. And you will never, ever, make your site 100% safe and secure. The best thing to do is have a recovery plan. When your site is exploited can you recover? If you can answer yes to that, you are in a good place.