Tracking down compromised files that are sending mail

Question

How do you track down compromised files that are sending mail?

Answer

Files will often be created in base64 when accounts are compromised to mask what they are doing. You can use a variety of tools to check if the file is in use. In our case, we'll use a script written by our cPanel staff and stat to check the modification time. By using the "modify" time, you can then check the access log in hopes of finding out what IP accessed the server and what specific file was affected.

Note: You may still wish to consult with your server administrator to investigate the mail abuse and potential security issues.

Here are some steps to help track this down.

1. Log in to WHM as the root user
2. Navigate to Home / Service Configuration / Exim Configuration Editor
3. Select the Advanced Editor tab
4. Look for log_selector

5. Change the options to the below:

   CONFIG_TEXT: +all -ident_timeout -pid

6. If you do not want all of the options, you can choose what you would like to add:

   CONFIG_TEXT: +address_rewrite +all_parents +arguments +connection_reject +delay_delivery +delivery_size +dnslist_defer +incoming_interface +incoming_port +lost_incoming_connection +queue_run +received_sender +received_recipients +retry_defer +sender_on_delivery +size_reject +skip_delivery +smtp_confirmation +smtp_connection +smtp_protocol_error +smtp_syntax_error +subject +tls_cipher +tls_peerdn

7. After the exim log has populated new entries, you'll be able to print the most common directories that direct Exim:

   # awk '$3 ~ /^cwd/{print $3}' /var/log/exim_mainlog | sort | uniq -c | sed "s|^ *||g" | sort -nr

8. You can usually ignore the main general directories, like those from WHM or root. But if a user is sending large amounts of email from a script somewhere in their home folder, this will show it:

   CONFIG_TEXT: 908136 cwd=/home/abusiveuser/public_html/abusivemailfolder
   12636 cwd=/home/notasbad/public_html 
   284 cwd=/var/spool/exim 
   15 cwd=/

9. Next, using grep, search the folders for anything encoded in base64:

   # [/home/abusiveuser/public_html/abusivemailfolder]cPs# grep -i base *badmailfile.php:PD9waHAgaWYoaXNzZXQoJF9GSUxFU1sidXNlcmZpbGUi
   .. 
   .. 
   IjsgZWNobyAiU3BpZGVyIFByb2plY3QiOyBleGl0OyA/Pg base64 --decode

10. Once you find a file that is base64-encoded, use stat to check its modified time:

    # stat /home/abusiveuser/public_html/abusivemailfolder]cPs# stat badmailfile.php
    File: `badmailfile.php' 
    Size: 1271 Blocks: 8 IO Block: 4096 regular file 
    Device: fd02h/64770d Inode: 33423367 Links: 1 
    Access: (0644/-rw-r--r--) Uid: ( 527/abusiveuser) Gid: ( 525/abusiveuser) 
    Access: 2020-06-25 17:39:02.539000002 -0600 
    Modify: 2020-05-25 13:31:30.276000041 -0600 
    Change: 2020-05-25 13:31:30.276000041 -0600

11. There may be other base64 files, so you'll want to check those as well and run stat on each one, noting the "Modify" time:

    CONFIG_TEXT: Modify: 2020-05-25 13:31:30.276000041 -0600

12. If the date is older than one month, you will need to look through the archived logs, which are located in /home/abusiveuser/archived-logs
13. If it's within the month, you can look at the access logs under /home/abusiveuser/access-logs
14. You will be looking for a POST command in the log. Once you find the entry that accessed the domain, you'll know the IP and file that was used to exploit the account.

Once you discover how the account was compromised, and you have backups, you'll want to restore the account and make the necessary changes to prevent the site from being compromised again.

Additional Resources

Is it possible to clean malware from a hacked website?

What do I do if I believe my server has been hacked?

Where can I find domain access logs?