Question
What do I do if I believe my server, or my users, have become compromised?
Answer
First and foremost, don't panic.
There are no quick fixes or solutions to this problem. In many cases, it is notoriously difficult to determine precisely when a compromise occurred or what vector was used to breach your server/users in the first place.
While you may be able to take steps to mitigate the damage that has been done, ultimately, the damage has been done. Your server's reputation may have been destroyed, or worse.
Don't take this personally. Even if it was a directed attack against your server to gain access to your server, do not take it as an affront to yourself. A significant number of admins have been compromised at one time or another. It's the nature of the internet as a whole, and, frankly, to be expected at some point for all servers with Publicly accessible endpoints.
With that understanding, your next steps should be to prevent the problem from becoming worse.
Change every related password you can change, and force password changes for any users that you cannot change on your own. To force specific users to change their passwords, perform the following steps within WHM's Force Password Change interface at Home »Account Functions »Force Password Change:
- Select the Forced? Radio buttons that correspond with the appropriate cPanel users.
- Select as many cPanel users as you wish.
- Use the buttons at the top of the interface to select or deselect all of your cPanel users.
- Click Submit.
If you have multiple servers, check your other servers and be certain that you are NOT accessing those other servers from the compromised one(s).
If the servers contained sensitive user data, there might be legal requirements depending on your jurisdiction; Check with your local legal team or a legal specialist if you are uncertain what your requirement is. Regardless of those requirements, it would be best if you immediately informed your users. No matter how upset they may be over possibly sensitive data being stolen, they will be infinitely more upset if they find out after someone has used their information without their consent, or if someone leaks news that your server was hacked and you knew about it but failed to inform your users.
What's next?
With the damage mitigation out of the way, the next step is to understand what went wrong and prevent it from occurring again.
First, do not place the server back online until the problems have been 100% resolved. It may harm your bottom line in the short term to keep your server offline while you investigate, but it will absolutely destroy it long term if you return the server online before you have resolved the underlying problems.
Determine the full scope of the compromise. Was it only a single user? Multiple users? Root? While there are many ways to go about this, and numerous utilities designed to assist with this, cPanel has created the cPanel Security Investigator(CSI) utility.
The CSI utility is a script that provides a variety of functions to assist with the investigation of both root- and user-level compromises. Please be aware, however, that we cannot provide any support for any information the script may turn up, and it shouldn't be considered the end-all-be-all for determining if a site, or account, has been compromised; It should be used as part of a full suite of security checks. The script itself can be found here:
https://github.com/CpanelInc/tech-CSI
You may also want to review the information outlined in our Determine Your System Status documentation.
In cases of root-level or multi-user compromise, bring the server 'offline' removing it from public access. Failing to perform this step will make nearly every other step impossible to complete with any degree of certainty. Additionally, if it is determined to be a root-level compromise, your only option would be to migrate to a new, clean server, re-install the OS and cPanel, and restore from known good offsite backups. Please review the following documentation for more information:
Why Can't I Clean a Hacked Machine?
In cases of only a single user having been compromised, the following article should be of assistance:
What can be done if a cPanel account is compromised?
With the knowledge gained from your previous inspection, next examine your server(s), in great detail, for anything and everything unusual. Do everything you can to understand how the compromise occurred, which accounts were compromised, and what you can do to prevent it from happening again.
Re-examine your server(s) in even more detail. Follow the path of the compromise. How did they gain access? What did they do once they had access? Did they leave anything behind? Did they use your server to launch attacks against others?
Once you know how they gained access, take steps to close that gateway. If they used some injection in an application, take steps to fix the code that allowed for this behavior in that application and review your other applications to ensure they are not vulnerable to the same attack vector.
Understand that often there is more than one way to gain illicit access to a server. Work with a specialist to close as many of the possible attack vectors as you conceivably can.
Ultimately this is going to be a long process. If you are unfamiliar with common security practices for sites and servers or are uncomfortable performing this sort of investigation on your own, we highly recommend reaching out to a specialist in server security and working with them to resolve all problems you may find.
To aid in the prevention of compromise, we have the following articles that you may want to review as well:
Security Suggestions - How to make your server more secure
How To Run Basic Security Audits On A System?