New user accounts appearing overnight in some form of compromise or cPanel exploit
Hi all,
Thought I would make a post here regarding this.
Put short, I've inherited a customer environment who despite using a stock cPanel experience with ALL features which don't pertain to email and email management via the web ui disabled they advised that they seem to experience compromise-looking behavior often. I have just recently seen my first evidence of this seeing new 'customer' accounts pop up under WHM's Users section with the account themselves as their own account manager.
Deleting/Disabling/Suspending this is one cluck but without any hesitation I've scheduled full rebuild of their WHM cPanel server and will migrate their single user account to a new one this week migrating the customer's single user account which handles their emails, migrated to a fresh WHM cPanel box without troublesome history.
The problem for my brain is that I can't seem to figure out how the likely automated remote attacker was able to compromise the system and even create their own customer account with the below known facts:
* The passwords for WHM and the single customer user account are random and long. Uncrackable by today's standards.
* The logs show no recent logins to have achieved this (Leading me to believe and the remote attacking host is leveraging some bug/exploit of cPanel to achieve their goals)
* The customer's single user account is not a reseller and thus cannot create new customers.
* Most features of the default customer profile have been disabled as this customer only uses it for Mail and the cPanel UI for managing mail-related tools.
* The cPanel box checks for and applies updates weekly and was upgraded from CentOS 7 to AlmaLinux-8 mid 2023.
* I've combed all files on the customer's single user account (Many email files) and have found nothing that could contribute to the compromise. The root filesystem also shows no obvious entry point.
It's in enumerating all these posibilities I noticed it isn't running anything like auditd for audit logs implying evidence could have been wiped. I'm not sure why the cPanel experience wouldn't include that for security sake so I installed and enabled it in case we see this again before migration day.
These accounts seem to pop up and create a DNS zone plus an AutoSSL-protected vhost for hosting their spam garbage. Effectively hijacking the platform for free hosting. The account doesn't seem to do anything else going through the system logs and cPanel's own. But the unanswered question is how are they doing that. There aren't enough logs anywhere at the time of the attack to help figure out what they leveraged for access to this machine and I would like to find the answer to that puzzle.
Really only here asking if anyone else has experienced this on black-box cPanel instances and may have any pointers to look? Everything regarding the single user account is Jailed, even PHP is disabled for the customer avoiding the most obvious php-shell-upload attacks.
As I said the moment I saw this for myself the other day I scheduled the host for a rebuild and will get to that ASAP. But I'm not going to be happy if this is some undiscovered CVE and a new cPanel instance on a new machine will meet the same fate again despite the highly restricted cpanel user account.
-
There are a few ways this could be happening (I assume you are talking about accounts showing up in WHM [port 2087]'s "List Accounts" section)
- Somebody knowing the root password and accessing WHM via port 2087. If these have been changed and randomized, this is unlikely.
- Somebody having previously setup an API token which has got exploited and accessing WHM. In WHM, go to "Development->Manage API Tokens" (searching for "API" in the top right search box will bring you to the same page). See if any tokens are listed there and Revoke them if you don't recognise them for any reason.
- Somebody having setup an SSH key and logging in via root SSH (changing the root password would not invalidate the SSH keys). In WHM, go to "Security Center -> Manage root's SSH keys" (search for "keys" will take you to the same page). Again, delete/revoke any keys you don't recognise.
- Somebody has setup a sudo account on the server. Via SSH, try the following:
grep -v '^ *#' /etc/sudoers | grep -v "^$"
should return:
Defaults !visiblepw
Defaults always_set_home
Defaults match_group_by_gid
Defaults always_query_group_plugin
Defaults env_reset
Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS"
Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin
root ALL=(ALL) ALL
%wheel ALL=(ALL) ALL- Somebody has setup a wheel or admin account on the server. Via SSH, try the following two commands:
getent group wheel
getent group rootwhich should both return
[wheel or group]:x:[number]:
- Your server has been potentially been compromised. Try this cPanel tool to scan for common signs:
curl -s https://ssp.cpanel.net/run | sh
It might be useful for you to check the following files (probably using "tail"):
- /usr/local/cpanel/logs/access_log
- /usr/local/cpanel/logs/api_tokens_log
- /usr/local/cpanel/logs/login_log
- /usr/local/cpanel/logs/session_log
- /var/log/secure
to see if there is any signs of unauthorized activity (times/dates/IPs etc).
I would also look at firewalling off any non-mail ports to only allow access to your "admin network" - especially on ports 22 (SSH), 2086, 2087 (WHM) and 2082, 2083 (cPanel).
1 -
This most recent attack which has happened during my watch doesn't appear to be an SSH-based attack.
I've been investigating this most of the day and there was something which caught my eye. `/etc/shadow` has a password hash for the customer's only user account which I cannot reproduce using its salt and our known password hash. Despite logging into the cPanel interface working just fine using the current password (Which seems to not be what's stored in `/etc/shadow`). SSH is disabled for their single user account but I suspect this may be a bootstrap from an earlier attack and they may have authenticated using the cPanel API to create their new account. Still though, this customer's user account doesn't have retailer access.
sudoers(.d) files and getent outout was fine. /etc/{group,passwd,shadow} all look correct though the permanently-suspended attacker user makes an appearance. They even have a shadow entry with a password hash. I wonder what that's set to but it's probably just a random string.
I've gone through all the content under /usr/local/cpanel however nothing stands out. I only today enabled some of cPanel's auditing options via WHM unfortunately too late for this instance.
Nothing in /var/log/secure except my own pubkey accesses (Good..)
I did install iptables & iptables-services a while back only allowing the customer's expected ports. The host accepts only 22,80,443 and all smtp + imap on the INPUT table.
I appreciate the suggestions though that's a funny way to invoke unknown code from the web given the thread context. I understand however that this is cPanel's own script and will read over and run it.
0 -
That script ran and had nothing worth noting. It threw some WARNs however each checks out without concern.
It's also worth noting despite `root`'s already scrambled root password it disallows root login (Key only).
I've re-set the hash for the customer-user account of their cPanel setup and /etc/shadow now contains a hash for their user I can reliably achieve with `openssl -6 -salt -stdin`. This leaves me thinking it may have been a potential vector in this attack if the cPanel API allowed their customer-user had access.
0 -
It would likely be best to submit a ticket to get this checked out.
0
Please sign in to leave a comment.
Comments
4 comments