Skip to main content

p0f failing

Spork Schivago
@cPanelLauren, I was unable to figure out how to send you a PM. I even went to your page, but saw no options. Is this something I should be worried about? Service Check Raw Output (XID 9xtngq) The "p0f" service is down. The subprocess "/usr/local/cpanel/scripts/restartsrv_p0f" reported error number 3 when it ended. Startup Log May 03 05:27:34 franklin.business.net p0f[3800]: [+] Intercepting traffic on interface 'any'. May 03 05:27:34 franklin.business.net p0f[3800]: [+] Custom filtering rule enabled: less 400 and not dst port 80 and not dst port 443 and tcp[13] & 8==0 May 03 05:27:34 franklin.business.net p0f[3800]: [+] Listening on API socket '/var/cpanel/userhomes/cpanelconnecttrack/p0f.socket' (max 20 clients). May 03 05:27:34 franklin.business.net p0f[3800]: [+] Privileges dropped: uid 989, gid 986, root '/var/cpanel/userhomes/cpanelconnecttrack'. May 03 05:27:34 franklin.business.net p0f[3800]: [+] Daemon process created, PID 3814 (stderr kept as-is). May 03 05:27:34 franklin.business.net p0f[3800]: Good luck, you're on your own now! May 03 05:27:34 franklin.business.net systemd[1]: Started p0f passive fingerprinter. May 03 05:38:14 franklin.business.net p0f[3800]: [!] WARNING: User-initiated shutdown. May 03 05:38:14 franklin.business.net systemd[1]: Stopping p0f passive fingerprinter... May 03 05:38:14 franklin.business.net systemd[1]: Stopped p0f passive fingerprinter.
Someone's been trying to get into my server for the past few weeks, but they really started kicking it up today / last night. I got a message from cPanel saying cPanel was killing a program because we were almost out of memory. That got me a bit worried. I've been watching them try to get in and they've always been attempting the exact same ports. They get blocked, they reconnect with a different IP, try the exact same ports. Well, after that cPanel message, I log in, and they're not trying the same ports anymore. No, they're trying ports that are pretty dang close to ports that I have open (one off, for instance). And the whole 8==0 doesn't seem right, and Good luck, you're on your own now! That worries me a bit!

Comments

11 comments

Date Votes
  • cPanelLauren
    Hi @Spork Schivago As you can see, I moved this to its own thread so it could be addressed. For p0f failing that's an interesting error as it indicates that it's a user level stop. if you run the following via CLI what is the output? grep -i "out of memory" /var/log/messages
    When you try to restart p0f now does it give you the same error? If it does can you show me the version of p0f you're using by running the following command; rpm -qa |grep p0f
    For example my version is as follows: [root@server ~]# rpm -qa |grep p0f p0f-3.09b-1.cp1150.x86_64
    More information on p0f can be found in our documentation here: Service Manager - Version 68 Documentation - cPanel Documentation and in theirs here: p0f/p0f
    0
  • Spork Schivago
    I temporarily shut down the server, but it wasn't just p0f that failed. That was the first, then there were messages about cPanel not being able to access 127.0.0.1:8(thousand something), which was the java stuff, and then I get a whooooole bunch of messages like that, all red. So I rebooted. But now I've temporarily shutdown the VPS, hoping after a while, they'll give up. But it makes me think perhaps they found away in through cPanel. Even with all the security software I have, it's hard to stop them because they have soooooo many IPs from all around the world. This is why I don't really care for people who pirate software. People think they're getting something for free, but 1) It's stealing, and 2) It's almost always infected with something. They don't notice any symptoms but now-a-days, "hackers", and I use the term very loosely here, I think would rather infect a machine and not be noticed, rather than infect a machine and directly be noticed. They create these large botnets that they can rent out or use for massive DDoS's, etc.
    0
  • Spork Schivago
    Here's some more for you to look at: The service "tailwatchd" appears to be down. Server hostname.business.net Primary IP Address Service tailwatchd Name Service failed ? Status Notification The service "tailwatchd" appears to be down. Service Check Raw (XID 4cr3ts) The "tailwatchd" service is down. Output Used 2.29 GB Memory Available 1.56 GB Information Installed 3.85 GB Load 1.93 0.94 0.36 Information Uptime 1 minute and 40 seconds IOStat avg-cpu: %user %nice %system %iowait %steal %idle 53.29 0.32 9.37 3.52 0.34 33.16 Information Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn loop0 6.89 40.92 7.07 4085 706 sda 367.40 16408.60 405.37 1637906 40464 sdb 0.43 11.74 0.00 1172 0 PID Owner CPU % Memory % Command Top 3460 git 38.90 11.71 unicorn master -D -E production -c Processes /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru 3152 git 33.90 12.18 sidekiq 5.0.5 gitlab-rails [0 of 25 busy] 4191 cpanelsolr 9.35 5.93 /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m - Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 - XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 - XX:+UseConcMarkSweepGC -XX:+UseParNewGC - XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 - XX:+CMSScavengeBeforeRemark - XX:PretenureSizeThreshold=64m - XX:+UseCMSInitiatingOccupancyOnly - XX:CMSInitiatingOccupancyFraction=50 - XX:CMSMaxAbortablePrecleanTime=6000 - XX:+CMSParallelRemarkEnabled - XX:+ParallelRefProcEnabled -XX:- OmitStackTraceInFastThrow -verbose:gc - XX:+PrintHeapAtGC -XX:+PrintGCDetails - XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps - XX:+PrintTenuringDistribution - XX:+PrintGCApplicationStoppedTime - Xloggc:/home/cpanelsolr/server/logs/solr_gc.log - XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 - XX:GCLogFileSize=20M - Dsolr.log.dir=/home/cpanelsolr/server/logs - Djetty.port=8984 -DSTOP.PORT=7984 -Dhost=127.0.0.1 - Duser.timezone=UTC - Djetty.home=/home/cpanelsolr/server - Dsolr.solr.home=/home/cpanelsolr/server/solr - Dsolr.install.dir=/home/cpanelsolr -Xss256k - Dsolr.autoSoftCommit.maxTime=3000 - Dsolr.log.muteconsole - XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 8984 /home/cpanelsolr/server/logs -jar start.jar -- module=http 4579 root 4.76 0.36 /usr/local/cpanel/scripts/restartsrv_tailwatchd 4561 root 3.70 0.23 /usr/local/cpanel/bin/tail-check
    0
  • cPanelLauren
    Hi @Spork Schivago It sounds like the attacker you've been talking about is beginning to cause serious issues with the stability of your server, if there aren't enough resources or if the server is under strain, services will begin to fail. I think in order to address this the situation with the attacker needs to be addressed further. If the attacks are coming from more than one IP address you might want to look at CSF's DDoS features with Connection Tracking which allows ou to set the limit on connections per IP address, there's also the SYNFLOOD protection, a synflood attack is a DoS attack exploiting the TCP connection process itself Here are some other links that talk about this: There's also some great info on the CSF forums on how to configure this - their site is down right now but I wouldn't expect it to remain that way for long.
    0
  • Spork Schivago
    I do have ConfigServer Firewall already setup and configured. I know understand why snyfloods are and we are protected against them. ConfigServer Firewall is banning them, but the issue is as soon as they get banned, they just use another IP address. This makes it extremely hard. They have a VERY large number of IP addresses. ModSec showed them finally trying to access a site on the webserver, /w00tw00t.at.blackhats.romanian.anti-sec I believe these are the people responsible for the attack. I believe the way it probably works is something like this. They offer a cracked version of some popular software or a keygen for some popular software to download, more than likely from a torrent site. People download it, without realizing it's infected. They don't notice any signs, everything seems to be working fine, and they think oh how, look how smart I am! I got this really expensive software for nothing! Then, one day, some "hacker" (using the term loosely here, because to me, I have a different definition of what a hacker is) decides to attack a site like mine, or maybe the Playstation Network. They either rent the botnet that the person who wrote the infected keygen / cracked software from or, they're the people that wrote it themselves, and take control of everyone's PC. They say okay, everyone, start attacking this one site. With me, as soon as one IP gets blocked, they just simply use another one. Perhaps ConfigServer Firewall isn't configured 100% properly and was keeping a log of every IP address. Eventually, RAM gets used it, and services start to fail. Hopefully, they've given up their attack by now. Gonna fire the VPS back up and see what dmesg shows.
    0
  • cPanelLauren
    A DDoS attack is exactly what it sounds like is happening - An attack distributed through a collection of IP addresses. That's why I suggested the connection tracking and synflood protection, they should keep too many ip's from connecting on any one port and keep the number of connections from IP down, though admittedly that will not help if they're using more separate IP's than your server can handle.
    0
  • Spork Schivago
    A DDoS attack is exactly what it sounds like is happening - An attack distributed through a collection of IP addresses. That's why I suggested the connection tracking and synflood protection, they should keep too many ip's from connecting on any one port and keep the number of connections from IP down, though admittedly that will not help if they're using more separate IP's than your server can handle.

    To me, a DDoS isn't just an attack distributed through a collection of IP addresses, but one that denies regular users access to the site, hence the name Distributed Denial of Service. I'm not denied service to my site, they're just using a distributed collection of IP addresses to try and hack my site. More like a Distributed Hacking of Service or something! {: -)
    0
  • inteldigital
    Upgrade to 70 and geo-ban IP addresses with more than 5 connections with CSF?
    0
  • Spork Schivago
    Upgrade to 70 and geo-ban IP addresses with more than 5 connections with CSF?

    I have to wait until 70 hits my current tier. Hopefully it won't be too much longer.
    0
  • inteldigital
    I have to wait until 70 hits my current tier. Hopefully it won't be too much longer.

    I'm with you, man. We're waiting for it to land on the release channel before we put it on the production server.
    0
  • Spork Schivago
    I'm with you, man. We're waiting for it to land on the release channel before we put it on the production server.

    Yes, for the same reason too it seems. Although the VPS does host a personal domain, it also hosts two business class domains, and even though v70 is close to completion, if we go up a tier, we won't be able to come back down. At that point, we might be stable, for a bit. But then what comes after v70 isn't something that should be ran on a production server. In one of my previous jobs, before I was made head of it the IT department, my boss had me write this PHP code and we needed to use a 3rd party module. I suggested one, but he already had one picked out. I told him it was in Alpha testing stage and that wasn't a good idea. That's the reason right there that I was promoted and he was demoted. Over 1/4 of all transactions where just ending up in /dev/null (or the equivalent of whatever the IBM Websphere uses). And we were dealing with very large banks where consumers don't really have accounts, but other banks do. And there where literally millions of transactions a day occuring. It was pretty crazy.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post