Skip to main content

CPHulk locked all access

kcorbin
We enabled country blocking on our server and CPhulk blocked everything, web and ssh, even blocked access to hosted sites. I tried the following; /usr/local/cpanel/bin/cphulk_pam_ctl --disable /usr/local/cpanel/etc/init/stopcphulkd
but cphulk would restart. I then tried; whmapi1 configureservice service=cphulkd enabled=0 monitord=0 iptables -F cphulk && /usr/local/cpanel/3rdparty/bin/sqlite3 /var/cpanel/hulkd/cphulk.sqlite "DELETE FROM login_track;"
and was able to get access to WHM and SSH again, so I have a few questions. What did I do wrong? I went to Home > Security Center > Cphulk Brute force protection and selected every nation except the US and chose blacklist. I didn't chose anything else, but I'm sure I probably missed something. Is there a way to reset cphulk through cli back to its defaults? I would like to restart it and use it again. What can I do to prevent this from happening again? Thank you all.

Comments

12 comments

Date Votes
  • cPanelLauren
    Hello, As far as managing cPHulk we have the following documentation that might be helpful: WHM API 1 Functions - set_cphulk_config_key - Developer Documentation - cPanel Documentation You can still make changes to the configuration in WHM with cPHulk's IP management disabled though and it might be easiest to make those modifications there.
    0
  • kcorbin
    Lauren thank you for the reply. Here is the last few entries in the cphulkd_errors.log [2020-02-01 21:30:38 -0500] die [cPhulkd] Timeout while waiting for response at /usr/local/cpanel/Cpanel/Hulkd.pm line 487. Cpanel::Hulkd::die(Cpanel::Hulkd=HASH(0xb02130), "Timeout while waiting for response") called at /usr/local/cpanel/Cpanel/Hulkd.pm line 417 Cpanel::Hulkd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 289 eval {...} called at /usr/local/cpanel/Cpanel/Hulkd/Processor.pm line 289 Cpanel::Hulkd::Processor::run(Cpanel::Hulkd::Processor=HASH(0xe00cf8), undef) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 423 Cpanel::Hulkd::__ANON__() called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 97 eval {...} called at /usr/local/cpanel/3rdparty/perl/528/lib/perl5/cpanel_lib/Try/Tiny.pm line 88 Try::Tiny::try(CODE(0xd50888), Try::Tiny::Catch=REF(0xc655a8)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 427 Cpanel::Hulkd::handle_one_connection(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::INET=GLOB(0xc65b18), undef) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 594 Cpanel::Hulkd::_handle_accepted_socket_and_reset_idleloops(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::INET=GLOB(0xc65b18)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 341 Cpanel::Hulkd::main_loop(Cpanel::Hulkd=HASH(0xb02130), Cpanel::Socket::UNIX=GLOB(0xb3a718), Cpanel::Socket::INET=GLOB(0xb3aa30)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 244 Cpanel::Hulkd::processor_run(Cpanel::Hulkd=HASH(0xb02130)) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 159 Cpanel::Hulkd::__ANON__(__CPANEL_HIDDEN__) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 170 Cpanel::Hulkd::launcher(Cpanel::Hulkd=HASH(0xb02130), 0) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 139 Cpanel::Hulkd::start_daemon(Cpanel::Hulkd=HASH(0xb02130), 0) called at /usr/local/cpanel/Cpanel/Hulkd.pm line 93 Cpanel::Hulkd::run_daemon(Cpanel::Hulkd=HASH(0xb02130)) called at libexec/cphulkd.pl line 32 [2020-02-01 21:30:38 -0500] info [cPhulkd] The system encountered an error while processing a request: exit level [die] [pid=20503] (Timeout while waiting for response) [2020-02-03 14:38:31 -0500] info [cPhulkd] The service:[login] unexpectedly sent the invalid remote IP address:[0.0.0.0]. (Consider disabling DNS resolution for this service) [2020-02-03 14:38:31 -0500] info [cPhulkd] The service:[login] unexpectedly sent the invalid remote IP address:[0.0.0.0]. (Consider disabling DNS resolution for this service)
    Here is some of cphulkd.log around the time the first user blocks happened. [2020-02-03 14:02:18 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 11073 [2020-02-03 14:07:10 -0500] info [cPhulkd] processor startup with pid 9091 [2020-02-03 14:07:10 -0500] info [cPhulkd] DB processor startup with pid 13789 [2020-02-03 14:08:04 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 13789 [2020-02-03 14:08:04 -0500] info [cPhulkd] processor startup with pid 9091 [2020-02-03 14:08:04 -0500] info [cPhulkd] DB processor startup with pid 13889 [2020-02-03 14:12:33 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[cpaneld] [Local IP Address]=[74.50.46.249] [Local Port]=[2083] [Remote IP Address]=[195.24.207.249] [Remote Port]=[33544] [Authentication Database]=[system] [Username]=[testsite] (1/30 failures) (blocked until [Tue Feb 4 19:12:33 2020 UTC/Tue Feb 4 14:12:33 2020 LOCAL]) [2020-02-03 14:12:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[cpaneld] [Local IP Address]=[74.50.46.249] [Local Port]=[2083] [Remote IP Address]=[195.24.207.249] [Remote Port]=[33544] [Authentication Database]=[system] [Username]=[testsite] [2020-02-03 14:23:04 -0500] info [cPhulkd] DB processor shutdown via SIGTERM with pid 13889 [2020-02-03 14:36:33 -0500] info [cPhulkd] processor startup with pid 9091 [2020-02-03 14:36:33 -0500] info [cPhulkd] DB processor startup with pid 17495 [2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[cruckrie] (1/30 failures) (blocked until [Tue Feb 4 19:36:33 2020 UTC/Tue Feb 4 14:36:33 2020 LOCAL]) [2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[XXXXXXX] [2020-02-03 14:37:15 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb 4 19:37:15 2020 UTC/Tue Feb 4 14:37:15 2020 LOCAL]) [2020-02-03 14:37:15 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] [2020-02-03 14:37:26 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb 4 19:37:26 2020 UTC/Tue Feb 4 14:37:26 2020 LOCAL]) [2020-02-03 14:37:26 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] [2020-02-03 14:37:27 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.31] [Authentication Database]=[system] [Username]=[cluescher] (1/30 failures) (blocked until [Tue Feb 4 19:37:27 2020 UTC/Tue Feb 4 14:37:27 2020 LOCAL]) [2020-02-03 14:37:27 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.31] [Authentication Database]=[system] [Username]=[XXXXXXX] [2020-02-03 14:38:09 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] (1/30 failures) (blocked until [Tue Feb 4 19:38:09 2020 UTC/Tue Feb 4 14:38:09 2020 LOCAL]) [2020-02-03 14:38:09 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.5] [Authentication Database]=[system] [Username]=[root] [2020-02-03 14:43:16 -0500] info [cPhulkd] Login Blocked: IP reached maximum auth failures for a one day block [Service]=[pure-ftpd] [Local IP Address]=[74.50.46.253] [Local Port]=[21] [Remote IP Address]=[78.131.193.167] [Authentication Database]=[system] [Username]=[bluegrasswireless.com] (1/30 failures) (blocked until [Tue Feb 4 19:43:16 2020 UTC/Tue Feb 4 14:43:16 2020 LOCAL]) [2020-02-03 14:43:16 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[pure-ftpd] [Local IP Address]=[74.50.46.253] [Local Port]=[21] [Remote IP Address]=[78.131.193.167] [Authentication Database]=[system] [Username]=[bluegrasswireless.com]
    This is about the time I turned on country blocking, I have blocked out some usernames. Thank you.
    0
  • cPanelLauren
    Hello, This is interesting because it's seeing your private IP address - 10.140.140.31 is not a public IP address. This means it's an IP that should/would only be accessible within your network. I assume your server is NAT routed, based on that I wonder if it's being properly recognized. Without listing your public IP address can you tell me if this file exists and properly maps your Private IPs to your Public IPs? cat /var/cpanel/cpnat
    0
  • kcorbin
    There are no private IP's in the cpnat file only the public IP's assigned to the server. The main reason I came to the forums with the issue is that it blocked private IPs too. No matter were anyone tried to login in from it would block the IP and maybe the account too, I couldn't tell at the time. If it wasn't for the vSphere console we would have lost all access to the server. Let me add these were not repeated failed logins, accounts were blocked even after a successful login. Their IP's were also blocked from hosted websites too. Thank you for your help.
    0
  • cPanelLauren
    Hello, I believe there's a bit of a misunderstanding and to clarify: on 1:1 NAT Routed systems (the only NAT configuration we support) the private IP address like 10.140.140.31 is mapped to a public IP address. The file at /var/cpanel/cpnat
    shows this mapping and is a reference for cPanel to recognize whether or not the system is NAT routed. cPHulk should not be seeing the private IP address to block and this is why I asked the question, in that output you provided the IP is a private IP and should never be listed. What I believe is happening is one of two things: 1. The NAT routing on the server is misconfigured 2. cPanel isn't seeing that the system is NAT routed - you could confirm if this is the case fairly easily by running the following: /scripts/build_cpnat
    If you're not NAT routed at all, this becomes a bit confusing as it would even further the point that something is misconfigured as private IP's aren't able to connect outside of the local network.
    0
  • kcorbin
    The server is not NAT routed and only has public IP's assigned to it, I believe we sys admins are to public IP's. I did run the script you provided and got the following. info [build_cpnat] All publicly routeable addresses are the same as the local address. Not a NAT system.
    The rest of the output was all public IP's, as for local network, we frequently SSH to our servers from private IP's. We are the ISP if that helps any so we have total control routing wise. Thank you.
    0
  • cPanelLauren
    That's interesting if you're accessing from a local network it would be the only instance in which it would make sense to see the private IP address, but it still doesn't explain why cPhulk would block it unless it sees the internal IP from originating from a blocked Country. I'd suggest opening a ticket so our analysts can take a look at the server, to be honest - I don't think it'd be a simple task to find out what's occurring without being able to see it.
    0
  • kcorbin
    Thank you I have created ticket number 93452104.
    0
  • cPanelLauren
    Hello, Thank you for that, I added the following note to the ticket as well:
    • They access SSH with an internal IP 10.140.140.28
    • They enabled CC blocking with cPHulkd
    • With CC blocking enabled their internal IP address is being blocked:
    [2020-02-03 14:36:33 -0500] info [cPhulkd] Login Blocked: The country is blacklisted. [Service]=[sshd] [Remote IP Address]=[10.140.140.28] [Authentication Database]=[system] [Username]=[XXXXXXX]
    I'll update here when there is more information on this.
    0
  • cPanelLauren
    Hi @kcorbin The analyst who spent a good portion of yesterday investigating this noted that she found an internal case which indicates that some IP's in the 10.X.X.X range are considered to be part of "Unknown Region" (ZZ) because of this when you block all except for the US (or another all but X method) it's also including anything that doesn't have a region assigned (unknown). Internal IP's wouldn't have a region assigned to them so the assumption is that it's blocking them for you because of this. She asked if you could un-include the "Unknown Region" (ZZ) selection when blocking countries, and let her know when it was done so that she could continue testing/or let us know if you continue to have the issue. Thanks!
    0
  • cPanelLauren
    Just as a final follow up, I wanted to note that the solution for this does as of right now appear to have been to add the unknown country code into the whitelist for cPHulk. I checked in on the ticket and it looks like things have been good since making that change. If you do run into any further issues with this or with cPhulk please don't hesitate to contact us again.
    0
  • kcorbin
    After working with cPanel support I can confirm that removing country code ZZ from the blacklist has fixed the issue. I would like to thank everyone at cPanel for the assistance and support. Thanks again.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post