Skip to main content

Apache down: 'lynx: Can't access startfile http://localhost/whm-server-status'

David_spm
I was just on one server and looking at apachetop when suddenly all sites went down and when I tried to restart apache I got this: Looking up localhost Making HTTP connection to localhost Sending HTTP request. HTTP request sent; waiting for response. HTTP/1.1 302 Found Data transfer complete HTTP/1.1 302 Found Using No chages, updates or configurations have been run at all. I have spent the last hr reading related threads here and have tried the following with no success: /scripts/upcp --force httpd -t (syntax ok) checked for syn flood attacks - all fine Plenty of resources, RAM & disk space available Restarting both HTTPD and PHP-FPM via WHm console seems to bring sites back but only briefly I also tried this (but it also did nothing): /scripts/rebuildhttpdconf Sorry, configuration data has not been successfully stored. Please execute the following commands: /usr/local/cpanel/bin/apache_conf_distiller --store-data --defaults touch /var/cpanel/conf/apache/success Execute the apache_conf_distiller without any flags to see its full usage. -bash-4.1$ sudo /usr/local/cpanel/bin/apache_conf_distiller --store-data --defaults Distilled successfully -bash-4.1$ sudo touch /var/cpanel/conf/apache/success A swift response would be greatly appreciated as Im out of ideas here and have 30 sites offline right now :|

Comments

9 comments

Date Votes
  • David_spm
    here's something else: sudo /scripts/restartsrv_apache Waiting for "httpd" to start ""waiting for "httpd" to initialize "finished. Service Status httpd (/usr/sbin/httpd -k start) is running as root with PID 6591 (pidfile+/proc check method). Startup Log [Thu Dec 06 07:04:48.523852 2018] [so:warn] [pid 6588:tid 47129664890208] AH01574: module status_module is already loaded, skipping Log Messages [Thu Dec 06 07:04:48.962164 2018] [mpm_worker:notice] [pid 6591:tid 47129664890208] AH00292: Apache/2.4.37 (cPanel) OpenSSL/1.0.2q mod_bwlimited/1.4 configured -- resuming normal operations [Thu Dec 06 07:04:48.705601 2018] [:notice] [pid 6588:tid 47129664890208] ModSecurity for Apache/2.9.2 (ModSecurity: Open Source Web Application Firewall) configured. [Thu Dec 06 06:55:51.615411 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/RESPONSE-980-CORRELATION.conf"> [line "37"> [id "980130"> [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): GET or HEAD Request with Body Content."> [tag "event-correlation"> [hostname "www.example.com"> [uri "/index.php"> [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8">, referer: https://www.example.com/ [Thu Dec 06 06:55:51.211763 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-949-BLOCKING-EVALUATION.conf"> [line "30"> [id "949110"> [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"> [severity "CRITICAL"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-generic"> [hostname "www.example.com"> [uri "/"> [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8">, referer: https://www.example.com/ [Thu Dec 06 06:55:51.210719 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Warning. Match of "rx ^0?$" against "REQUEST_HEADERS:Content-Length" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"> [line "143"> [id "920170"> [rev "1"> [msg "GET or HEAD Request with Body Content."> [data "247"> [severity "CRITICAL"> [ver "OWASP_CRS/3.0.0"> [maturity "9"> [accuracy "9"> [tag "application-multi"> [tag "language-multi"> [tag "platform-multi"> [tag "attack-protocol"> [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"> [tag "CAPEC-272"> [hostname "www.example.com"> [uri "/"> [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8">, referer: https://www.example.com/ Dec 6 07:04:48 server sudo: david : TTY=pts/0 ; PWD=/home/david ; USER=root ; COMMAND=/scripts/restartsrv_apache httpd started successfully.
    but I dont think these modsecurity warnings should be causing an Apache outage?
    0
  • David_spm
    In apache error_logs Im seeing things like this: [Thu Dec 06 08:08:05.141717 2018] [mpm_worker:alert] [pid 797:tid 140122596484864] (11)Resource temporarily unavailable: AH03142: apr_thread_create: unable to create worker thread but thats only when I restart apache and its running (briefly) until it crashes again minutes later. then Im seeing other errors like this: [Thu Dec 06 08:15:31.785302 2018] [proxy_fcgi:error] [pid 21009:tid 47421194983168] [client 46.229.168.142:52974] AH01071: Got error 'Primary script unknown\n' [Thu Dec 06 08:14:51.504584 2018] [ssl:warn] [pid 21001:tid 47421001491808] AH01909: server.myserver.com:443:0 server certificate does NOT include an ID which matches the server name All my httpd settings are at the defaults though and there has never been any reason to change them: StartServers 5 MinSpareServers 5 MaxSpareServers 10 ServerLimit 256 MaxRequestWorkers 150 MaxConnectionsPerChild 10000 KeepAlive On KeepAliveTimeout 5 MaxKeepAliveRequests 768 Timeout 300 Im also using autossl, and for a long time without any issues
    0
  • David_spm
    been trying some more things, it seems that port 80 keeps closing after httpd is restarted and also its blocked to localhost: sudo netstat -tulpn | grep 80 tcp 0 0 0.0.0.0:2080 0.0.0.0:* LISTEN 19826/cpdavd - acce udp 0 0 fe80::225:90ff:fe77:c824:123 :::* 21553/ntpd udp 0 0 fe80::225:90ff:fe77:c825:123 :::* 21553/ntpd -bash-4.1$ curl -v 127.0.0.1 * Rebuilt URL to: 127.0.0.1/ * Trying 127.0.0.1... * connect to 127.0.0.1 port 80 failed: Connection refused * Failed to connect to 127.0.0.1 port 80: Connection refused * Closing connection 0 curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused -bash-4.1$ telnet localhost 80 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused -bash-4.1$ sudo nmap -sS 127.0.0.1 -p 80 Starting Nmap 5.51 ( Nmap: the Network Mapper - Free Security Scanner ) at 2018-12-06 10:34 EST Nmap scan report for localhost (127.0.0.1) Host is up (0.000070s latency). PORT STATE SERVICE 80/tcp closed http Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds -bash-4.1$ sudo iptables -t nat -nvL Chain PREROUTING (policy ACCEPT 134K packets, 7860K bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 115K packets, 8260K bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 115K packets, 8266K bytes) pkts bytes target prot opt in out source destination 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 25,26,465,587 owner GID match 206 222 11544 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 25,26,465,587 owner GID match 12 0 0 RETURN tcp -- * * 0.0.0.0/0 127.0.0.1 multiport dports 25,26,465,587 owner UID match 201 166 8632 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 25,26,465,587 owner UID match 0 0 0 REDIRECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 25,26,465,587
    I cant see anything in iptables that could be causing this though.... Very much open to suggestions right now!
    0
  • cPanelMichael
    Hello @David_spm, Can you open a
    0
  • David_spm
    I went ahead and completed the support ticket even though the prepare server access stage never worked, however I dont have ssh keys on and provided the root pw so you should be able to access the server correct?
    0
  • David_spm
    I got a pretty swift response from support, Im posting what they said here for future reference in case anyone else needs it: I think the problem was a custom apache include file that forced all requests to redirect to https Specifically, in /etc/apache2/conf.d/includes/pre_main_global.conf : ============= RewriteEngine On RewriteCond %{HTTPS} !=on RewriteRule ^/?(.*) / 302 Found

    Found

    The document has moved ">here.

    ============ Or this: ============== [20:45:58 server root@10905649 ~]cPs# apachectl fullstatus ELinks: SSL error ============== Or this: ================ [20:51:48 server root@10905649 ~]cPs# lynx --dump localhost/whm-server-status Looking up localhost Making HTTP connection to localhost Sending HTTP request. HTTP request sent; waiting for response. HTTP/1.1 302 Found Data transfer complete HTTP/1.1 302 Found Using ================== Apache itself was able to run fine -- but any port 80 connections were forced to reconnect on port 443. ################ Why would this just happen now? ################ When I logged in, I checked the timestamps on the include file that was causing the issue. It was modified this morning: ================== [20:53:32 server root@10905649 ~]cPs# stat /etc/apache2/conf.d/includes/pre_main_global.conf File: `/etc/apache2/conf.d/includes/pre_main_global.conf' Size: 509 Blocks: 8 IO Block: 4096 regular file Device: 803h/2051d Inode: 14293732 Links: 1 Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-12-06 08:22:44.249566194 -0500 Modify: 2018-12-06 08:22:44.249566194 -0500 Change: 2018-12-06 08:22:44.249566194 -0500 ================== I suspect the rules were only recently added. So that seems to have fixed it BUT I didnt add those rewrite rules and it looks like the time they were added was when I ran apachetop. Does that add rules itself? This is solved now anyway.
    0
  • cPanelMichael
    So that seems to have fixed it BUT I didnt add those rewrite rules and it looks like the time they were added was when I ran apachetop. Does that add rules itself?

    ============= RewriteEngine On RewriteCond %{HTTPS} !=on RewriteRule ^/?(.*) . Thank you.
    0
  • David_spm
    OK thanks, no-one else has any kind of access. I had some problems with autossl not renewing certs on two sites a few weeks ago which I tried to fix and I think I might have added those rewrite rules but like I said that was weeks ago so it doesn't explain why these apache problems happened just now. Thanks for the input though.
    0
  • cPanelMichael
    I had some problems with autossl not renewing certs on two sites a few weeks ago which I tried to fix and I think I might have added those rewrite rules but like I said that was weeks ago so it doesn't explain why these apache problems happened just now.

    Hi @David_spm, Here are a couple of cases included in the most recent
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post