cPanel reporting Service Down but its Up

Hello,

We did an Elasticsearch upgrade yesterday. Now chkserv is sending out alert emails that the service is down and it's trying to restart it.

Server server.removed.com
Primary IP Address IP removed
Service Name elasticsearch
Service Status failed ⛔
Notification The service “elasticsearch” appears to be down.
Service Check Method The system failed to connect to this service’s TCP/IP port.
Reason Service check failed to completeelasticsearch: [HTTP/1.0 400 Ba != HTTP/1.1 200 OK]
Number of Restart Attempts 2
Startup Log Nov 14 09:36:00 server.removed.com systemd[1]: Starting Elasticsearch...
Nov 14 09:36:01 server.removed.com systemd-entrypoint[117860]: [0.002s][warning][logging] Output options for existing outputs are ignored.
Nov 14 09:36:03 server.removed.com systemd-entrypoint[117860]: CompileCommand: dontinline java/lang/invoke/MethodHandle.setAsTypeCache bool dontinline = true
Nov 14 09:36:03 server.removed.com systemd-entrypoint[117860]: CompileCommand: dontinline java/lang/invoke/MethodHandle.asTypeUncached bool dontinline = true
Nov 14 09:36:03 server.removed.com systemd-entrypoint[117860]: CompileCommand: dontinline java/lang/invoke/MethodHandle.setAsTypeCache bool dontinline = true
Nov 14 09:36:03 server.removed.com systemd-entrypoint[117860]: CompileCommand: dontinline java/lang/invoke/MethodHandle.asTypeUncached bool dontinline = true
Nov 14 09:36:12 server.removed.com systemd[1]: Started Elasticsearch.
Memory Information
Used 39.42 GB
Available 22.87 GB
Installed 62.29 GB
Load Information 0.46 0.56 0.53
Uptime 1 hour, 5 minutes, and 34 seconds
IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 2.48 0.04 1.00 0.04 0.00 96.45 Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn nvme0n1 121.61 2683.88 364.96 10556529 1435513 loop0 1.83 1.42 7.92 5569 31154

When I check the status of the service via terminal, it reports up and I can verify that it is up and running.

[root@server ~]# /bin/systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-11-14 09:36:12 EST; 14min ago
Docs: https://www.elastic.co
Main PID: 117701 (java)
Tasks: 137 (limit: 408012)
Memory: 33.3G
CGroup: /system.slice/elasticsearch.service
├─117701 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/to>
├─117860 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouc>
└─119088 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 14 09:36:00 server.removed.com systemd[1]: Starting Elasticsearch...
Nov 14 09:36:01 server.removed.com systemd-entrypoint[117860]: [0.002s][warning][logging] Output options for existing outputs are ignored.
Nov 14 09:36:03 server.removed.com systemd-entrypoint[117860]: CompileCommand: dontinline java/lang/invoke/MethodHandle.setAsTypeCache bool dontinline = true
Nov 14 09:36:03 server.removed.com systemd-entrypoint[117860]: CompileCommand: dontinline java/lang/invoke/MethodHandle.asTypeUncached bool dontinline = true
Nov 14 09:36:03 server.removed.com systemd-entrypoint[117860]: CompileCommand: dontinline java/lang/invoke/MethodHandle.setAsTypeCache bool dontinline = true
Nov 14 09:36:03 server.removed.com systemd-entrypoint[117860]: CompileCommand: dontinline java/lang/invoke/MethodHandle.asTypeUncached bool dontinline = true
Nov 14 09:36:12 server.removed.com systemd[1]: Started Elasticsearch.

I think the line in the etc/chkserv.d/elasticsearch file may be incorrect.

service[elasticsearch]=9200,curl -XGET,HTTP/1.1 200 OK,/bin/systemctl start elasticsearch.service

I also tried

service[elasticsearch]=9200,curl -XGET,HTTP/1.0 400 Ba,/bin/systemctl start elasticsearch.service

When I reboot the server, the elasticsearch service starts fine.

Any help would be greatly appreciate`.

Thanks!

Andrew

November 14, 2024 16:44

Have a look at the chkservd log and see why it can't verify the service status exactly:

tail -f /var/log/chkservd.log

Andrew N. - cPanel Plesk VMWare Certified Professional
Do you need immediate assistance? 20 minutes response time!* Open a ticket
EmergencySupport - Professional Server Management and One-time Services

tommyxv

November 14, 2024 17:09

[root@server ~]# tail -f /var/log/chkservd.log
dnsadmin [[http_service_auth:1][check command:+][socket connect:+]]...
crond [[check command:+][socket connect:N/A]]...
cpsrvd [[http_service_auth:1][check command:N/A][socket connect:+]]...
cphulkd [[check command:+][socket connect:+]]...
cpgreylistd [[check command:N/A][socket connect:N/A]]...
cpdavd [[check command:+][socket connect:N/A]]...
cpanellogd [[check command:+][socket connect:N/A]]...
cpanel_php_fpm [[check command:+][socket connect:N/A]]...
apache_php_fpm [[check command:+][socket connect:N/A]]...Done
Service Check Finished

November 14, 2024 17:15
Edited

elasticsearch [Service check failed to completeelasticsearch: [HTTP/1.0 400 Ba != HTTP/1.1 200 OK]

This is the line in the etc/chkserv.d/elasticsearch/ file

cPRex Jurassic Moderator

November 14, 2024 18:24

Hey there! I am also assuming that everything is working well, and the chksrvd monitoring isn't correctly checking for the new service.

Since cPanel doesn't provide support for Elasticsearch I can't say for sure what may need to be adjusted, so I would recommend reviewing our guide on creating custom chkservd monitors here, as that gives you a few different options to check the service:

https://support.cpanel.net/hc/en-us/articles/1500000535122-How-to-add-a-new-service-for-chkservd-to-monitor

cPanel reporting Service Down but its Up

Comments

Didn't find what you were looking for?