Server hangs during nightly maintenance

My server is running on an AWS t3.small instance. I only have a few websites with not a lot of traffic. CPU usage is always low. Several mornings I wake up to find my websites have been offline all night. I cannot access the server through SSH and all I can do it use the AWS console to reboot. After a reboot it's all working fine again. After a reboot, I usually get a bunch of emails through from CPanel about high loads, but after weeks of looking I cannot figure out what is causing this to happen. 01:50 - Email from UptimeRobot saying my websites are down 03:04 - cPanel Monitoring: Failed MySQL 03:10 - cPanel Monitoring: MySQL recovered 03:10 - cPanel Monitoring: Failed spam 03:19 - cPanel Monitoring: spamd recovered 03:54 - Excessive processes running under user Notice time difference in email. Time: Mon Apr 24 00:04:08 2023 +0000 Account: Process Count: 14 (Not killed) Process Information: User: PID:90151 PPID:65705 Run Time:1295(secs) Memory:584316(kb) RSS:85488(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90152 PPID:23760 Run Time:1295(secs) Memory:584316(kb) RSS:88740(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90211 PPID:12586 Run Time:1270(secs) Memory:584316(kb) RSS:86360(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90212 PPID:49731 Run Time:1270(secs) Memory:584316(kb) RSS:85744(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90370 PPID:12581 Run Time:1249(secs) Memory:505768(kb) RSS:11136(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90385 PPID:90208 Run Time:1206(secs) Memory:271004(kb) RSS:9220(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90386 PPID:47981 Run Time:1206(secs) Memory:271004(kb) RSS:7992(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90470 PPID:65705 Run Time:0(secs) Memory:376(kb) RSS:8(kb) exe:/home/virtfs//usr/local/cpanel/cgi-sys/ea-php81 cmd:/usr/local/cpanel/cgi-sys/ea-php81 User: PID:90471 PPID:12581 Run Time:0(secs) Memory:376(kb) RSS:8(kb) exe:/home/virtfs//usr/local/cpanel/cgi-sys/ea-php81 cmd:/usr/local/cpanel/cgi-sys/ea-php81 User: PID:90472 PPID:47981 Run Time:0(secs) Memory:376(kb) RSS:8(kb) exe:/home/virtfs//usr/local/cpanel/cgi-sys/ea-php81 cmd:/usr/local/cpanel/cgi-sys/ea-php81 User: PID:90473 PPID:23760 Run Time:0(secs) Memory:376(kb) RSS:8(kb) exe:/home/virtfs//usr/local/cpanel/cgi-sys/ea-php81 cmd:/usr/local/cpanel/cgi-sys/ea-php81 User: PID:90474 PPID:90208 Run Time:0(secs) Memory:376(kb) RSS:8(kb) exe:/home/virtfs//usr/local/cpanel/cgi-sys/ea-php81 cmd:/usr/local/cpanel/cgi-sys/ea-php81 User: PID:90475 PPID:49731 Run Time:0(secs) Memory:376(kb) RSS:8(kb) exe:/home/virtfs//usr/local/cpanel/cgi-sys/ea-php81 cmd:/usr/local/cpanel/cgi-sys/ea-php81 User: PID:90476 PPID:12586 Run Time:0(secs) Memory:376(kb) RSS:8(kb) exe:/home/virtfs//usr/local/cpanel/cgi-sys/ea-php81 cmd:/usr/local/cpanel/cgi-sys/ea-php81
03:54 - Excessive resource usage (90152 (Parent PID:23760)) Time: Mon Apr 24 00:55:32 2023 +0000 Account: Resource: Process Time Exceeded: 1831 > 1800 (seconds) Executable: /home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi Command Line: /opt/cpanel/ea-php81/root/usr/bin/php-cgi PID: 90152 (Parent PID:23760) Killed: No
03:54 - Excessive resource usage: (90151 (Parent PID:65705)) Time: Mon Apr 24 00:55:12 2023 +0000 Account: Resource: Process Time Exceeded: 1831 > 1800 (seconds) Executable: /home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi Command Line: /opt/cpanel/ea-php81/root/usr/bin/php-cgi PID: 90151 (Parent PID:65705) Killed: No
03:54 - cPanel Monitoring: HANG: ?: chkservd chkservd The previous service check was still running (1306 second). It was terminated. dnsadmin [Service Check Started exim [[check command:+][socket connect:+]]... httpd [[check command:N/A][socket connect:+]]... imap [[socket_service_auth:1][check command:+][socket connect:+]]... interval [[check command:N/A][socket connect:N/A]]... ipaliases [[check command:+][socket connect:N/A]]... lfd [[check command:+][socket connect:N/A]]... lmtp [[check command:+][socket connect:+]]... mailman [[check command:+][socket connect:N/A]]... mysql [[check command:+][socket connect:N/A]]... named [[check command:+][socket connect:N/A]]... nscd [[check command:+][socket connect:N/A]]... p0f [[check command:N/A][socket connect:N/A]]... pop [[check command:+][socket connect:+]]... rsyslogd [[check command:+][socket connect:N/A]]... spamd [[check command:+][socket connect:N/A]]... sshd [[check command:+][socket connect:N/A]]... queueprocd [[check command:+][socket connect:N/A]]... [2023-04-23 23:42:40 +0000] Service check .... [2023-04-23 23:42:39 +0000] OOM check ......OOM Event:[anon_rss=289212kB,file_rss=0kB,is_cgroup=0,pid=89946,seconds_since_boot=97275.892715,time=1682293338,total_vm=710448kB,uid=0,user=root].....Skipped OOM Notification (too soon)...... Done [2023-04-23 23:42:38 +0000] Disk check .... /var/tmp (/var/tmp) [6.77%] ... /tmp (/tmp) [6.77%] ... / (/) [62.2%] ... {status:ok} ... Done [2023-04-23 23:42:37 +0000] Loading list of mount points to ignore... ignoring mount points that match: virtfs|cagefs... Done Loading services .....apache_php_fpm....cpanellogd....cpdavd....cphulkd....cpsrvd....crond....dnsadmin....exim....httpd....imap....ipaliases....lfd....lmtp....mailman....mysql....named....nscd....pop....queueprocd....rsyslogd....spamd....sshd..Done Service Check Started Service Check Finished apache_php_fpm [[check command:+][socket connect:N/A]]...Done cpanel_php_fpm [[check command:N/A][socket connect:N/A]]... cpanellogd [[check command:+][socket connect:N/A]]... cpdavd [[check command:+][socket connect:N/A]]... cpgreylistd [[check command:N/A][socket connect:N/A]]... cphulkd [[check command:+][socket connect:+]]... cpsrvd [[http_service_auth:1][check command:N/A][socket connect:+]]... crond [[check command:+][socket connect:N/A]]... dnsadmin [[http_service_auth:1][check command:+][socket connect:+]]... exim [[check command:+][socket connect:+]]... httpd [[check command:N/A][socket connect:+]]... imap [[socket_service_auth:1][check command:+][socket connect:+]]... ipaliases [[check command:+][socket connect:N/A]]... lfd [[check command:+][socket connect:N/A]]... lmtp [[check command:+][socket connect:+]]... mailman [[check command:+][socket connect:N/A]]... mysql [[check command:+][socket connect:N/A]]... named [[check command:+][socket connect:N/A]]... nscd [[check command:+][socket connect:N/A]]... p0f [[check command:N/A][socket connect:N/A]]... pop [[check command:+][socket connect:+]]... rsyslogd [[check command:+][socket connect:N/A]]... spamd [[check command:+][socket connect:N/A]]... sshd [[check command:+][socket connect:N/A]]... queueprocd [[check command:+][socket connect:N/A]]... [2023-04-23 23:36:17 +0000] Service check .... [2023-04-23 23:36:17 +0000] OOM check ....Done [2023-04-23 23:36:17 +0000] Disk check .... /var/tmp (/var/tmp) [6.01%] ... /tmp (/tmp) [6.01%] ... / (/) [62.26%] ... {status:ok} ... Done [2023-04-23 23:36:17 +0000] Loading list of mount points to ignore... ignoring mount points that match: virtfs|cagefs... Done Loading services .....apache_php_fpm....cpanellogd....cpdavd....cphulkd....cpsrvd....crond....dnsadmin....exim....httpd....imap....ipaliases....lfd....lmtp....mailman....mysql....named....nscd....pop....queueprocd....rsyslogd....spamd....sshd..Done Service Check Started Service Check Finished apache_php_fpm [[check command:+][socket connect:N/A]]...Done cpanel_php_fpm [[check command:N/A][socket connect:N/A]]... cpanellogd [[check command:+][socket connect:N/A]]... cpdavd [[check command:+][socket connect:N/A]]... cpgreylistd [[check command:N/A][socket connect:N/A]]... cphulkd [[check command:+][socket connect:+]]... cpsrvd [[http_service_auth:1][check command:N/A][socket connect:+]]... crond [[check command:+][socket connect:N/A]]... dnsadmin [[http_service_auth:1][check command:+][socket connect:+]]... exim [[check command:+][socket connect:+]]... httpd [[check command:N/A][socket connect:+]]... imap [[socket_service_auth:1][check command:+][socket connect:+]]... ipaliases [[check command:+][socket connect:N/A]]... lfd [[check command:+][socket connect:N/A]]... lmtp [[check command:+][socket connect:+]]... mailman [[check command:+][socket connect:N/A]]... mysql [[check command:+][socket connect:N/A]]... named [[check command:+][socket connect:N/A]]... nscd [[check command:+][socket connect:N/A]]... p0f [[check command:N/A][socket connect:N/A]]... pop [[check command:+][socket connect:+]]... rsyslogd [[check command:+][socket connect:N/A]]... spamd [[check command:+][socket connect:N/A]]... sshd [[check command:+][socket connect:N/A]]... queueprocd [[check command:+][socket connect:N/A]]... [2023-04-23 23:30:03 +0000] Service check .... [2023-04-23 23:30:03 +0000] OOM check ....Done [2023-04-23 23:30:03 +0000] Disk check .... /tmp (/tmp) [6.01%] ... / (/) [62.26%] ... /var/tmp (/var/tmp) [6.01%] ... {status:ok} ... Done [2023-04-23 23:30:03 +0000] Loading list of mount points to ignore... ignoring mount points that match: virtfs|cagefs... Done Loading services .....apache_php_fpm....cpanellogd....cpdavd....cphulkd....cpsrvd....crond....dnsadmin....exim....httpd....imap....ipaliases....lfd....lmtp....mailman....mysql....named....nscd....pop....queueprocd....rsyslogd....spamd....sshd..Done Service Check Started Service Check Finished apache_php_fpm [[check command:+][socket connect:N/A]]...Done cpanel_php_fpm [[check command:N/A][socket connect:N/A]]... cpanellogd [[check command:+][socket connect:N/A]]... cpdavd [[check command:+][socket connect:N/A]]... cpgreylistd [[check command:N/A][socket connect:N/A]]... cphulkd [[check command:+][socket connect:+]]... cpsrvd [[http_service_auth:1][check command:N/A][socket connect:+]]... crond [[check command:+][socket connect:N/A]]... dnsadmin [[http_service_auth:1][check command:+][socket connect:+]]... exim [[check command:+][socket connect:+]]... httpd [[check command:N/A][socket connect:+]]... imap [[socket_service_auth:1][check command:+][socket connect:+]]... ipaliases [[check command:+][socket connect:N/A]]... lfd [[check command:+][socket connect:N/A]]... lmtp [[check command:+][socket connect:+]]... mailman [[check command:+][socket connect:N/A]]... mysql [[check command:+][socket connect:N/A]]... named [[check command:+][socket connect:N/A]]... nscd [[check command:+][socket connect:N/A]]... p0f [[check command:N/A][socket connect:N/A]]... pop [[check command:+][socket connect:+]]... rsyslogd [[check command:+][socket connect:N/A]]... spamd [[check command:+][socket connect:N/A]]... sshd [[check command:+][socket connect:N/A]]... queueprocd [[check command:+][socket connect:N/A]]... [2023-04-23 23:23:57 +0000] Service check .... [2023-04-23 23:23:57 +0000] OOM check ....Done [2023-04-23 23:23:57 +0000] Disk check .... /var/tmp (/var/tmp) [6.01%] ... /tmp (/tmp) [6.01%] ... / (/) [62.26%] ... {status:ok} ... Done [2023-04-23 23:23:57 +0000] Loading list of mount points to ignore... ignoring mount points that match: virtfs|cagefs... Done Loading services .....apache_php_fpm....cpanellogd....cpdavd....cphulkd....cpsrvd....crond....dnsadmin....exim....httpd....imap....ipaliases....lfd....lmtp....mailman....mysql....named....nscd....pop....queueprocd....rsyslogd....spamd....sshd..Done Service Check Started Service Check Finished apache_php_fpm [[check command:+][socket connect:N/A]]...Done cpanel_php_fpm [[check command:N/A][socket connect:N/A]]... cpanellogd [[check command:+][socket connect:N/A]]... cpdavd [[check command:+][socket connect:N/A]]... cpgreylistd [[check command:N/A][socket connect:N/A]]... cphulkd [[check command:+][socket connect:+]]... cpsrvd [[http_service_auth:1][check command:N/A][socket connect:+]]...
tailwatchd [1580] [2023-04-24 00:04:10 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682294400 [1580] [2023-04-24 00:04:07 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 23:40:19 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers [1580] [2023-04-23 23:40:19 +0000] [Cpanel::TailWatch::Eximstats] Loading email sending limits from 1682290800 - 1682294400 [1580] [2023-04-23 23:40:19 +0000] [Cpanel::TailWatch] [INFO] tailwatch saving positions and reloading configuration on SIG [88401] [2023-04-23 23:26:20 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 23:26:20 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 23:07:40 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers [1580] [2023-04-23 23:07:40 +0000] [Cpanel::TailWatch] [INFO] tailwatch saving positions and reloading configuration on SIG [1580] [2023-04-23 23:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682290800 [86933] [2023-04-23 22:56:18 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 22:56:18 +0000] [Cpanel::TailWatch] Updating jails [85411] [2023-04-23 22:26:17 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 22:26:17 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 22:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682287200 [83832] [2023-04-23 21:56:16 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 21:56:15 +0000] [Cpanel::TailWatch] Updating jails [82442] [2023-04-23 21:26:14 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 21:26:14 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 21:10:52 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers [1580] [2023-04-23 21:10:52 +0000] [Cpanel::TailWatch] [INFO] tailwatch saving positions and reloading configuration on SIG [1580] [2023-04-23 21:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682283600 [80848] [2023-04-23 20:56:13 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 20:56:13 +0000] [Cpanel::TailWatch] Updating jails [79371] [2023-04-23 20:26:11 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 20:26:11 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 20:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682280000 [77844] [2023-04-23 19:56:10 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 19:56:09 +0000] [Cpanel::TailWatch] Updating jails [76460] [2023-04-23 19:26:08 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 19:26:08 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 19:14:05 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers [1580] [2023-04-23 19:14:05 +0000] [Cpanel::TailWatch] [INFO] tailwatch saving positions and reloading configuration on SIG [1580] [2023-04-23 19:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682276400 [74858] [2023-04-23 18:56:06 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 18:56:06 +0000] [Cpanel::TailWatch] Updating jails [73309] [2023-04-23 18:26:05 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 18:26:05 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 18:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682272800 [71670] [2023-04-23 17:56:03 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 17:56:02 +0000] [Cpanel::TailWatch] Updating jails [70107] [2023-04-23 17:26:01 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 17:26:00 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 17:00:35 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers [1580] [2023-04-23 17:00:35 +0000] [Cpanel::TailWatch] [INFO] tailwatch saving positions and reloading configuration on SIG [1580] [2023-04-23 17:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682269200 [68530] [2023-04-23 16:55:59 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 16:55:59 +0000] [Cpanel::TailWatch] Updating jails [66813] [2023-04-23 16:25:58 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 16:25:57 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 16:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682265600 [65342] [2023-04-23 15:55:56 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 15:55:56 +0000] [Cpanel::TailWatch] Updating jails [63687] [2023-04-23 15:25:55 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 15:25:54 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 15:03:46 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers [1580] [2023-04-23 15:03:46 +0000] [Cpanel::TailWatch] [INFO] tailwatch saving positions and reloading configuration on SIG [1580] [2023-04-23 15:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682262000 [62275] [2023-04-23 14:55:53 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 14:55:53 +0000] [Cpanel::TailWatch] Updating jails [60824] [2023-04-23 14:25:52 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 14:25:52 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 14:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682258400 [59222] [2023-04-23 13:55:50 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 13:55:50 +0000] [Cpanel::TailWatch] Updating jails [57782] [2023-04-23 13:25:49 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 13:25:49 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 13:06:57 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers [1580] [2023-04-23 13:06:57 +0000] [Cpanel::TailWatch] [INFO] tailwatch saving positions and reloading configuration on SIG [1580] [2023-04-23 13:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682254800 [56063] [2023-04-23 12:55:47 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 12:55:47 +0000] [Cpanel::TailWatch] Updating jails [54572] [2023-04-23 12:25:46 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 12:25:46 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 12:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682251200 [52926] [2023-04-23 11:55:44 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 11:55:44 +0000] [Cpanel::TailWatch] Updating jails [51462] [2023-04-23 11:25:43 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 11:25:42 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 11:09:55 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers [1580] [2023-04-23 11:09:55 +0000] [Cpanel::TailWatch] [INFO] tailwatch saving positions and reloading configuration on SIG [1580] [2023-04-23 11:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682247600 [50028] [2023-04-23 10:55:41 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 10:55:41 +0000] [Cpanel::TailWatch] Updating jails [48425] [2023-04-23 10:25:39 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 10:25:39 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 10:00:00 +0000] [Cpanel::TailWatch::Eximstats] Resetting email limits to new starttime of 1682244000 [46921] [2023-04-23 09:55:38 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 09:55:38 +0000] [Cpanel::TailWatch] Updating jails [45307] [2023-04-23 09:25:36 +0000] [Cpanel::TailWatch] Finished updating jails [1580] [2023-04-23 09:25:36 +0000] [Cpanel::TailWatch] Updating jails [1580] [2023-04-23 09:13:08 +0000] [Cpanel::TailWatch] [INFO] Flushing all readers
03:54 - High 5 minute load average alert - 117.82 (Logs available) 03:54 - cPanel Monitoring: HANG: ?: chkservd 03:54 - Excessive resource usage (90211 (Parent PID:12586)) 03:54 - Excessive processes running under user Time: Mon Apr 24 00:04:29 2023 +0000 Account: Process Count: 14 (Not killed) Process Information: User: PID:90151 PPID:65705 Run Time:1317(secs) Memory:590460(kb) RSS:94936(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90152 PPID:23760 Run Time:1317(secs) Memory:590460(kb) RSS:94508(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90211 PPID:12586 Run Time:1292(secs) Memory:590460(kb) RSS:95192(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90212 PPID:49731 Run Time:1292(secs) Memory:590460(kb) RSS:94676(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90370 PPID:12581 Run Time:1271(secs) Memory:530864(kb) RSS:38096(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90385 PPID:90208 Run Time:1228(secs) Memory:526768(kb) RSS:31764(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90386 PPID:47981 Run Time:1228(secs) Memory:526768(kb) RSS:31720(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90470 PPID:65705 Run Time:21(secs) Memory:526776(kb) RSS:31528(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90471 PPID:12581 Run Time:21(secs) Memory:526776(kb) RSS:31692(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90472 PPID:47981 Run Time:21(secs) Memory:526776(kb) RSS:31440(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90473 PPID:23760 Run Time:21(secs) Memory:526776(kb) RSS:31692(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90474 PPID:90208 Run Time:21(secs) Memory:526776(kb) RSS:31664(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90475 PPID:49731 Run Time:21(secs) Memory:526776(kb) RSS:31748(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi User: PID:90476 PPID:12586 Run Time:21(secs) Memory:526776(kb) RSS:31500(kb) exe:/home/virtfs//opt/cpanel/ea-php81/root/usr/bin/php-cgi cmd:/opt/cpanel/ea-php81/root/usr/bin/php-cgi
It goes on like this with a few more. Can anyone shed any light on what's happening? My thinking is it's part of the cPanel update process?

cPRex Jurassic Moderator

April 24, 2023 17:05

Hey there! I have a few thoughts on this. The first thought, is that the excessive warnings from LFD can be ignored. Those are just telling you a process ran longer than their default setting of 1800 seconds, which is completely possible with those main PHP processes. Those likely wouldn't not be related to your issue. That 4th block of text you posted shows there there was an OOM event for the system overall, so something is causing the machine to run out of memory, and also to experience a high load condition. I would recommend checking the sar logs on the system to see if that helps you identify any specific issues on the machine: If your server consistently has this issue overnight at the same time, it might be simplest to just watch the server in real-time so you can see what happens. However, if the system gets to the point where even SSH does not work, the issues indicate a deeper problem than the cPanel software.

ddaddy

April 24, 2023 17:53

Thanks for the reply. All `sar` gives me, regardless which file I choose, is:


Linux 4.18.0-425.19.2.el8_7.x86_64 (myserver..com)        04/24/2023      _x86_64_        (2 CPU)

06:26:19     LINUX RESTART      (2 CPU)

As it's an AWS EC2 instance, i'd have thought any HDD issues would be flagged to them internally, but it's possible I guess. When I say SSH doesn't work, I think it's just a case of OOM like you say stopping it connecting/working properly.

April 24, 2023 17:59

That's odd - I would expect the sar logs to have much more data there. We'd be happy to take a look in a ticket if you'd like to submit one, but I'm not sure there's going to be much for us to check if those logs don't exist. It might be best to reach out to the hosting provider to see if they can check the system, or if they have a way to do more advanced monitoring at that time. I still think the best action to get the most useful details would be to watch the server load in real-time during the time the issue happens.

April 24, 2023 18:05

Thanks. I'm going to gather a bit more info by changing the times of the cron jobs. Currently update is running just before midnight, and backup at 2am. So it could be either of these causing the issue. I've changed them 2am and 9am and will come back if I get the same thing happen.

sparek-3

April 25, 2023 02:03

What specific distribution are using? What kind of virtualization is AWS using these days? [font="courier new">dmidecode -s system-product-name

SimpleSonic

April 25, 2023 02:27

I"d be interested to see the Apache logs as well as any netstat output around the time it starts going south to rule out any possible layer 7 type attack.

April 25, 2023 07:14

What specific distribution are using?

AlmaLinux release 8.7 (Stone Smilodon) [QUOTE] What kind of virtualization is AWS using these days? dmidecode -s system-product-name
This just returns the instance type t3.small

The logs have been deleted due to The "Delete each domain"s access logs after statistics are gathered" option is enabled
but if it happens again i'll be sure to grab them.

May 10, 2023 07:27

It's been a couple of weeks and this has started again. It now happens just after 9am which to me confirms it has something to do with the upcp Cpanel update as I changed it to run at 9am. The last 2 days at approx 9:05 I am notified that my websites are down. The CPU's max out causing everything to be unresponsive and it stays like that until I reboot the server.

May 10, 2023 10:55

Can you submit a ticket to our team so we can take a look?

Server hangs during nightly maintenance

Comments

Didn't find what you were looking for?