splitlogs core dumps (fclose in flush_handles) on cPanel with LiteSpeed Enterprise, about 100 accounts
Hi all,
I’m seeing intermittent splitlogs crashes on a dedicated cPanel/WHM server running LiteSpeed Enterprise instead of Apache.
Environment
- AlmaLinux 8.10, kernel 4.18.0-553.x
- cPanel/WHM (the crashes were seen on a build before 136.0 build 47)
- LiteSpeed Enterprise 6.3.7, 20 worker processes
- About 100 cPanel accounts, with about 60
splitlogsprocesses running at any time - Domlogs live in
/etc/apache2/logs/domlogs(/usr/local/apache/domlogsis a symlink to it) - Single socket, 40 threads, 125GB RAM, NVMe storage; load is low (about 90% idle) when this happens
What happens
-
splitlogssegfaults and core dumps at irregular times (for example, around 09:15 UTC on one day and 04:42 UTC the next). - The backtrace points at
fcloseinsideflush_handles. - When it happened, the domlog analysis step of my diagnostics returned nothing for that run, so I suspect log processing is being interrupted.
- It isn’t tied to memory pressure: no OOM kills, no I/O errors in
dmesg, and about 98GB of RAM was available.
What I’ve checked
- Disk space and inodes on
/and/homeare fine. - The system is otherwise healthy, with no failed systemd units and no 5xx spikes.
Questions
- Is this a known
splitlogsbug when many domlogs are open at once? With 100 accounts, is there a limit I should raise (open file handles,ulimit -nfor the splitlogs process, or a cPanel tweak setting)? - Is there a supported way to reduce how many handles
splitlogskeeps open, or to run it with debug logging so I can send you a full trace? - Does anyone running LiteSpeed (rather than Apache) at this account count see the same crashes?
Happy to attach the core dump backtrace and /usr/local/cpanel/logs/error_log excerpts if useful.
-
Hey there!
1 - This isn't a known issue and I don't have any other reports in our system when I checked just now. "maxopen" has a default of 16000, which most people don't change, so I'd expect that to be more than enough (https://docs.cpanel.net/knowledge-base/web-services/the-splitlogs-binary/)
2 - Yes. If you want to try fewer handles you can create /var/cpanel/cong/splitlogs.conf with either maxopen=no and/or buffer=no, then rebuild Apache (/scripts/rebuildhttpdconf, followed by a restart). Splitlogs itself doesn't have any specific debug tools but you may find something helpful in /usr/local/cpanel/logs/splitlogs_log.
3 - I'm not seeing any similar reports from users with Litespeed either.
0
Please sign in to leave a comment.
Comments
1 comment