How-to capture core files from Apache

Rarely, Apache will stop unexpectedly because of a segmentation fault.  It stops suddenly, and the error log has an entry like:

[core:notice] [pid 1234] AH00060: seg fault or similar nasty error detected in the parent process

or

[core:notice] [pid 1234] AH00051: child pid 1234 exit signal Segmentation fault (11), possible coredump in /etc/apache2

To debug these types of problems, it's frequently advantageous to examine the "core dump" left by Apache when it ended. To do this, you must know how to use the gdb debugger, and you must turn on capturing core dump files from Apache.  It's important to note that the steps below do NOT solve any problems themselves - the process sets up one more tool for determining the root cause.

Procedure

1. Create Backups

Use the following three commands to create backups of the values you're going to change so that you can set them back once you have completed troubleshooting.  Core dumps can be large and contain sensitive data, so you probably do not wish to keep them enabled on a production server longer than necessary to capture your troubleshooting information.

mkdir /root/backup_proc_core_values

cp /proc/sys/fs/suid_dumpable /root/backup_proc_core_values/suid_dumpable

cp /proc/sys/kernel/core_pattern /root/backup_proc_core_values/core_pattern

2. Set up /proc

Now you'll set those variables you just backed up the values you need to capture core dumps for Apache.  The core pattern can put the core files anywhere you want.  Make sure that the directory you choose exists. Since core files may be large, I recommend putting them in /home. The /home partition frequently has the most free space on a cPanel server.

echo 2 > /proc/sys/fs/suid_dumpable

mkdir /home/core_files

echo '/home/core_files/%e.core.%t.%p' > /proc/sys/kernel/core_pattern

3. Set cPanel and Apache to generate core dumps in Tweak Settings & restart Apache

Go to Tweak Settings, and on the Security tab, there's an option to "Generate Core Dumps".  It is off by default.  Turn it on and use the save button at the bottom of the page.

Next, run systemctl edit to create a drop-in systemd configuration file to override the LimitCORE values:

 systemctl edit httpd

Paste the following in the empty text editor that opens:

[Service]
LimitCORE=infinity

Save the changes and then run systemd daemon-reload:

systemctl daemon-reload

Daemon-reload reloads all unit files, so the latest changes go into effect. After this, confirm the LimitCORE value is infinity:

systemctl show httpd | grep LimitCORE
LimitCORE=infinity
LimitCORESoft=infinity

Restart Apache using systemctl:

systemctl restart httpd

Now the server is ready to generate core dumps from Apache.  Please don't forget to turn these features off when you're done, as core dumps can contain sensitive data and be very large. 

To disable core dumps when troubleshooting is completed:

1. Change the "Generate Core Dumps" setting to "Off" in Tweak Settings
2. Set the suid_dumpable back to its previous value (probably 0):

cp /root/backup_proc_core_values/suid_dumpable /proc/sys/fs/suid_dumpable

Remove the line with LimitCORE  in /etc/systemd/system/httpd.service.d/override.conf file and run the systemctl daemon-reload command  again:

systemctl daemon-reload

Optionally, set the core pattern back to its previous value as well:

cp /root/backup_proc_core_values/core_pattern /proc/sys/kernel/core_pattern