Solr Is killed by its OutOfMemory shutdown hook

Symptoms

When Solr gets killed, you will see an entry containing OnOutOfMemoryError in the /home/cpanelsolr/server/logs/solr_gc.log.0.current log file.

CommandLine flags: -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 -XX:MaxTenuringThreshold=8 -XX:NewRatio=3 -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 8984 /home/cpanelsolr/server/logs -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC

You may also see log files with names starting with solr_oom_killer in /home/cpanelsolr/server/logs/.

solr_oom_killer-8984-2023-10-02_08_51_40.log

Description

The solr script registers theoom_solr.sh script to be called by the JVM if an OutOfMemoryError occurs. The oom_solr.sh script will issue a kill -9 to the Solr process that experiences an out-of-memory event. Increasing Solr's memory limit (default 512 MB) and restarting Solr can prevent Solr from encountering out-of-memory events.

Workaround

1. Access the server's command line as the 'root' user via SSH or "Terminal" in WHM.
2. Open the /etc/sysconfig/cpanel-dovecot-solr configuration file in your preferred text editor.
3. Scroll down to the SOLR_MEMORY line.
4. Change the SOLR_MEMORY value to the new value in megabytes.

   SOLR_MEMORY="512m"

5. Save the changes and exit the text editor.
6. Restart the Solr service.

   /usr/local/cpanel/scripts/restartsrv_cpanel_dovecot_solr

Additional resources

What Is the OOM Killer?