Solr Is Killed By Its OutOfMemory Shutdown Hook

Symptoms: 

The errors that we are looking for might appear in different places and different formats, however, we are looking for any reference to the execution of this script /home/cpanelsolr/bin/oom_solr.sh or any mention of the OOM Killer script/shutdown hook script (the above script) being executed. 

You should be able to run a grep on the whole Solr's default log directory like this: 

[15:47:01 root@server ~]cPs# grep -ri oom /home/cpanelsolr/server/logs

/home/cpanelsolr/server/logs/archived/solr_gc.log.0.current:CommandLine flags: -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 -XX:MaxTenuringThreshold=8 -XX:NewRatio=3 -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 8984 /home/cpanelsolr/server/logs -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
/home/cpanelsolr/server/logs/solr_gc.log.0.current:CommandLine flags: -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 -XX:MaxTenuringThreshold=8 -XX:NewRatio=3 -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 8984 /home/cpanelsolr/server/logs -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC

Note: Scroll across to see the occurrence of the shutdown hook script /home/cpanelsolr/bin/oom_solr.sh

Or alternatively, sometimes Solr creates specific log files for each time the OOM killer script had been executed and you will be able to find these script in the Solr's default log directory: 

[06:57:45 root@server /home/cpanelsolr/server/logs]cPs# ls -alhtr /home/cpanelsolr/server/logs | grep -i oom | grep -i oct

-rw-r--r--  1 cpanelsolr cpanelsolr  91 Oct  2 08:51 solr_oom_killer-8984-2020-10-02_08_51_40.log
-rw-r--r--  1 cpanelsolr cpanelsolr  91 Oct  2 09:05 solr_oom_killer-8984-2020-10-02_09_05_53.log
-rw-r--r--  1 cpanelsolr cpanelsolr  91 Oct  2 09:10 solr_oom_killer-8984-2020-10-02_09_10_15.log
-rw-r--r--  1 cpanelsolr cpanelsolr  91 Oct  2 09:29 solr_oom_killer-8984-2020-10-02_09_29_48.log
-rw-r--r--  1 cpanelsolr cpanelsolr  91 Oct  2 09:33 solr_oom_killer-8984-2020-10-02_09_33_09.log
-rw-r--r--  1 cpanelsolr cpanelsolr  91 Oct  2 09:38 solr_oom_killer-8984-2020-10-02_09_38_58.log

Looking at the content of one of these files, you should see the OOM killer script being directly alluded to: 

[07:01:22 root@server /home/cpanelsolr/server/logs]cPs# cat solr_oom_killer-8984-2020-10-12_17_53_13.log

Running OOM killer script for process 1713942 for Solr on port 8984
Killed process 1713942

Explanation:

The bin/solr script registers the bin/oom_solr.sh script to be called by the JVM if an OutOfMemoryError occurs. The oom_solr.sh script will issue a kill -9 to the Solr process that experiences the OutOfMemoryError . OutOfMemoryError is very similar in principle to the usual OOM mechanism in Linux. For more information on Linux's OutOfMemory mechanism you can refer to this link: 

What Is the OOM Killer?

Workaround: 

The default memory limit for Solr is defined in /etc/sysconfig/cpanel-dovecot-solr:

SOLR_MEMORY="512m"

You can try increasing this value from 512m to any higher value to see if it helps.

Remember to restart the service after making the change:

/usr/local/cpanel/scripts/restartsrv_cpanel_dovecot_solr

However, please bear in mind that Solr's memory usage depends on many factors, such as how many emails are stored on your server, how often the index-worker is running, how often searches are taking place, and how long they usually last, and general RAM usage for other Solr processes. So naturally, you might need to increase the memory limitation even more if you still run into the same issue. 

Please refer to these links for more information on what the OutOfMemoryError is: 

https://www.searchstax.com/blog/solr-out-of-memory-oom-causes-and-solutions/