What to do if the "/scripts/find_and_fix_rpm_issues" timeouts or doesn't respond?

Question

What to do if the "/scripts/find_and_fix_rpm_issues" timeouts or doesn't respond?

Answer

If you run the following script by command-line on the server, you should receive some output, like so:

# /scripts/find_and_fix_rpm_issues
Checking that the RPM DB is OK...
glibc-2.17-317.el7.x86_64
OK: RPM DB is responding to queries
Testing if rpm_is_working RPM is installed
package rpm_is_working is not installed
Testing if it's possible to install a simple RPM
Preparing... ########################################
Updating / installing...
rpm_is_working-1.0-0 ########################################

However, if you do not get output for a long period of time (like over a minute), the output will look like this:

# /scripts/find_and_fix_rpm_issues
Checking that the RPM DB is OK...

There won't be any extra output. 

We would recommend looking at the process list and seeing multiple copies of 'rpm' running simultaneously.

Example from test server:

# ps awux | grep rpm
root      7890  0.0  0.0  70848  3904 ?        S    10:15  0:00 /usr/bin/rpm -q --nodigest --nosignature --queryformat %{NAME} %{VERSION}-%{RELEASE}\n -a kernelcare
root    12529  0.0  0.0  70848  1824 ?        S    Nov28  0:00 /usr/bin/rpm -q --nodigest --nosignature --queryformat %{NAME} %{VERSION}-%{RELEASE}\n -a kernelcare
root    13645  0.0  0.0  70848  652 ?        S    Oct19  0:00 /usr/bin/rpm -q --nodigest --nosignature --queryformat %{NAME} %{VERSION}-%{RELEASE}\n -a kernelcare
root    13926  0.0  0.0  70852  3920 pts/0    S    11:33  0:00 /bin/rpm -q --nosignature --nodigest glibc
root    14314  0.0  0.0  70848  944 ?        S    Nov14  0:00 /usr/bin/rpm -q --nodigest --nosignature --queryformat %{NAME} %{VERSION}-%{RELEASE}\n -a kernelcare
root    14434  0.0  0.0  70852  3928 pts/0    S    11:36  0:00 /bin/rpm -q --nosignature --nodigest glibc
root    14560  0.0  0.0  70852  3916 ?        SN  11:38  0:00 /bin/rpm -qa --nodigest --nosignature --queryformat %{NAME}\t%{VERSION}\t%{RELEASE}\t%{VENDOR}\t%{INSTALLTIME}\t%{RSAHEADER:pgpsig}\n
root    14561  0.0  0.0  70852  3912 ?        SN  11:38  0:00 /bin/rpm -qa --nodigest --nosignature --queryformat %{NAME}\t%{VERSION}\t%{RELEASE}\t%{VENDOR}\t%{INSTALLTIME}\t%{RSAHEADER:pgpsig}\n
root    14562  0.0  0.0  70852  3920 ?        SN  11:38  0:00 /bin/rpm -qa --nodigest --nosignature --queryformat %{NAME}\t%{VERSION}\t%{RELEASE}\t%{VENDOR}\t%{INSTALLTIME}\t%{RSAHEADER:pgpsig}\n
root    15732  0.0  0.0  70852  3920 pts/0    S    11:48  0:00 /bin/rpm -q --nosignature --nodigest glibc
root    16080  0.0  0.0  70852  3924 pts/0    S    11:51  0:00 /bin/rpm -q --nosignature --nodigest glibc
root    16102  0.0  0.0 174572  4024 pts/0    S+  11:51  0:00 rpm -qa
root    16625  0.0  0.0  73476  680 ?        S    Sep29

This indicates that 'rpm', which depends on locking the RPM database, competes with other copies of itself. All of these 'rpm' commands will not be complete as they will not timeout if there are too many.

We would recommend finding and killing those 'rpm' processes until they are all gone.

If you use the following command to 'kill' the process, please use caution when doing so.

Command:

kill -9 `ps awux | grep -v grep rpm | awk ' { print $2 } ' | xargs`

Once those 'rpm' are dead, you should be able to run the '/scripts/find_and_fix_rpm_issues' script successfully.

Command:

/scripts/find_and_fix_rpm_issues