Services Fail With The "inotify resources exhausted" Error Message

Symptoms

You might run into a situation where a specific service or a command fails with similar error messages: 

/scripts/restartsrv_httpd --start
[...............]

Mar 11 00:07:13 server.domain.com systemd[1]: Failed to set a watch for httpd.service's PID file /run/apache2/httpd.pid: No space left on device

tail -fn0 /var/log/messages
tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling

Description 

This suggets that the server is running out of Inotify watches. For more information about Inotify watches and how to count them per PID you can refer to the following article: 

How to find what has the most inotify registered watches

Workaround

In these situations you can temporarily increase the fs.inotify.max_user_watches system limit to some number bigger than the number of used Inotify watches on the server so you are able to start a service or run a specific command. The following command should help with that: 

sysctl fs.inotify.max_user_watches=$NEW_VALUE

For a more permanent resolution to this issue, you would need to edit the /etc/sysctl.d system by creating a custom file and adding that value to it.  You can do this by running the following command on the server, which will keep the setting persistent after reboots:

touch /etc/sysctl.d/inotify.conf && echo "fs.inotify.max_user_watches=65536" > /etc/sysctl.d/inotify.conf