How to find what has the most inotify registered watches

Introduction

Inotify is an API that allows for files to be monitored for certain events. Different software may make use of this API to provide functionality. However, there may be a process or task that is registering too many watches to the point where a limit is reached and it prevents other services from working as they may rely on an available inotify watch.

Procedure

You can simply run the following command as the root user and you will be presented with a formatted output showing the PID (process ID) and the number of registered watches that the PID has allocated.

printf "PID\t\tWATCH TOTAL\n\n";find /proc/*/fd -lname anon_inode:inotify -printf '%hinfo/%f\n' 2>/dev/null | xargs grep -c '^inotify'|awk -F [/:] '{print $3"\t\t"$NF}'|sort -rnk2

Example Output:

[root@the ~]$ printf "PID\t\tWATCH TOTAL\n\n";find /proc/*/fd -lname anon_inode:inotify -printf '%hinfo/%f\n' 2>/dev/null | xargs grep -c '^inotify'|awk -F [/:] '{print $3"\t\t"$NF}'|sort -rnk2
PID WATCH TOTAL

4598 100786
445 7
52800 4
2264163 4
698262 3
2264163 3

In the example output above, we can see that the PID 4598 has a total of 100786 registered watches currently open.

Using the systcl command, we can see that our machine limits are set to 100,000

[root@the ~]$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 100,000

This means that our limits have been exceeded and we must either increase the max user watches available or address the issue with the task that is generating this many watches.

To increase the value simply use this command:

(Be sure to replace $NEW_VALUE with the new numerical value you desire) 

sysctl fs.inotify.max_user_watches=$NEW_VALUE

For more details on modifying systcl values, please refer to the following article.