Disk Usage Vs. Inodes Usage

Question

My file system has plenty of space left, but I'm getting errors about a lack of inodes. What are inodes & how do I detect and fix this problem?

Note:

Normal disk usage is different from inode usage. For more information on normal disk usage management you may refer to this link from our official documentation: 

How to Manage Your Hard Drive Space (cPanel Docs)

Answer

What is an inode?

Inodes, "Index Nodes", are fixed-length table entries in a Linux file system, each of which holds information about one specific file (regular files, folders, hard links, etc). An inode stores basic information about a regular file, directory, or other file system objects. Every inode in Linux has a unique number associated with it which can be seen with this command: 

ls -i $FILENAME 

For example:

ls -i /etc/passwd
232394 /etc/passwd

One way to run out of space in a filesystem is to use up all your inodes. If many small files are present, this can cause the pool of inodes to be consumed prematurely. Even if you might still have enough free space on your disk, you won’t be able to create new files, because the file system is incapable of allocating an inode number to the new file.  

The important information to remember about inodes is that they are always limited in number and once exhausted you cannot simply make more. The only way to increase the number of inodes would be to resize the disk.

For some file systems, inodes are pre-allocated at the time a file system is created. However, for some more modern file systems inodes are created dynamically as they are needed. Either way, their number is always limited regardless of how they are allocated. 

The number and the location of all the available inodes for a given disk are saved in a special record inside a file system called a superblock. The number of available/used/free inodes can be seen by running this command against the disk: (Run df -h to see available disks)

df -i /dev/(device)

For example:

df -i /dev/vda1

Filesystem  Inodes   IUsed   IFree   IUse% Mounted on
/dev/vda1  20970944  555418 20415526  3%     /

What to do when the inode usage (IUse%) has exceeded the recommended value?

You might face situations where a process fails due to "insufficient available inodes" or some other similar error messages. The only way to circumvent this issue would be to either resize the disk or to free up inodes by removing unwanted files from the file system. Here we look at the latter option: 

First, you need to identify what directories contain more files (AKA inodes): 

for i in $DIR*; do echo " $(find $i -maxdepth 12 | wc -l) $i"; done | sort -nr

This command would give you a rough idea of where you might want to start the cleaning process. You must replace the $DIR in the above command with the directory you wish to examine (Usually the root directory "/"). Here is an example output for demonstration purposes: 

for i in /*; do echo $i; find $i -maxdepth 12 |wc -l; done

/backup
643
/bin
1
/boot
344
/dev
331
/etc
74332
/home
191506 # Note how this value is much higher than the rest
/opt
23642
/proc
96355
/root
2422
/run
625
/sbin
/sys
23505
/test
176
/tmp
88
/usr
260373 # Note how this value is much higher than the rest
/var
35456

The highlighted directories in the above output are potential candidates where we can narrow down our search since they seem to contain more files/inodes. (Please remember that you must refrain from removing files from the system directories or any files that are essential to the operation of the system). 

Consequently, you can run the command once more, but this time replace $DIR with the new directory. By repeating the above process sufficiently, you should be able to identify those directories that are the best candidates for clean-up. 

What if most of the files are mail-related? 

In those cases, where the usage of inodes is mainly concentrated in email accounts you can consider converting the mailbox format as an alternative method to free up inodes on the file system. The recommended format to use is MDBox. Please refer to this link for more information on how to convert the mailbox format to MDBox: 

Mailbox Conversion (cPanel Docs)

What is the required free inode number for a cPanel installation/update? 

cPanel requires at least 360,000 free inodes at '/usr/local/cpanel' in order to successfully update.