/home/virtfs/ is using up large amounts of disk space

Symptoms

You may see significant disk usage reported for /home/virtfs by some utilities like the following example.

-bash-4.2# du -h --max-depth=0 /home/virtfs/
50G /home/virtfs/

Description 

Virtfs uses bind mounts to provide a jailed environment for cPanel users. These bind mounts allow them to have access to important binaries and other files but not be able to see other users on the system. Certain utilities will report these bind mounts using disk space, even though no disk space is used.

Warning: Do not use the rm command to remove any mounted file or directory within the /home/virtfs/ directory. If you run this on any mounted file or directory within the /home/virtfs/ directory, you will also delete all of the files in the directory to which it is mounted. This action can render your server nonfunctional.

Explanation

VirtFS does not use any space on the server. This is an artifact of how certain utilities calculate disk space. The bind mounts used by VirtFS cause some files to be counted more than once in certain utilities, causing the appearance that VirtFS is using up disk space. The files are counted at both their actual location on the server and at the bind mount locations in Virtfs. This can be seen in the example below.

We start with two files in /root/. One is 1 GB, and the other is 0 bytes. Note that the inode (the first number) is different for the files.

-bash-4.2# ll -ih file*
4242849 -rw-r--r-- 1 root root 1.0G Mar 29 01:16 file1.txt
4242894 -rw-r--r-- 1 root root 0 Apr 1 03:12 file2.txt

Checking the disk usage, we see that there is a rough agreement on the disk usage.

-bash-4.2# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 7.7G 33G 20% /

-bash-4.2# du -hax --max-depth=0 /
7.6G /

Then we bind mount one file on top of the other. This tells the kernel that when a request is made for file2.txt, it should use the file1.txt file instead.

-bash-4.2# mount -o bind /root/file1.txt /root/file2.txt

We can now see that file2.txt is actually pulling up file1.txt's data. This can be seen by the matching inodes

-bash-4.2# ll -ih file*
4242849 -rw-r--r-- 1 root root 1.0G Mar 29 01:16 file1.txt
4242849 -rw-r--r-- 1 root root 1.0G Mar 29 01:16 file2.txt

Lastly, we now see that df is reporting the same disk usage, but du is reporting an extra gigabyte.

-bash-4.2# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 7.7G 33G 20% /

-bash-4.2# du -hax --max-depth=0 /
8.6G /

This is due to du counting the disk usage of indode 4242849 for both file1.txt and file2.txt. You can also cause du to report less disk space by reversing the bind mount as shown below.

-bash-4.2# umount /root/file2.txt
-bash-4.2# mount -o bind /root/file2.txt /root/file1.txt
-bash-4.2# ll -ih file*
4242894 -rw-r--r-- 1 root root 0 Apr 1 03:12 file1.txt
4242894 -rw-r--r-- 1 root root 0 Apr 1 03:12 file2.txt
-bash-4.2# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 7.7G 33G 20% /
-bash-4.2# du -hax --max-depth=0 /
6.6G /

As you can see above, du is now counting inode 4242894 for both file1.txt and file2.txt, causing the reported disk usage to drop by 2 GB, but the real disk usage has not changed at all. This is why some utilities will erroneously report that VirtFS is using disk space, when in fact, VirtFS does not use any disk space.