/home/virtfs/ is using up large amounts of disk space

Symptoms

You may see significant disk usage reported for /home/virtfs by some utilities, like the following example.

[root@server]cPs# du -h --max-depth=0 /home/virtfs/
50G /home/virtfs/

Description 

Virtfs uses bind mounts to provide a jailed environment for cPanel users. These bind mounts allow them to have access to essential binaries and other files but not be able to see other users on the system. Certain utilities will report these bind mounts using disk space, even though they do not.

Warning: Do not use the rm command to remove any mounted file or directory within the /home/virtfs/ directory!

If you remove any mounted file or directory within the /home/virtfs/ directory, you will also delete those files and directories in the location where they are actually stored. As such, removals of files in /home/virtfs/ can render your server nonfunctional.

Explanation

VirtFS does not use any space on the server. This behavior is an artifact of how certain utilities calculate disk space. The bind mounts used by VirtFS cause some files to be counted more than once in certain utilities, causing the appearance that VirtFS is using up disk space. The files are counted at their actual location on the server and the bind mount locations in Virtfs. This behavior can be seen in the example below.

We start with two files in /root/. One is 1 GB, and the other is 0 bytes. Note that the inode (the first number) differs for the files.

[root@server]cPs# ll -ih file*
4242849 -rw-r--r-- 1 root root 1.0G Mar 29 01:16 file1.txt
4242894 -rw-r--r-- 1 root root 0 Apr 1 03:12 file2.txt

Checking the disk usage, we see that there is a rough agreement on the disk usage.

[root@server]cPs# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 7.7G 33G 20% /

[root@server]cPs# du -hax --max-depth=0 /
7.6G /

Then we bind mount one file on top of the other. This mount tells the kernel that when a request is made for file2.txt, it should use the file1.txt file instead.

[root@server]cPs# mount -o bind /root/file1.txt /root/file2.txt

We can now see that requests for file2.txt are pulling up file1.txt's data, as indicated by the matching inode reference numbers on the left side:

[root@server]cPs# ll -ih file*
4242849 -rw-r--r-- 1 root root 1.0G Mar 29 01:16 file1.txt
4242849 -rw-r--r-- 1 root root 1.0G Mar 29 01:16 file2.txt

Lastly, we now see that df reports the same disk usage, but du reports an extra gigabyte:

[root@server]cPs# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 7.7G 33G 20% /

[root@server]cPs# du -hax --max-depth=0 /
8.6G /

This difference is due to du counting the disk usage of inode 4242849 for file1.txt and file2.txt. You can also cause du to report less disk space by reversing the bind mount, as shown below:

[root@server]cPs# umount /root/file2.txt
[root@server]cPs# mount -o bind /root/file2.txt /root/file1.txt
[root@server]cPs# ll -ih file*
4242894 -rw-r--r-- 1 root root 0 Apr 1 03:12 file1.txt
4242894 -rw-r--r-- 1 root root 0 Apr 1 03:12 file2.txt
[root@server]cPs# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 7.7G 33G 20% /
[root@server]cPs# du -hax --max-depth=0 /
6.6G /

As you can see above, du is now counting inode 4242894 for both file1.txt and file2.txt, causing the reported disk usage to drop by 2 GB, but the actual disk usage has not changed at all. This is why some utilities will erroneously report that VirtFS is using disk space when in fact, VirtFS does not use any disk space.

When running disk space checks, you can use the following command to exclude virtfs and get a more accurate result:

[root@server]cPs# du -hs /home/ --exclude=virtfs