cPanel user-backups seem to disappear after completing

Symptoms

Creating a user-backup via the cPanel interface (cPanel >> Backup) or the API results in the backup disappearing after completion.

Description 

When a backup is being created, the contents are written to a temporary location such as /home/. Then once the backup is completed, the system uses the rename system-call to move the content from the temporary location to the destination point which is usually the user's home directory (/home/$USER/).

If the source (/home/) & destination (/home/$USER/) paths are not mounted on the same filesystem, then the rename system-call will fail to execute.

Using a strace of the process will display it as follows:

rename("/home/24937.BIN_ADMIN_CPANEL_BACKUP_PL__.423a7848.tmp/backup-12.6.2020_$USER.tar.gz", 
"/home/$USER/backup-12.6.2020_$USER.tar.gz") = -1 EXDEV (Invalid cross-device link)

This is documented in the error section of the system-call:

EXDEV  oldpath and newpath are not on the same mounted filesystem.
              (Linux permits a filesystem to be mounted at multiple points,
              but rename() does not work across different mount points, even
              if the same filesystem is mounted on both.)

Source: https://man7.org/linux/man-pages/man2/renameat2.2.html

Workaround

It is not standard to have a user's home directory mounted in a separate filesystem, generally, these have been mounts created by accident or by any troubleshooting steps the system's administrator may have taken.

If this is the case, you can address this by removing the mount.

# umount $MOUNTED_DIRECTORY

These can generally be located using this command:

mount|grep -P '\s/home.?/(?!virtfs)'

In the example for this test server the mount is as follows:

# mount|grep -P '\s/home.?/(?!virtfs)'
/dev/vda1 on /home/cloud-user type xfs (rw,relatime,attr2,inode64,usrquota)

We can then unmount it to ensure the system-calls complete without issue:

# umount /home/cloud-user/