Symptoms
When trying to restart MySQL, this error will appear in system logs (journalctl OR /var/log/messages):
Oct 07 09:38:18 server.test-001.net systemd[13595]:Failed at step NAMESPACE spawning /bin/sh: No space left on device
Oct 07 09:38:18 server.test-001.net systemd[1]: mariadb.service: control process exited, code=exited status=226
Oct 07 09:38:18 server.test-001.net systemd[1]: Failed to start MariaDB 10.3.24 database server.
Oct 07 09:38:18 server.test-001.net systemd[1]: Unit mariadb.service entered failed state.
Oct 07 09:38:18 server.test-001.net systemd[1]: mariadb.service failed.
Oct 07 09:38:18 server.test-001.net systemd[1]: Starting MariaDB 10.3.24 database server...
Oct 07 09:38:18 server.test-001.net systemd[13605]: Failed at step NAMESPACE spawning /bin/sh: No space left on device
Oct 07 09:38:18 server.test-001.net systemd[1]: mariadb.service: control process exited, code=exited status=226
Oct 07 09:38:18 server.test-001.net systemd[1]: Failed to start MariaDB 10.3.24 database server
Description
This usually means that the system or at least one of the storage devices on the system is running out of space. Here "space" could mean both normal block storage or inode usage. In this specific case when checking the storage situation on the server, I can see that the /tmp directory is at 100% inode usage:
df -hi
Filesystem Inodes IUsed IFreeIUse%
Mounted on
devtmpfs 3.7M 413 3.7M 1% /dev
tmpfs 3.7M 2 3.7M 1% /dev/shm
tmpfs 3.7M 1.4K 3.7M 1% /run
tmpfs 3.7M 16 3.7M 1% /sys/fs/cgroup
/dev/mapper/centos-root 13M 254K 13M 2% /
/dev/sda1 512K 353 512K 1% /boot
/dev/mapper/w32fast-homefast 100M 78K 100M 1% /homefast
/dev/mapper/w32slow-homeslow 250M 1.8M 249M 1% /homeslow
/dev/mapper/w32bak-bak 150M 605 150M 1% /backup/dev/loop0 66K 66K 0 100% /tmp
tmpfs 3.7M 1 3.7M 1% /run/user/0
Please refer to this link for more information on the difference between normal storage and inode usage:
Workaround
You need to clear some of the files on the problematic directory/file system/device and then attempt to restart MySQL again using this command:
/scripts/restartsrv_mysql --restart