500 Error login, restart login

redayhost

December 01, 2016 11:48

Hello today I started to live in cPanel troubles. The error logs are like below. Cpanel login error;


Internal Server Error

500

The system failed to open the session file "/var/cpanel/sessions/raw/:lO5kTot8kkZsX_CQ" because of an error: No such file or directory at /usr/local/cpanel/Cpanel/Session.pm line 268,  line 2.
cpsrvd/11.60.0.25 Server at example.com

Cpanel restart error;

root@cpanel1 [~]# service cpanel restart
Stopping tailwatchd:  [7070] [2016-12-01 20:27:04 +0300] [] No PID in /var/run/tailwatchd.pid
                                                           [  OK  ]
Stopping cPanel services:  Failed to write lock file: /var/run/restartsrv_cpsrvd.lock: Bad file descriptor at /usr/local/cpanel/Cpanel/SafeFileLock.pm line 27.
warn [restartsrv_base] could not acquire a lock for '/var/run/restartsrv_cpsrvd': No such file or directory

Could not create lock file "/var/run/restartsrv_cpsrvd" at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25.
                                                           [FAILED]
Stopping cPanel dav services:  Failed to write lock file: /var/run/restartsrv_cpdavd.lock: Bad file descriptor at /usr/local/cpanel/Cpanel/SafeFileLock.pm line 27.
warn [restartsrv_base] could not acquire a lock for '/var/run/restartsrv_cpdavd': No such file or directory
Could not create lock file "/var/run/restartsrv_cpdavd" at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25.
                                                           [FAILED]
Stopping cPanel queue services:  Failed to write lock file: /var/run/restartsrv_queueprocd.lock: Bad file descriptor at /usr/local/cpanel/Cpanel/SafeFileLock.pm line 27.
warn [restartsrv_base] could not acquire a lock for '/var/run/restartsrv_queueprocd': No such file or directory
Could not create lock file "/var/run/restartsrv_queueprocd" at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25.
                                                           [FAILED]
Stopping cPanel brute force detector services:  Failed to write lock file: /var/run/restartsrv_cphulkd.lock: Bad file descriptor at /usr/local/cpanel/Cpanel/SafeFileLock.pm line 27.
warn [restartsrv_base] could not acquire a lock for '/var/run/restartsrv_cphulkd': No such file or directory
Could not create lock file "/var/run/restartsrv_cphulkd" at /usr/local/cpanel/Cpanel/RestartSrv/Lock.pm line 25.
                                                           [FAILED]
Stopping cPanel log services:                              [FAILED]

Stopping mailman services:  Shutting down Mailman's master qrunner
mailmanctl: no process killed
                                                           [FAILED]
Starting cpanel:

*** /usr/local/cpanel/etc/init/startup ERROR: ***
    Refusing to start because log directory /usr/local/cpanel/logs is not writable!
    The specific error was: No space left on device at /usr/local/cpanel/Cpanel/LogCheck.pm line 29.

    Please rectify this problem and then try again.

root@cpanel1 [~]#                                          [FAILED]

/scripts/upcp Clamp


root@cpanel1 [~]# /scripts/upcp
process is already running

These problems happened after you updated.

Comments

3 comments

cPanelMichael

December 01, 2016 19:21
Hello, Could you review the output from /var/log/messages and /var/log/dmesg and verify if you notice any issues with your hard drive or filesystem? Also, please post the output from the following commands:
grep "\sro[\s,]" /proc/mounts touch /var/cptestfile123 df -h df -i
Thank you.
0
redayhost

December 02, 2016 13:11
Hello, Could you review the output from /var/log/messages and /var/log/dmesg and verify if you notice any issues with your hard drive or filesystem? Also, please post the output from the following commands:
grep "\sro[\s,]" /proc/mounts touch /var/cptestfile123 df -h df -i
Thank you.

Hi grep "\sro[\s,]" /proc/mounts
root@cpanel1 [~]# grep "\sro[\s,]" /proc/mounts /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/lib ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/lib ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/lib64 ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/lib64 ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/include ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/share/locale ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/share/terminfo ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/share/zoneinfo ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/share/vim ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/local/lib/php ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/local/cpanel/Cpanel ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/perl ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/lib ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/share ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/php ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/local/easy ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/bin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/var/lib/spamassassin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/suphp/sbin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php70/root/usr/bin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php70/root/etc ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php56/root/usr/bin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php56/root/etc ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php54/root/usr/bin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php54/root/etc ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php55/root/usr/bin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php55/root/etc ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php53/root/usr/bin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php53/root/etc ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php71/root/usr/bin ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 /dev/mapper/vg_cpanel1-lv_root /usr/share/cagefs-skeleton/opt/cpanel/ea-php71/root/etc ext4 ro,nosuid,relatime,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=quota.user 0 0 root@cpanel1 [~]#
touch /var/cptestfile123
root@cpanel1 [~]# touch /var/cptestfile123 root@cpanel1 [~]#
dh -h
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_cpanel1-lv_root 50G 33G 14G 71% / tmpfs 3.9G 6.6M 3.9G 1% /dev/shm /dev/sda1 477M 194M 258M 43% /boot /dev/mapper/vg_cpanel1-lv_home 439G 31G 391G 8% /home /usr/tmpDSK 2.1G 81M 1.9G 5% /tmp
df -i
root@cpanel1 [~]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vg_cpanel1-lv_root 3276800 3262066 14734 100% / tmpfs 1003005 6 1002999 1% /dev/shm /dev/sda1 128016 68 127948 1% /boot /dev/mapper/vg_cpanel1-lv_home 29147440 786885 28360555 3% /home /usr/tmpDSK 134640 276 134364 1% /tmp
0
cPanelMichael

December 02, 2016 14:09
Hello, Your "/" partition is at 100% disk inode usage. Inodes represent the number of files. You will need to remove some files from this partition to get below 100% usage. Here are some commands to see which directories in the / partition are utilizing the most inodes:
cd / echo "Inode usage for: $(pwd)" ; for d in `find -maxdepth 1 -type d |cut -d\/ -f2 |grep -xv . |sort`; do c=$(find $d |wc -l) ; printf "$c\t\t- $d\n" ; done ; printf "Total: \t\t$(find $(pwd) | wc -l)\n"
If you see a directory with a high inode count, browse to it and run the same command again to see the results for the individual directory. This command is also helpful in some cases where the comet files in the /root partition utilize a large number of inodes:
/usr/local/cpanel/bin/purge_dead_comet_files
Thanks!
0

Please sign in to leave a comment.

Comments

Didn't find what you were looking for?