Skip to main content

quotas and disk space usage problem

ouki
Hi, My server was reporting quota and disk space usage correctly, until yesterday when I had the crazy idea to run /scripts/fixquotas. Since then, the disk space usage by all hosted accounts is reported by WHM either 0 or with very low values, that are untrue. I searched the Internet and this forum for advice and I implemented all relevant instructions, failing to solve the problem. During the last 20 hours, I ran /scripts/fixquotas at least three times, without having any success. So, as last resort, I am kindly asking for your help. What can I do to fix it. Thank you for your help. MY SYSTEM INFO
grep '' /etc/redhat-release /usr/local/cpanel/version /var/cpanel/envtype ; grep CPANEL= /etc/cpupdate.conf ; httpd -v ; php -v ; mysql -V /etc/redhat-release:CentOS release 5.11 (Final) /usr/local/cpanel/version:11.52.4.2 /var/cpanel/envtype:standard CPANEL=11.52 Server version: Apache/2.2.29 (Unix) Server built: Apr 25 2015 00:55:09 Cpanel::Easy::Apache v3.28.8 rev9999 PHP 5.4.40 (cli) (built: Apr 25 2015 01:00:10) Copyright (c) 1997-2014 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies with the ionCube PHP Loader v4.6.1, Copyright (c) 2002-2014, by ionCube Ltd., and with Zend Guard Loader v3.3, Copyright (c) 1998-2013, by Zend Technologies mysql Ver 14.14 Distrib 5.5.47, for Linux (x86_64) using readline 5.1
/etc/fstab
LABEL=/ / ext3 usrjquota=quota.user,jqfmt=vfsv0 1 1 LABEL=/home /home ext3 usrjquota=quota.user,jqfmt=vfsv0 1 2 LABEL=/var /var ext3 usrjquota=quota.user,jqfmt=vfsv0 1 2 LABEL=/usr /usr ext3 usrjquota=quota.user,jqfmt=vfsv0 1 2 LABEL=/tmp /tmp ext3 defaults 1 2 LABEL=/boot /boot ext3 defaults 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 LABEL=SWAP-sda7 swap swap defaults 0 0 /tmp /var/tmp ext3 defaults,bind,noauto 0 0 //10.0.5.5/IS-12571 /nas cifs username=IS-12571,password=ms4u5Pmxwb 1 1
/scripts/fixquotas output
Installing Default Quota Databases......Done journaled quota support: kernel supports, user space tools supports (available) Warning virtual file systems are mounted. Quotas may be counted as double for users who are currently logged in. Please have jailshell users logout before running quotacheck in the future! All jailed users will be logged out in 60 seconds! Broadcast message from root (Fri Mar 25 09:41:14 2016): Warning! The system is about to perform quota maintenance. All users will be logged out in 60 seconds. PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your account. jailshell: no process killed checking out /tmp/backup checking out /backup checking out /backup Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process. / (already configured quotas = 1). /home (already configured quotas = 1). /var (already configured quotas = 1). /usr (already configured quotas = 1). /tmp (already configured quotas = 0). /boot (already configured quotas = 0). Updating Quota Files...... quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 2965 directories and 435752 files quotacheck: Old file not found. quotacheck: Scanning /dev/sda8 [/home] done quotacheck: Something weird happened while scanning. Error 2133571361 quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 1767 directories and 24100 files quotacheck: Old file not found. quotacheck: Scanning /dev/sda6 [/usr] done quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 14003 directories and 147670 files quotacheck: Old file not found. done ....Done Quotas have been enabled and updated. Reading package NNNNNN Resetting quota for XXXXX to 1500 M edquota: Quota file not found or has wrong format. Reading package NNNNNN Resetting quota for XXXXX to 600 M edquota: Quota file not found or has wrong format. etc..

Comments

11 comments

Date Votes
  • vanessa
    Try moving the aquota.* files out of / and /home, then run: /scripts/initquotas /scripts/fixquotas
    0
  • ouki
    Hi Vanessa, Thank you for your reply! There are no aquota* files inside / and /home. Inside /home there is only a quota.group file. I rename it to /home/__quota.group. Inside / there is only a quota.user file. I tried to rename it to /__quota.user but system doesn't allow me to do it. So I renamed /home/__quota.group back to /home/quota.group. After that, I haven't tried to run the two commands that you suggested.
    0
  • ouki
    Hi Vanessa, I managed to move quota files out of / and /home directories (the i attribute was set). A) I ran /scripts/initquotas Now WHM shows quota as unlimited and disk used with low values (from 0 to a few MB which is incorrect). In / there is a quota.user.new file in /home there is no quota file /scripts/initquotas outputed:
    journaled quota support: kernel supports, user space tools supports (available) Warning virtual file systems are mounted. Quotas may be counted as double for users who are currently logged in. Please have jailshell users logout before running quotacheck in the future! All jailed users will be logged out in 60 seconds! Broadcast message from root (Sat Mar 26 14:10:34 2016): Warning! The system is about to perform quota maintenance. All users will be logged out in 60 seconds. PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your account. jailshell: no process killed checking out /tmp/backup checking out /backup checking out /backup Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process. / (already configured quotas = 1). /home (already configured quotas = 1). /var (already configured quotas = 1). /usr (already configured quotas = 1). /tmp (already configured quotas = 0). /boot (already configured quotas = 0). Updating Quota Files...... quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 2966 directories and 435787 files quotacheck: Cannot find checked quota file for users on /dev/root! quotacheck: Scanning /dev/sda8 [/home] done quotacheck: Something weird happened while scanning. Error 2133571361 quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 1764 directories and 24133 files quotacheck: Old file not found. quotacheck: Scanning /dev/sda6 [/usr] done quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 14003 directories and 147691 files quotacheck: Old file not found. done ....Done Quotas have been enabled and updated.
    B) Then I ran /scripts/fixquotas Now WHM shows quota correctly, but still disk used with low values (from 0 to a few MB which is incorrect). In / there is a quota.user.new file in /home there is no quota file /scripts/fixquotas outputed:
    Installing Default Quota Databases......Done journaled quota support: kernel supports, user space tools supports (available) Warning virtual file systems are mounted. Quotas may be counted as double for users who are currently logged in. Please have jailshell users logout before running quotacheck in the future! All jailed users will be logged out in 60 seconds! Broadcast message from root (Sat Mar 26 14:23:30 2016): Warning! The system is about to perform quota maintenance. All users will be logged out in 60 seconds. PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your account. jailshell: no process killed checking out /tmp/backup checking out /backup checking out /backup Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process. / (already configured quotas = 1). /home (already configured quotas = 1). /var (already configured quotas = 1). /usr (already configured quotas = 1). /tmp (already configured quotas = 0). /boot (already configured quotas = 0). Updating Quota Files...... quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 2966 directories and 435788 files quotacheck: Can't create new quotafile //quota.user.new: File exists quotacheck: Cannot initialize IO on new quotafile: File exists quotacheck: Scanning /dev/sda8 [/home] done quotacheck: Something weird happened while scanning. Error 2133571361 quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 1765 directories and 24089 files quotacheck: Old file not found. quotacheck: Scanning /dev/sda6 [/usr] done quotacheck: Cannot stat old user quota file: No such file or directory quotacheck: Old group file not found. Usage will not be substracted. quotacheck: Checked 14003 directories and 147666 files quotacheck: Old file not found. done ....Done Quotas have been enabled and updated. Reading package PACKAGE Resetting quota for USERNAME to 1500 M edquota: Quota file not found or has wrong format. Reading package PACKAGE Resetting quota for USERNAME to 600 M edquota: Quota file not found or has wrong format. ... ... ...
    C) The situation is the same as it was at the beginning. Per account disk usage is reported incorrectly. Also, please note the "Something weird happened while scanning. Error 2133571361" message, while running both commands. Thank you!
    0
  • cPanelMichael
    Also, please note the "Something weird happened while scanning. Error 2133571361" message, while running both commands.

    Hello :) Could you review /var/log/messages at the corresponding time this error message was generated to see if you notice any specific output? I've seen instances where this stems from a failing hard drive. Thank you.
    0
  • ouki
    Hello, Thank you for your reply! I searched /var/log/messages for "error" and found several messages like the following: kernel: [14859083.169308] Buffer I/O error on device sda8, logical block 207389274 kernel: [14772771.647017] Buffer I/O error on device sda8, logical block 207389308 kernel: [14859083.169101] sd 0:0:0:0: SCSI error: return code = 0x08100002 kernel: [14681695.865730] php[18675]: segfault at 00002b80cd721479 rip 000000000072d910 rsp 00007fff536cf1a0 error 4 This server has 2 SATA disks using hardware RAID 1. RAID monitoring (provided by data center), shows disks as healthy.
    0
  • cPanelMichael
    You may want to search for the specific time stamps associated with the /scripts/fixquotas commands as opposed to only searching for the "error" term so that you can determine if there are any specific messages associated with that particular quota error. Thank you.
    0
  • ouki
    Hello, I just checked logs around the time i've last ran initquotas and fixquotas commands. I didn't find something special, or I am missing it because I am not sure what to look for. Could you please tell me what to look for? Thank you!
    0
  • cPanelMichael
    You are looking for error messages that suggest a damaged hard drive. EX:
    Buffer I/O error
    You can also review /var/log/dmesg for additional output, or utilize a utility such as smartctl: Linux utility for Disk health Monitoring Thank you.
    0
  • ouki
    Hello, Thank you for your reply. You are right! Indeed I found the following messages in /var/log/messages:
    kernel: [14468436.849848] sd 0:0:0:0: Unhandled sense code kernel: [14468436.849852] sd 0:0:0:0: SCSI error: return code = 0x08100002 kernel: [14468436.849854] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK kernel: [14468436.849927] sda: Current: sense key: Hardware Error kernel: [14468436.849929] Add. Sense: Internal target failure kernel: [14468436.849934] kernel: [14468436.849996] Buffer I/O error on device sda8, logical block 207389274 kernel: [14468436.850067] Buffer I/O error on device sda8, logical block 207389275 kernel: [14468436.850134] Buffer I/O error on device sda8, logical block 207389276 kernel: [14468436.850203] Buffer I/O error on device sda8, logical block 207389277 kernel: [14468436.850270] Buffer I/O error on device sda8, logical block 207389278 kernel: [14468436.850342] Buffer I/O error on device sda8, logical block 207389279 kernel: [14468436.850412] Buffer I/O error on device sda8, logical block 207389280 kernel: [14468436.850479] Buffer I/O error on device sda8, logical block 207389281 kernel: [14468436.850548] Buffer I/O error on device sda8, logical block 207389282 kernel: [14468436.850624] Buffer I/O error on device sda8, logical block 207389283 kernel: [14468436.952903] sd 0:0:0:0: Unhandled sense code kernel: [14468436.952905] sd 0:0:0:0: SCSI error: return code = 0x08100002 kernel: [14468436.952908] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK kernel: [14468436.952975] sda: Current: sense key: Hardware Error kernel: [14468436.952977] Add. Sense: Internal target failure kernel: [14468436.952980] kernel: [14468437.037448] sd 0:0:0:0: Unhandled sense code kernel: [14468437.037451] sd 0:0:0:0: SCSI error: return code = 0x08100002 kernel: [14468437.037454] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK kernel: [14468437.037525] sda: Current: sense key: Hardware Error kernel: [14468437.037527] Add. Sense: Internal target failure kernel: [14468437.037531] kernel: [14468437.088533] sd 0:0:0:0: Unhandled sense code kernel: [14468437.088536] sd 0:0:0:0: SCSI error: return code = 0x08100002 kernel: [14468437.088538] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK kernel: [14468437.088606] sda: Current: sense key: Hardware Error kernel: [14468437.088608] Add. Sense: Internal target failure
    I opened a ticket with the data center and I am awaiting for their reply. I hope there is a way to replace disks (RAID 1) without losing any data. Thank you!
    0
  • cPanelMichael
    Thank you for updating us with the outcome. You should be able to backup or transfer your accounts as long as it's just quotas that are affected at this point: Transfer Tool - Documentation - cPanel Documentation Thank you.
    0
  • ouki
    Thank you both you and vanessa for your support! I am on a discussion with technicians in data center about the best method to deal with this situation.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post