Max out inodes issue
Disk usage warning
The filesystem "/usr" mounted at "/usr" reached "critical" status because you currently use 99.74% of its available inodes.
Notification Type: critical
Server: crz01.[redacted]
Primary IP Address:
[redacted]
Filesystem: /usr
Mount Point: /usr
Percent Used: 99.74%
Inode Usage:
Used: 1,045,922
Available: 2,654
Total: 1,048,576
ChkServd Version: 17.0and so on.. now how does a default cPanel install (on centOS 6.x) with a single site running on it max out inodes in /usr? More importantly, how can I fix it?
-
/usr folder has sub folders which contains log files and important cpanel stuff. Check the size of the folders accordingly with du -sh * command and it will give you an idea of what is consuming the disk space. 0 -
mod_sec audit logs was the cause.. one per day in a folder since it was installed/enabled. removed all but the latest and went from 100% inode usage to 17% in /usr 0 -
mod_sec audit logs was the cause.. one per day in a folder since it was installed/enabled. removed all but the latest and went from 100% inode usage to 17% in /usr
Hello, Is this system using EasyApache 3 or EasyApache 4? Can you provide an example of how the directory structure looked so we can verify the logs were being rotated? Thank you.0 -
Hi, Yeah, Its using EA 3. here is a copy/pasta from my ssh shell session: root@crz01 [~]# cd /usr/local/apache/logs/modsec_audit root@crz01 [/usr/local/apache/logs/modsec_audit]# ls -al total 64 drwx-wx-wt 5 root root 4096 Dec 19 07:08 ./ drwxr-xr-x 4 root root 4096 Dec 19 12:00 ../ drwxr-x--- 3 504 515 4096 Sep 15 2015 crzaccou/ -rw-r--r-- 1 504 515 1 Sep 15 2015 crzaccou.offset drwxr-x--- 3 505 516 4096 Sep 15 2015 crzadmn/ -rw-r--r-- 1 505 516 1 Sep 15 2015 crzadmn.offset -rw-r--r-- 1 crzscore crzscore 1 Nov 21 2016 crzscore.offset drwxr-x--- 4 nobody nobody 32768 Dec 19 02:05 nobody/ root@crz01 [/usr/local/apache/logs/modsec_audit]# cd crzadmn/ root@crz01 [/usr/local/apache/logs/modsec_audit/crzadmn]# ls -al total 12 drwxr-x--- 3 505 516 4096 Sep 15 2015 ./ drwx-wx-wt 5 root root 4096 Dec 19 07:08 ../ drwxr-x--- 4 505 516 4096 Sep 15 2015 20150915/ root@crz01 [/usr/local/apache/logs/modsec_audit/crzadmn]# cd ../nobody root@crz01 [/usr/local/apache/logs/modsec_audit/nobody]# ls -al total 56 drwxr-x--- 4 nobody nobody 32768 Dec 19 02:05 ./ drwx-wx-wt 5 root root 4096 Dec 19 07:08 ../ drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 20171218/ drwxr-x--- 83 nobody nobody 4096 Dec 19 07:05 20171219/ root@crz01 [/usr/local/apache/logs/modsec_audit/nobody]# cd 20171218/ root@crz01 [/usr/local/apache/logs/modsec_audit/nobody/20171218]# ls -al total 1188 drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 ./ drwxr-x--- 4 nobody nobody 32768 Dec 19 02:05 ../ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:00 20171218-0000/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:05 20171218-0005/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:10 20171218-0010/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:15 20171218-0015/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:19 20171218-0019/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:20 20171218-0020/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:25 20171218-0025/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:30 20171218-0030/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:35 20171218-0035/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:40 20171218-0040/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:45 20171218-0045/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:49 20171218-0049/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:50 20171218-0050/ drwxr-x--- 2 nobody nobody 4096 Dec 18 00:55 20171218-0055/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:00 20171218-0100/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:05 20171218-0105/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:09 20171218-0109/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:10 20171218-0110/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:15 20171218-0115/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:20 20171218-0120/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:25 20171218-0125/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:30 20171218-0130/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:35 20171218-0135/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:40 20171218-0140/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:45 20171218-0145/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:50 20171218-0150/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:51 20171218-0151/ drwxr-x--- 2 nobody nobody 4096 Dec 18 01:55 20171218-0155/ [shortened] drwxr-x--- 2 nobody nobody 4096 Dec 18 16:30 20171218-1630/ drwxr-x--- 2 nobody nobody 4096 Dec 18 16:34 20171218-1634/ drwxr-x--- 2 nobody nobody 4096 Dec 18 16:35 20171218-1635/ drwxr-x--- 2 nobody nobody 4096 Dec 18 16:40 20171218-1640/ drwxr-x--- 2 nobody nobody 4096 Dec 18 16:45 20171218-1645/ root@crz01 [/usr/local/apache/logs/modsec_audit/nobody/20171218]# cd 20171218-1645/ root@crz01 [/usr/local/apache/logs/modsec_audit/nobody/20171218/20171218-1645]# ls -al total 20 drwxr-x--- 2 nobody nobody 4096 Dec 18 16:45 ./ drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 ../ -rw-r----- 1 nobody nobody 2080 Dec 18 16:45 20171218-164501-Wjg23UpWraoAAAEufK0AAAAD root@crz01 [/usr/local/apache/logs/modsec_audit/nobody/20171218/20171218-1645]#
etc... note that this is under the 'nobody' apache account. the end user account(s) don't seem to be having the issue.. (are rotated) it does that for every day in its dir.0 -
Hello, It's possible this is isolated to EasyApache 3. Is there anything in-particular we can do to help you convert the system to EasyApache 4? Please see: EasyApache 3: It's been a long road, but it will be time to say goodbye soon. Thank you. 0 -
Not sure.. getting a bunch of warnings when the pre-flight checks are run: /snag.gy/yP6nGK.jpg safe to ignore or ?? 0 -
Hello, Those are non-fatal errors, and are displayed to note a difference in the EA4 profile compared to the EA3 profile: SOLVED - Migrate EA3 to EA4 issues The primary notice I see relates to suPHP, as it is not compatible with the mpm_itk or the mod_ruid2 Apache module. Thus, the EasyApache 4 conversion script is letting you know it will not install suPHP since you are using Mod_Ruid2. You can review the following documents to determine which PHP handlers and MPM options are supported: Multi-Processing Modules - MPMs - EasyApache 4 - cPanel Documentation PHP Handlers - EasyApache 4 - cPanel Documentation Let us know of any additional questions. Thank you. 0 -
well looks like that server migrated just fine (so far) then again it's only hosting a single site. Will update if anything strange happens or if that inode / log issue returns. 0 -
The OP looks satisfied, so I hope I'm not hijacking this thread... I have a similar problem: The filesystem "/var/lib/snapd/snap/emacs/966" mounted at "/var/lib/snapd/snap/emacs/966" reached "critical" status because you currently use 100% of its available inodes. I've verified this is indeed a mountpoint, though why and what all the consequences of this are are lost on me. I don't believe any files (like logs) get added to this hierarchy at runtime, I think it's just the result of my snap installation of emacs. I've run the du -sh over it and the big components seem to be runtime libraries and lisp. What, if anything, do I do about this? 0 -
@Macs R We - that's definitely an odd mount path. You may want to speak with your host to see why that is setup like that, as I haven't come across such a path before. 0 -
@Macs R We - that's definitely an odd mount path. You may want to speak with your host to see why that is setup like that, as I haven't come across such a path before.
My hosting provider isn't going to know what's going on. I am the admin, and I had a fresh machine recently on which the Emacs was gone, so I did a web search for installing Emacs on CentOS. I found instructions from what seemed to be a reputable source (Install GNU Emacs on CentOS using the Snap Store | Snapcraft) for doing it with a service called snap. So I just ran the two or three commands on the webpage, and that's what I ended up with. It works great as being able to use Emacs, but about a month later I'm getting these messages, and I have no idea what's going on under the hood here. I didn't customize anything, the installation was pretty simple and looked a lot like services I had used before on other UNIX platforms.0 -
I don't know if it's going to help much, but can you cd into /var/lib/snapd/snap/emacs/966 (so you are in that mount point) and then run: df -i .
That at least will show you how many inodes are available and how many are used.0 -
I don't know if it's going to help much, but can you cd into /var/lib/snapd/snap/emacs/966 (so you are in that mount point) and then run:
df -i .
That at least will show you how many inodes are available and how many are used.
Thanks for the reference.Filesystem Inodes IUsed IFree IUse% Mounted on /dev/loop1 24398 24398 0 100% /var/lib/snapd/snap/emacs/966
I have no idea what that is, so I stat-ed it:File: "loop1" Size: 0 Blocks: 0 IO Block: 4096 block special file Device: 5h/5d Inode: 12362 Links: 1 Device type: 7,1 Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) Access: 2020-12-01 20:35:54.284724168 -0700 Modify: 2020-12-01 20:35:35.720000000 -0700 Change: 2020-12-01 20:35:35.720000000 -0700 Birth: -
I'm out of my depth here, especially since I did not set up this VM.0 -
That's definitely something to speak with the host about. I have no idea why someone would want or need a separate partition for emacs tools. 0 -
That's definitely something to speak with the host about. I have no idea why someone would want or need a separate partition for emacs tools.
Well, the host didn't have any input to this as far as I know. It must have been done by the Snap installation tool. I guess I'll ask there.0 -
The snapcraft folks explained this 100% to my satisfaction (below). So I can either find a control in WHM to tell it not to run that particular check on that filesystem, or at worst, just ignore the warning messages. Snaps are distributed as compressed squashfs filesystems, rather than being extracted, they are mounted at runtime, this applies to all snaps and not just the Emacs snap. This is similar to the approach AppImage takes, and certain application installers on MacOS. It provides some benefits, and to some people some disadvantages in that it appears in "lsblk" output and similar, and similarly this is why your software is concerned that it sees a filesystem that is 100 percent full. But it"s supposed to be 100 percent full anyway, because being less than fully at capacity would just be wasted space, the automatic updates will replace the mountpoint rather than add to it, so theres nothing to worry about. Tl:;Dr it"s working as designed, you can ignore the error in this case and if you can exclude a directory from being checked, safely silence it by excluding /var/lib/snapd/snap 0 -
This is definitely what's going on here... but I can't find any configuration tweaker in WHM to tell the system not to scan those pathnames. Is there one? 0 -
Thanks, I just did this. 0 -
v94 is out. Fix requires addition of mountpoints to a new table, /var/cpanel/chkservd_ignored_mounts, but I can't find any guidance as to its syntax. Asking over on the buglist website just results in my additions being "held for moderation" -- an earlier question of mine is still being held after two months, so I am not optimistic about ever getting seen there. Any help here? 0 -
@Macs R We - do you mean the feature request site? If so, send me the links that are waiting on approval and I can make this happen :D 0 -
Thanks for the details - it should be showing up soon! 0
Please sign in to leave a comment.
Comments
23 comments