Skip to main content

Re: CPANEL-50629 - /dev/null file has become a regular file

spaceman

https://support.cpanel.net/hc/en-us/articles/36894715049751-The-dev-null-file-has-become-a-regular-file

/dev/null is destroyed and recreated as a regular file every morning during the 0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1 root cron run, on two independent servers. We believe this matches known case CPANEL-50629 ("The /dev/null file has become a regular file").

We instrumented both servers with a bpftrace VFS tracer (kprobes on vfs_unlink, vfs_mknod, vfs_rename, security_inode_create, filtered on dentry name "null") so that attribution would be independent of mount namespace, bind mounts, or the specific syscall path used — auditd rules alone had proven unable to see the event (see below). The very first night the tracer was live it caught the clobber on both servers simultaneously, at the exact second of the impostor file's Birth timestamp

The process responsible is exim_tidydb, running as uid 0 (root), with its immediate parent process also named exim_tidydb (i.e. the cPanel wrapper /usr/local/cpanel/scripts/exim_tidydb, invoked by the daily root cron 0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1, execs or forks the real exim exim_tidydb binary, and that child performs an UNLINK immediately followed by a CREATE of a dentry literally named null, with parent the devtmpfs root /dev itself — not an open() of the existing device). This is consistent with an atomic rewrite pattern (write-temp-then-rename, or rename-aside-then-recreate) applied to one of Exim's hint/retry databases, executed from an incorrect working directory or with an empty/misresolved path variable so that the operation lands in /dev instead of /var/spool/exim/db. No other process touched the null dentry in the 05:58–06:05 window on either host; this is a clean, repeatable, two-independent-host confirmation

  1. Is this CPANEL-50629? What is its status, and which cPanel/exim build carries the fix?
  2. Given the trace evidence above showing exim_tidydb (uid 0, parent also exim_tidydb) unlinking+recreating a dentry named null at /dev at the exact moment of corruption on two independent servers — can cPanel confirm the specific code path in scripts/exim_tidydb (or the exim exim_tidydb binary it invokes via Cpanel::SafeRun::Object) that performs this rename/recreate, and why it would resolve to /dev/null rather than the intended /var/spool/exim/db/* hint database path?
  3. Workaround guidance while unfixed? (We have deployed a 60-second self-healing systemd timer as a stopgap containment measure — restores /dev/null to the correct c 1,3 device within ~60s of every occurrence — but obviously would prefer the upstream fix.)

Instrumentation used to gather this evidence

bpftrace v0.24.2 (EL9 AppStream) VFS tracer, kprobes on vfs_unlink/vfs_mknod/vfs_rename/ security_inode_create filtered on dentry name "null" — logs pid/comm/uid/ppid/parent-comm regardless of path, bind mount, or mount namespace. Validated with synthetic create/rename/unlink events on both production kernels before relying on it for attribution. (Separately: persistent auditd rules devnull_kill/devnull_kill2 plus a time-boxed 05:58–06:05 audit window were also in place but did not catch the event — the clobber operates through a devtmpfs-internal path that path-based audit watches on /dev/null don't see; the bpftrace kprobe approach was necessary because it hooks the VFS layer itself rather than a specific path.)

 

Comments

9 comments

Date Votes
  • cPRex Jurassic Moderator

    Hey there!  This issue does certainly sound like CPANEL-50629, but it's actually not something we fixed.  At the time the case was active we determined that this was caused by third-party software and not something inside the cPanel system.  All reports that we had around this problem with /dev/null were related to this tool:

    https://github.com/netdata/netdata/issues/21427

    and they patched the software to resolve the problem.

    Would you happen to have that tool installed on your machine?

    0
  • spaceman

    Thanks for the quick response. We checked both affected servers immediately: netdata is not installed and never has been — no netdata packages, no /opt/netdata, /etc/netdata, /usr/libexec/netdata or /var/lib/netdata, no netdata processes, no netdata-updater cron entries. So netdata cannot be the cause on these hosts.

    The mechanism also differs from netdata#21427. That bug rm -f'd /dev/null via a misresolved variable, leaving the device deleted. What we capture is an atomic unlink + create pair by a single process, recreating /dev/null as a root-owned regular file (mode 0600 or 0666 depending on the night) — and that process is cPanel's own exim_tidydb.

    We have now caught it on three consecutive nights on both servers with a kernel-level bpftrace tracer (kprobes on vfs_unlink/security_inode_create, validated with synthetic events before use). Latest capture, 2026-07-06:

    [CloudLinux server]: 06:00:04 UNLINK parent=/ pid=3924026 comm=exim_tidydb uid=0 ppid=3923960 pcomm=exim_tidydb
    [CloudLinux server]: 06:00:04 CREATE parent=/ pid=3924026 comm=exim_tidydb uid=0 ppid=3923960 pcomm=exim_tidydb
    [AlmaLinux server]: 06:00:02 UNLINK parent=/ pid=2460859 comm=exim_tidydb uid=0 ppid=2460828 pcomm=exim_tidydb
    [AlmaLinux server]: 06:00:02 CREATE parent=/ pid=2460859 comm=exim_tidydb uid=0 ppid=2460828 pcomm=exim_tidydb

    Each time the imposter's statx Birth matches the traced CREATE to the sub-second (e.g. 06:00:04.182939779). Both servers run the stock root cron 0 6 * * * /usr/local/cpanel/scripts/exim_tidydb >/dev/null 2>&1 — one CloudLinux 9.8, one AlmaLinux 9.8, both cpanel-exim-4.99.4-1.cp130~el9. We suspect the reports you received around Dec 2025 coincided with the (since-patched) netdata bug, but this is a distinct cause inside the exim_tidydb maintenance chain.

    Given that, could CPANEL-50629 be reopened or a new case raised? The event reproduces every morning at 06:00 like clockwork, and we're happy to provide full trace logs or run additional diagnostics on an affected host.

    0
  • cPRex Jurassic Moderator

    Let me talk to the team and see what they'd like the plan to be and I'll let you know!

    0
  • cPRex Jurassic Moderator

    Alright, I have a few thoughts now.

    When this initially came to our attention our team wasn't able to reproduce the issue on a plain cPanel server, as all the previous reports had that tool I linked.  Since that is the case, we don't have any data from live servers on our end about this.

    If I'm remembering correctly, you also aren't able to open tickets directly with us, so that makes things a bit more complicated.

    Here's what I know based on looking at the way our system and exim_tidydb runs.  You're correct that the exim_tidydb process, which runs at 6 by default, runs the tool that triggers this behavior.  If that binary exits with code 1 it assumes the file is corrupt, then it hard-links it aside to a .broken.<ISO-timestamp> name, then unlinks the original path. Exim then recreates a fresh, empty Berkeley DB file at that path on its next use. That's the "unlink and replace" you're seeing.

    Why /dev/null would trigger it

    /dev/null is not a valid Berkeley DB, so exim_tidydb returns error code 1 on it. So the scenario is almost certainly: one of those db paths was pointed at /dev/null (possibly from someone running something like "ln -sf /dev/null /var/spool/exim/db/retry" which is a common trick to "disable"/nullify a hints file, sometimes done by admins or older cleanup scripts).

    Once that's in place, every 6 AM run does:
    1. exim_tidydb sees a non-DB file → exit 1.
    2. Wrapper links it to retry.broken.<timestamp> and unlinks the path (removing the symlink, or decrementing the link count if it was a hard link to the /dev/null inode).
    3. Exim recreates retry as a real regular DB file.

    So, that's a lot of words for me to ask these questions:

    1 - Do you see the retry.broken files on the system located in /var/spool/exim/db or possibly /var/spool/exim_incoming/db if your machine has it??  
    2 - Do you see any symlinks to /dev/null in those directories?
    3 - Confirm that /dev/null is crw-rw-rw- permissions.

    Let me know those three things and we can go from there.

    0
  • spaceman

    Thanks - quick status update, full write-up to follow tomorrow.

    Answers to your three questions:
    (1) no .broken files in /var/spool/exim/db on either server (no exim_incoming present);
    (2) no symlinks to /dev/null anywhere under /var/spool/exim;
    (3) /dev/null is crw-rw-rw- 1,3, though only because our auto-heal restores it each morning.

    Beyond that: we've now reproduced the corruption on demand by running the exim_tidydb cron command under strace, and we've root-caused the full chain - it involves /usr/sbin/exim_tidydb segfaulting (SIGSEGV, core dumped) in combination with a kernel core-dump interaction on our side, not symlinks or netdata. We've applied a config fix on our side and expect tomorrow's 06:00 run to produce a full core backtrace of the exim_tidydb crash, plus the hints DB file it crashes on.

    Will report back when we have more.

    0
  • cPRex Jurassic Moderator

    That actually seems like it might be a slight different, but related issue.  All the ones we say had those .broken files or a symlink and that explained why things were breaking, but you may be in uncharted territory here.

    Let me know if you have more details.  I know you can open a ticket with your provider, but that's not always a great experience for some users.

    0
  • spaceman

    Yes, our provider aren't cPanel/WHM experts per se, so likely all they'd be doing is passing messages between the two of us.

    Other than this community forum, do you have any other secure methods we can exchange more details, if necessary?

    In the meantime... agreed it's a different mechanism from the .broken/symlink cases, and we can now explain it completely. Full write-up as promised:

    Root cause - two factors combining:

    1. cPanel's /usr/sbin/exim_tidydb segfaults every daily 06:00 run on both of our servers (CloudLinux 9.8 and AlmaLinux 9.8, cpanel-exim-4.99.4-1.cp130~el9), specifically on the wait-remote_smtp hints DB: /usr/sbin/exim_tidydb /var/spool/exim wait-remote_smtp → SIGSEGV, core dumped. Reproducible on demand outside cron.
    2. Our /etc/sysctl.conf carried kernel.core_pattern = /dev/null (a legacy "discard cores" tuning line). This is the trap: when a process dumps core to a file path, the kernel's do_coredump() (fs/coredump.c) unlinks any pre-existing file at that path and recreates it O_CREAT|O_EXCL mode 0600, in the crashing task's context, via in-kernel calls. So every root segfault silently replaced the /dev/null char device with a root-owned 0600 regular file. The in-kernel path is why syscall-based auditd rules never saw it, while our bpftrace kprobes attributed the UNLINK+CREATE to comm=exim_tidydb - the crashing process itself.

    We proved factor 2 exim-independently: bash -c 'ulimit -c unlimited; kill -SEGV $$' as root clobbers /dev/null identically with that sysctl set. After commenting the line out and restoring the systemd-coredump default, this morning (2026-07-08) was our first clean 06:00 after daily rec.




    0
  • cPRex Jurassic Moderator

    "Other than this community forum, do you have any other secure methods we can exchange more details, if necessary?"

    Unfortunately I don't - any communication like that would have to be through our ticket system for security reasons.  You always have the option of purchasing your cPanel license directly through us if that is something that interests you.

    I'm so glad you found the root cause though!  That's certainly not something I would have come up with through a forums post.

    0
  • spaceman

    For the record, and for anyone else who lands on CPANEL-50629: the corruption on our end was ultimately triggered by /usr/sbin/exim_tidydb segfaulting during its daily run, combined with a kernel.core_pattern = /dev/null setting on our servers (an old "discard cores" trick) - on that config the kernel unlinks and recreates the core_pattern target on every crash, which is what replaced /dev/null. We've corrected the core_pattern setting and the daily corruption has stopped. Stock cPanel boxes use the systemd-coredump default, so they'd never see the /dev/null damage - which likely explains why your team couldn't reproduce it.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post