Skip to main content

Subject: System-wide corruption and "0 bytes" binaries after disk filled to 100%

Angel Katsikandarakis

Hello,

I am facing a critical issue on my server (AlmaLinux 8 / cPanel v114). The disk reached 100% usage, which resulted in widespread file corruption within the cPanel directory structure.

Current Situation:

  • SSH Access: Intermittent/Lost after reboot.

  • WHM Access: ERR_CONNECTION_REFUSED (Port 2087).

  • Mail Services: Completely down. Users receive "Socket errors".

  • Binaries: Several core binaries, including /usr/local/cpanel/cpsrvd, were found to be 0 bytes in size.

Steps Taken So Far:

  1. Disk Space: Freed up space; current usage is well below limits.

  2. Service Restarts: Attempted /scripts/restartsrv_cpsrvd and restartsrv_cpdoveauthd, but received No such file or directory. It appears the /scripts symlink or directory is missing/corrupted.

  3. Update Attempts:

    • Ran /usr/local/cpanel/scripts/upcp --force. It reported "Maintenance complete", but many files are still missing.

    • Attempted sh /root/cpupdate.sh --force. It failed with the error: "You must install cPanel & WHM on a clean server".

  4. Service Status: systemctl restart cpdoveauthd returns Unit not found. Manual execution of /usr/local/cpanel/cpsrvd --debug yields no output (due to the 0-byte file corruption).

  5. Firewall: Disabled csf (csf -x) to rule out network blocking, but connection is still refused.

Specific Errors:

  • -bash: /scripts/restartsrv_cpdoveauthd: No such file or directory.

  • (XID cknxf6) The “cpdoveauthd” service is not a known service.

  • ls -lh /usr/local/cpanel/cpsrvd shows -rwx------ 1 root root 0 May 4 04:41 /usr/local/cpanel/cpsrvd.

Request: How can I force a complete re-installation of all cPanel binaries and scripts without performing a clean OS install, as I need to preserve client data? Is there a way to override the "clean server" check in the installer to repair the current corrupted environment?

Comments

7 comments

Date Votes
  • cPRex Jurassic Moderator

    Hey there!  I'm not seeing a "/scripts/restartsrv_cpdoveauthd" on a test system so it seems that missing is normal behavior.

    However, not having the cpsrvd script isn't normal.

    I would try running /scripts/check_cpanel_pkgs to see if that helps get missing tools installed, and then do an additional upcp after that.  Can you try that and let me know the results?

    0
  • Angel Katsikandarakis

    Dear cPRex and cPanel Support Team,

    I am providing an update and requesting a definitive migration strategy for my agency. I need to move from a "repair" attempt to a "migration" workflow immediately.

    1. Current Status of the Corrupted Server

    The original server (AlmaLinux 8 / cPanel v114) experienced 100% disk usage, causing severe binary corruption:

    • The Problem: Core cPanel binaries (like /usr/local/cpanel/cpsrvd) are 0 bytes.
    • Access Issues: SSH is broken/denied, and WHM is unreachable. chroot via Rescue System fails with a Segmentation fault.
    • Active Services: Despite the cPanel corruption, the websites are currently online, and the MySQL service is still running and serving data.
    1. New Environment Ready
    • Drive A (Corrupted/Active): Original NVMe with all client data, running LiteSpeed and MySQL.
    • Drive B (Fresh): A second NVMe with a clean installation of AlmaLinux 9. cPanel is not yet installed.
    1. Critical Questions for Migration

    Given that the corrupted OS cannot be managed via WHM or SSH, but the sites and MySQL are still active:

    1. Transfer Tool via Local Mount: Can I install cPanel on the fresh AlmaLinux 9 drive and use the Transfer Tool to migrate accounts directly from the corrupted drive if I mount it locally?
    2. Live MySQL Migration: What is the best practice for migrating MySQL databases from a drive where the OS is corrupted but the MySQL service is currently active? Should I attempt a mysqldump from the Rescue System or copy the data directory?
    3. LiteSpeed Preservation: Since the old server uses LiteSpeed, what is the correct sequence to ensure these settings migrate correctly to the new AlmaLinux 9 environment?

    Please provide a definitive recommendation. I cannot risk further downtime by attempting to fix a "clinically dead" OS. I need to migrate to the new drive now.

     

    0
  • cPRex Jurassic Moderator

    I'm sorry to hear about the situation.  I'll make sure to answer these questions in order to ensure we don't miss anything.

    1 - No, there isn't a way to do this with any tools provided by cPanel.  The fastest way to get things working would be to restore from backups.  While you could perform a manual restoration from the corrupted drive it would be a process of manually creating each cPanel account and manually moving the data to the correct areas - a very tedious job that is both error prone and slow.

    2 - Since this isn't work that cPanel handles I don't have a recommendation on this one.  We just don't have tools for this, so I can't tell you what the best option is.  Ultimately, restoring from a backup would ensure the data isn't corrupted.

    3 - It would depend which settings you're talking about here.  As LiteSpeed is a drop-in replacement on cPanel systems, many of the settings would be standard Apache values, which you may be able to find in /etc/apache2/conf/httpd.conf.  LiteSpeed-specific values that are *not* handled by Apache can be found in /usr/local/lsws/conf/httpd_config.xml

    I'm sorry I don't have better options for you, although I still think it would be worth a ticket to have an admin check the cPanel installation to see if it can be resolved.  Running a normal "yum update" to confirm the operating system itself is working and then following up with "/scripts/check_cpanel_pkgs" would tell you a lot about the state of the machine.

    0
  • Angel Katsikandarakis

    I am experiencing a fatal error while installing cPanel on a fresh AlmaLinux 9 server (Hetzner).

    The main issues are:

    1. Installation Error: The installer fails with Can't locate DBI.pm in @INC. It seems the internal cPanel Perl environment is corrupted or incomplete.

    2. Filesystem Lock: I am unable to perform a clean re-installation because the directory /usr/local/cpanel contains immutable files. Commands like rm -rf and chattr -R -i return "Operation not permitted," even as root.

    3. Environment: The server has a broken RAID 1 configuration (Disk A: AlmaLinux 8, Disk B: AlmaLinux 9). I am installing on Disk B.

    I have already installed MariaDB-devel and perl-DBD-MySQL at the system level, but the installer still fails.

    I have enabled the Hetzner Rescue System if needed, or I can provide SSH credentials for the current AlmaLinux 9 environment. Please advise on how to proceed with a forced clean installation."

    0
  • cPRex Jurassic Moderator

    You shouldn't be installing *anything* before a cPanel installation.  It should be a clean installation and then you would immediately install cPanel after that.

    You can create a ticket through https://docs.cpanel.net/whm/support/create-support-ticket/#open-a-technical-support-ticket since you don't have access to a server at the moment.

    0
  • Angel Katsikandarakis

    I want to clarify that I was indeed performing a clean installation on a fresh AlmaLinux 9 environment. The files and errors currently on the system are remnants of the cPanel installer itself after it failed during the initial run. It is not pre-installed software.

    I currently have full SSH access, but I am unable to clear the corrupted files (immutable attributes) or fix the Perl DBI dependency errors created by the failed attempt. I am trying to open a technical ticket, but the system identifies my license as "Partner Supported" by Hetzner.

    Since this is a specific internal installer failure that Hetzner cannot resolve, could you please advise on how to bypass the partner check so your technical team can assist with this "stuck" installation?

    0
  • cPRex Jurassic Moderator

    In general, we don't clean "stuck" installations like this - we would just recommend to build another system and perform a fresh install.

    Since your license is purchased through Hetzner they should be the ones offering you support, and then escalating the issue to us if needed.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post