Skip to main content

Accounts Showing 404 Error

Kyle
  • Edited

I'm hoping someone can point me in the right direction here.  The last two days I've been dealing with some issues where an account will not load and simply show a "404 Not Found" error no matter the page that's trying to be accessed.  It first happened this morning on one account after a server reboot.  I second reboot cleared it up.  Tonight I created a new account, and it's doing the same thing.  Any attempt to access any page on the site brings up a "404 Not Found" message.  I even cleared out everything and tried a simple html file with just "hello world" in it, and nothing.

Configuration is pretty basic, no NGINX or CloudLinux or anything like that.  EasyApache 4 with PHP 8.3.  On the latest stable cPanel Version 120.0.5.

It feels like whatever tells Apache where the files for the account just isn't loading properly.  But honestly I'm not sure where to begin to troubleshoot this.  Any thoughts?

Edit: I just wanted to add, even shortcuts like https://www.newdomain.com/cpanel don't work (which redirect to https://www.newdomain.com:2083/ for all other accounts on the server).  It all simply shows up in the Apache log as:

68.234.48.21 - - [09/May/2024:22:16:29 -0400] "GET /cpanel HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

 

Edit 2: As I continue down the rabbit hole of troubleshooting, I stumbled on this command which has proved to show interesting results:

[root@server ~]# /scripts/update_users_jail myaccount
jailshell: failed to create symlink: usr/lib -> /home/virtfs/myaccount/lib
jailshell: failed to create symlink: usr/lib64 -> /home/virtfs/myaccount/lib64
jailshell: failed to create symlink: usr/bin -> /home/virtfs/myaccount/bin

I tried running the command on a known working account, and it returns nothing.  So this clearly is indicative of the problem at hand...

Comments

7 comments

Date Votes
  • Kyle

    Alright... not really understanding this documentation:

    https://docs.cpanel.net/knowledge-base/accounts/virtfs-jailed-shell/

    And since I don't offer my users shell access (it's disabled for everyone in Account Functions > Manage Shell Access), I saw at the bottom of this documentation this command: The clear orphaned VirtFS mounts script

    /usr/local/cpanel/scripts/clear_orphaned_virtfs_mounts --clearall

    And so I ran it thinking maybe something on this account broke and that's what's causing my issues.  Well, after running that command, it successfully broke every site on my server and they were all returning the "404 Not Found" error. 😑

    However I then ran /scripts/update_users_jail accountnamehere for each account, including the one that was broken, and it actually worked...

     

    At this point... I'd love some clarification as to what just happened here...

    0
  • cPRex Jurassic Moderator

    Hey there!  Just to confirm, you're saying that after you want the update_users_jail command for each account, the 404 page issue was resolved and things started working normally?

    0
  • Kyle
    • Edited

    Hello there cPRex,

    Correct - a new account I made was having this 404 issue.  I ran the update_users_jail command but got an error.  But then after I ran

    /usr/local/cpanel/scripts/clear_orphaned_virtfs_mounts --clearall

    and broke every site, running the  update_users_jail command for each account fixed each one, including the one that initially didn't work in the first place.

    Is this indicative of some larger issue that I need to address? Or something I should investigate further?

    0
  • cPRex Jurassic Moderator
    • Edited

    That's odd to me, because those really shouldn't be related.

    It sounds like there is something else happening on the system.  Could you submit a ticket so we can check it out, if your license is purchased through us?

    0
  • Kyle

    Sure thing!  Thank you.

    1
  • Kyle

    Just a quick update on this issue (for future folks if they run into this).

    I received word from cPanel support a while back with some suggestions.  I didn't get back to them only because it's a production server and if it's working... I don't really want to mess with it.  But, as with how things go, today I needed to reboot the server because it hung and, as luck would have it, a handfull of sites re-exhibited the issues previously discussed here, 404 error and no obvious reason or solution.  So back to the support email I received and lets go through it.

    The suggestions focused on:
    1. Verify cPanel Jailshell
    2. Analyze Apache Error Logs
    3. Check File Permissions

    The Apache Error logs I reviewed and didn't see anything abnormal, outside of the 404 errors I previously posted about.  And the file permissions were fine... which I would hope so since they should change with a system reboot (I would at least hope so).

    But that first one.... "Verify cPanel Jailshell" was interesting.

    I don't provide my users with shell access.  But I know from the documentation that "You cannot completely remove the jailed shell system (VirtFS)."  Support elaborated in their response:

    The error messages you received from /scripts/update_users_jail myaccount indicate an issue with cPanel Jailshell creating symbolic links. This functionality is crucial for isolating user accounts.

    Double-check if cPanel Jailshell is enabled in WHM. Navigate to "Home >> Server Configuration >> Tweak Settings" and ensure "Jailshell" is enabled under the "Security" tab.

    If it's already enabled, try disabling and re-enabling it to see if that resolves the symbolic link issue.

    Under the security tab, I do have the following option: "EXPERIMENTAL: Jail Apache Virtual Hosts using mod_ruid2 and cPanel® jailshell." and it was ticked on.

    If I'm not enabling shell access for any of my users... can I turn off this "experimental" feature?

    Turned it off, and after a minute or two, the server did "something" and now all the sites, including the ones showing "404" errors, are back online.

    I know why it's on.  In the Security Advisor screen, I get the critical warning that "Apache vhosts are not segmented or chroot()ed." and the solution is to install mod_ruid2 and enable this experimental option in Tweak Settings.  So I go back, turn it back on, and after a few minutes I see all of my sites are having trouble making a database connection.  I try to restart MySQL and I get a configuration error has occurred and MySQL failed to start.

    Turned off the experimental Jailshell option in tweak settings, wait a minute or two, then restarted MySQL, and all is right in the world.

    So... I guess the question is, is this a major vulnerability to have this experimental feature turned off?  Is there a way to have it on and work reliably?  I'm not going to touch it any more right now, it's the middle of the day for my clients and I don't want to anger them further! But this really has me scratching my head.  I almost feel like spinning up a new server and just move them off thinking that something within the bowels of cPanel is borked (honestly, probably my doing), and I just need a fresh start.

     

    0
  • cPRex Jurassic Moderator

    It's impossible to say without knowing what the root cause is, really.  We'd prefer that be enabled for security, and while it affects jailshell it also provides protection for Apache.  In addition to jailshall tools, it does the following:

    You do not need to enable or disable JailManager in the Service Manager interface (WHM » Home » Service Configuration » Service Manager) because this setting controls the module’s state.

        The mod_ruid2 module uses the chroot command on Apache virtual hosts if you enable this setting. This action runs Apache virtual hosts in an environment with an altered root directory.
        You can use this setting when you compile Apache through EasyApache and you have installed mod_ruid2 version 0.9.4a or later.
        You can use this setting on AlmaLinux OS, Rocky Linux™, or Ubuntu® servers. The CloudLinux™ operating system does not support the mod_ruid2 module.

    Those details are taken from our docs here:

    https://docs.cpanel.net/whm/server-configuration/tweak-settings/

    That being said, have zillions of cPanel servers run fine for years without that enabled?  They sure have.

    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post