Skip to main content

EA4 and Bluehost Apache patch.

Spork Schivago
Hi. I rented a VPS and purchased a license for cPanel through buycpanel.com. I had a VPS from a previous company and things are a bit different with the new company and the full version of cPanel. I run the security advisor and see:
Apache Symlink Protection: the Bluehost provided Apache patch is in effect It appears that the Bluehost provided Apache patch is being used to provide symlink protection. This is less than optimal. Please review Symlink Race Condition Protection.
But I'm not sure how the Apache source code is getting patched. I run CentOS 7. I run:
yum info ea-apache24 Loaded plugins: fastestmirror, replace, universal-hooks Loading mirror speeds from cached hostfile * EA4: 69.72.212.11 * base: mirrors.linode.com * city-fan.org: www.city-fan.org * epel: mirror.math.princeton.edu * extras: mirrors.linode.com * ius: ius.mirror.constant.com * updates: mirrors.linode.com Installed Packages Name : ea-apache24 Arch : x86_64 Version : 2.4.23 Release : 5.7.1.cpanel Size : 7.7 M Repo : installed From repo : EA4 Summary : Apache HTTP Server URL : http://httpd.apache.org/ License : ASL 2.0 Description : The Apache HTTP Server is a powerful, efficient, and extensible : web server.
Does anyone know how my Apache is getting patched with the bluehost Apache symlink protection patch and how I can prevent it from getting patched? I'm running cPanel 11.60.0.25 Release and EA4. Thanks!

Comments

23 comments

Date Votes
  • Vs Nu
    Symlink Patch Protection is Recommended to Avoid symlink attack.it won't cause you any issue's.
    0
  • danielpmc
    Hello Spork, Take a look at this link, the info i posted is on that page. If you choose to UNINSTALL Bluehost make sure you have something in place to protect from SYMLINK attacks. Symlink Race Condition Protection - EasyApache - cPanel Documentation Bluehost.com-provided patch (available via EasyApache): Links See Below Upside You can install this option via EasyApache. Downside [LIST]
  • Protection from this patch is not as good as a kernel-level or a filesystem-level solution.
  • This patch may slow the performance of high-traffic servers.
  • Incompatible with Mailman.
  • Incompatible with CGI Center apps. Hope this helps you out, danielpmc
    • 0
  • Spork Schivago
    Symlink Patch Protection is Recommended to Avoid symlink attack.it won't cause you any issue's.

    Hello. Thank you for responding to my question. I'm sorry if I wasn't clear enough. I understand what the symlink race condition is, I understand the various ways to protect against it. It appears the bluehost symlink race condition patch has been applied to my system and I did not apply it. I am interested in knowing how it got applied and I am interested in removing the patch. Thank you.
    0
  • Spork Schivago
    Hello Spork, Take a look at this link, the info i posted is on that page. If you choose to UNINSTALL Bluehost make sure you have something in place to protect from SYMLINK attacks. Symlink Race Condition Protection - EasyApache - cPanel Documentation Bluehost.com-provided patch (available via EasyApache): Links See Below Upside You can install this option via EasyApache. Downside [LIST]
  • Protection from this patch is not as good as a kernel-level or a filesystem-level solution.
  • This patch may slow the performance of high-traffic servers.
  • Incompatible with Mailman.
  • Incompatible with CGI Center apps. Hope this helps you out, danielpmc

    • Thank you. I will take a look at the link right now. I understand that I need protection in place to protect against the race condition. The idea is to remove the bluehost patch and see if I'm running a patched kernel. If I am not, I will download the source and run a custom compiled kernel, with the GRSecurity patch. The bluehost patch is good for Apache, but I don't think it provides any protection for symlinks created by PHP (or any other process that isn't Apache). In my case, I think the bluehost patch wouldn't provide very much protection at all. Thanks! **EDIT: Daniel, I looked at the link you posted. I have been there before. I believe that link is for people running EA3, not EA4. This is why I'm having such a hard time trying to figure out how to remove the patch. There is no, "Symlink Race Condition Protection" in EA4. I've searched all the options for keywords:
    symlink sym patch bluehost
    And it always comes back with nothing. I've contacted buycpanel.com and they said they didn't patch cPanel. I know there are ways to manually implement the bluehost patch. I'd really like to know how it got enabled on my system, who applied it, and how to remove it. Any more suggestions? Thanks!
    0
  • danielpmc
    Hello Spork Schivago, Your host or buycpanel installed Bluehost as it is a plugin. I suspect they did it because: buycpanel: If they installed Bluehost it is probably to support and/or allow something they install with cPanel. Perhaps another plugin. Solution: Contact support and ask if, why, how to remove and how to secure after removing Bluehost. Your current host: If they installed Bluehost it is probably to support their server setups as some type of workaround. Solution: Contact support and ask if, why, how to remove and how to secure after removing Bluehost. WHMxtra: If you have this system it may have installed Bluehost. Solution: Look in the admin controls and see if you can uninstall and/or secure afterwards. This is just my opinion on how to approach the situation starting out. Unfortunately i can not offer much more advice than this. Please post additional info as you progress with the problem solving of this, as a help to others on such a complex issue. Hope all works out sooner than later, danielpmc
    0
  • Spork Schivago
    Here's the packages that are selected for my EA4 profile:
    Current Profile The currently installed packages on the server. Apache 2.4 ---------- config config-runtime mod_bwlimited mod_cgi mod_deflate mod_expires mod_headers mod_mpm_prefork mod_proxy mod_proxy_fcgi mod_proxy_http mod_ruid2 mod_security2 mod_speling mod_ssl mod_unique_id tools PHP 7.0 ------- libc-client pear php-bcmath php-bz2 php-calendar php-cli php-common php-curl php-devel php-fileinfo php-fpm php-ftp php-gd php-imap php-mcrypt php-mysqlnd php-pdo php-posix php-sockets php-xml runtime Others ------ apr apr-util cpanel-tools documentroot libmcrypt modsec-sdbm-util php-cli profiles-cpanel
    There's a few things that confuse me here. For example, under the Apache 2.4 section, what the heck are the config, config-runtime, and tools package? I don't see them as choices for EA4. Also, what's the Others section all about? I don't remember that before. Could any of these be providing the patch? Maybe I should try reinstalling EA4?
    0
  • danielpmc
    Hello Spork Schivago, 1. What and where is the Apache 2.4 section from? 2. Are you in fact running EA4? 3. What do you mean you dont see them as choices for EA4? 4. Are you comparing logs from your old server and your new one? If you are running 2.4/EA4 and pulling up 2.4/EA3 logs something is not right. danielpmc
    0
  • Spork Schivago
    Hello Spork Schivago, Your host or buycpanel installed Bluehost as it is a plugin. I suspect they did it because: buycpanel: If they installed Bluehost it is probably to support and/or allow something they install with cPanel. Perhaps another plugin. Solution: Contact support and ask if, why, how to remove and how to secure after removing Bluehost. Your current host: If they installed Bluehost it is probably to support their server setups as some type of workaround. Solution: Contact support and ask if, why, how to remove and how to secure after removing Bluehost. WHMxtra: If you have this system it may have installed Bluehost. Solution: Look in the admin controls and see if you can uninstall and/or secure afterwards. This is just my opinion on how to approach the situation starting out. Unfortunately i can not offer much more advice than this. Please post additional info as you progress with the problem solving of this, as a help to others on such a complex issue. Hope all works out sooner than later, danielpmc

    Daniel, When I say bluehost, I'm talking about the Apache 2.4 patch that prevents the symlink race condition. I did contact buycpanel and they did not provide this patch, nor did they install it. They said they do not do that stuff, and if they ever had to, they would contact the customer first. When EA3 provided Symlink Protection, they were using the patch provided by Bluehost. I simply refer to it as the bluehost patch, because Bluehost was the company who created the patch. WHMxtra has been completely removed because I didn't like it, however, a quick google search seems to show it didn't come from them. The Bluehost patch modifies the source code to Apache and then when Apache is compiled, it provides protection. **EDIT: I think I just found the solution to my question: Symlink Race Condition Protection - EasyApache 4 - cPanel Documentation It seems that perhaps cPanel does provide the Bluehost Apache now, but it isn't listed under EA4, like it was with EA3. This is what the cPanel documentation says:
    cPanel & WHM only provides the Bluehost.com-provided patch with some modifications. This is an Apache-level patch and considered a last-resort defense. We provide this patch with EasyApache 4, but the protection is not as reliable as kernel or file system level solutions. This patch can slow the performance of high-traffic servers and is not compatible with Mailman or CGI Center applications. About the Bluehost patch The Bluehost patch's goal is to improve Apache's ability to detect a race condition. The Bluehost patch modifies Apache and APR so that Apache cannot access certain files. The patch helps to ensure that the system can only access files that the domain's owner owns. However, the Bluehost patch only affects requests for static files such as .html and image files. The Bluehost patch does not affect requests that the system processes with application content handlers. Content handlers include the mod_php, mod_ruid2, mod_cgi, and mod_suphp Apache modules. If your system does not handle requests on a per-user basis, then the system serves requests as the nobody user. When an application creates files that the nobody user owns, the file's owner does not match the domain's owner. Because the patch prevents access to static content that does not match the owner, the user cannot view these files. Enable the symlink race condition patch with EasyApache 4 To use the symlink protection patch, select On for the Symlink Protection option in the Global Configuration section of WHM's Apache Configuration interface (Home >> Service Configuration >> Apache Configuration).
    0
  • danielpmc
    Hey Spork Schivago,
    Does anyone know how my Apache is getting patched with the bluehost Apache symlink protection patch and how I can prevent it from getting patched?

    Yes, success! Now we both know it does come from cPanel. Next question is how to uninstall and protect against Symlinks. For that answer i would suggest searching cPanel Forums first, then open a thread: How to uninstall the EA4 Bluehost patch and secure afterwards? I have learned a lot from this, danielpmc
    0
  • danielpmc
    Hello Spork Schivago,
    However, the Bluehost patch only affects requests for static files such as .html and image files.

    Enable the symlink race condition patch with EasyApache 4 To use the symlink protection patch, select On for the Symlink Protection option in the Global Configuration section of WHM's Apache Configuration interface (Home >> Service Configuration >> Apache Configuration).

    Can a person disable this since its only protecting html and images? That and cPanel says it is used as a last resort. What is the first resort? danielpmc
    0
  • Spork Schivago
    Hello Spork Schivago, 1. What and where is the Apache 2.4 section from?

    Hello Daniel, It's from EasyApache 4 (EA4). I click on EasyApache 4 (Home >> Software >> EasyApache 4) and then I click on View all packages (from my current profile).
    2. Are you in fact running EA4?

    Yes, from the start. I did not migrate from EA3 to EA4. When I installed cPanel for the first time, I installed the latest release (but not EDGE). I'm running 60.0.25 and that's what was installed the first time around.
    3. What do you mean you dont see them as choices for EA4?

    When I provision (or customize) the packages for my current profile, I'm provided with choices as to what I want installed. There's four steps to the provisioning system.
    * Apache MPM * Apache Modules * PHP Versions * PHP Extensions * Review
    During the Apache MPM phase, I pick which Multi-Processing Module I want Apache to use (ie, mod_mpm_prefork). If you look in the list I posted, you'll see how it shows mod_mpm_prefork as one of the installed packages. Notice how there isn't a step called Others. When I go through the Apache Modules section, I'm not provided with any options for config, config-runtime, or tools. I thought perhaps the patch was getting enabled in one of these hidden options that I don't have access to. I now think that perhaps that train of thought was wrong and that every EA4 person has those options
    4. Are you comparing logs from your old server and your new one?

    No. I do have access to the logs somewheres from the old server but I have no compared them. Which logs would I be comparing? The upcp update logs? I don't know where the EA4 logs are stored, but I don't think Apache gets recompiled when we change something with EA4. It's a lot different than EA3. With EA3, every time you changed something, the Apache source would get recompiled.
    If you are running 2.4/EA4 and pulling up 2.4/EA3 logs something is not right.

    I am running Apache 2.4 and EasyApache 4. I am not pulling up any EA3 logs. My cPanel configuration is very similar to the old server (same version of cPanel, etc). I haven't completely hardened it yet though. For example, I didn't change the SSL / TLS cipher suite to support perfect forward secrecy, I haven't tightened down Exim yet, etc. Just trying to figure out how to remove this patch. It is NOT listed under the Global Configuration Options. I might need to create a ticket here at cPanel and have the developers look into this for me.
    0
  • Spork Schivago
    Hey Spork Schivago, Yes, success! Now we both know it does come from cPanel. Next question is how to uninstall and protect against Symlinks. For that answer i would suggest searching cPanel Forums first, then open a thread: How to uninstall the EA4 Bluehost patch and secure afterwards? I have learned a lot from this, danielpmc

    According to that document Daniel, I just go to the Global Configuration Options in cPanel for Apache and turn On to Off for Symlink Protection. However, on my server, there is no Symlink Protection under the Global Configuration Options in cPanel for Apache. I've grepped through the entire /etc/apache2 directory for SymlinkProtect as well and didn't get any returns
    root@franklin:[/etc/apache2]# grep -iR SymlinkProtect * root@franklin:[/etc/apache2]#
    I'm wondering where else the SymlinkProtect option might be stored. Maybe someone manually set it. The documentation isn't exactly clear and what file I'd set this directive in. As for how to protect myself from this type of attack after removing the bluehost patch, I'm going to go with the kernel solution, if my kernel doesn't currently support it (which I believe it does). There's a kernel GRSecurity patch, but it seems my kernel has some sort of SymLink protection built in by default. I see in one of the sysctl.conf files:
    # Enable hard and soft link protection fs.protected_hardlinks = 1 fs.protected_symlinks = 1
    Thanks.
    0
  • Spork Schivago
    Back to the drawing board I guess. I took a peek at the EA4 Apache's template, the file that generates the /etc/apache2/conf/httpd.conf file from the various options set in cPanel. This template is:
    /var/cpanel/templates/apache2_4/ea4_main.default
    I see the code that enables the symlink protection:
    # These are hard-coded values that are required by cPanel & WHM PidFile [% paths.dir_run %]/httpd.pid User nobody Group nobody ExtendedStatus [% IF main.exists('extendedstatus') %][% main.extendedstatus.item.extendedstatus %][% ELSE %]Off[% END %] LogLevel [% IF main.exists('loglevel') %][% main.loglevel.item.loglevel %][% ELSE %]warn[% END %] [%- IF main.exists('symlink_protect') %] SymlinkProtect [% main.symlink_protect.item.symlink_protect %] SymlinkProtectRoot [% paths.dir_docroot %] [% END -%]
    I check my /etc/apache2/conf/httpd.conf file and see:
    # These are hard-coded values that are required by cPanel & WHM PidFile /etc/apache2/run/httpd.pid User nobody Group nobody ExtendedStatus On LogLevel warn
    As you can clearly see, there is no SymlinkProtect or SymlinkProtectRoot options set. I wonder if that sysctl.conf file has anything to do with it....To me though, I think something is very wrong. Why don't I have the option to turn Symlink Protection on and off in WHM like the documentation says to do? A clean install of cPanel....it should be there....the template expects it to be there.
    0
  • Spork Schivago
    I wish @cPanelMichael was here. He's real good with this stuff and really knows the ins and outs of the cPanel source code. I bet he'd know what was going on. He'd ask some questions and have me copy and paste some outputs, maybe have me run a command, and then he'd say could you please create a support ticket so I can take a closer look? And then in the support ticket section, he'd post updates with what was wrong and how he fixed it. I bet I'm gonna have to create a ticket.
    0
  • danielpmc
    Hello Spork Schivago,
    During the Apache MPM phase, I pick which Multi-Processing Module I want Apache to use (ie, mod_mpm_prefork). If you look in the list I posted, you'll see how it shows mod_mpm_prefork as one of the installed packages. Notice how there isn't a step called Others. When I go through the Apache Modules section, I'm not provided with any options for config, config-runtime, or tools.

    Your server is showing modules that you are unable to physically control. Why? And why is the option to disable Symlink not available to you? Something is a little odd. I agree this is probably a good time for you to ask for cPanel Technical assistance. danielpmc
    0
  • Spork Schivago
    Hello Spork Schivago, Your server is showing modules that you are unable to physically control. Why? And why is the option to disable Symlink not available to you? Something is a little odd. I agree this is probably a good time for you to ask for cPanel Technical assistance. danielpmc

    I think perhaps no one has access to some of the things that show up in EA4, like the Others section, for example. My original train of thought was the list was customizable, like many things in cPanel. For example, that template I was talking about, it has the .default extension. EA3 used similar templates but you could override them and use a custom template if you gave it a .local extension. EA3 would parse the .local template instead of the .default. With EA4, the developers were trying to move away from that style though and where encouraging users to use those pesky Pre Main Include, Pre Virtual Host Include, and Post Virtual Host Include files. I'm willing to bet that Others section is supposed to be there and includes stuff users aren't supposed to have control over. I'm willing to bet those things that look like modules in the Apache 2.4 section aren't actual modules and are cPanel specific stuff that we shouldn't have control over. EA4 has to be storing the profile somewhere on disk and I bet when I click the View all packages, when it reads the file in, it's just not hiding the stuff. The reason I say this is because when EA4 came out, I remember people asking if cPanel would implement the Bluehost patch into EA4, just like they did with EA3. Eventually, cPanel gave the users the link to the patch and told them that if they wanted to, they could download the source code to Apache, patch and install it manually. If there was a way to implement the patch into EA4, they sure as heck didn't make it clear. There's documentations on how to customize various things in cPanel from the shell. For example, I can override the installation of the version of Roundcube that comes with cPanel and have cPanel install any version I want, with my own custom configuration. They have documents telling you how to do this. It's not something the average user would be doing, I wouldn't think. But if they're willing to share that information with users, I couldn't see why they would try to hide the ability to custom configure EA4, you know?
    0
  • JacobPerkins
    Hi, We are adding in the SymlinkProtection patch to EA4. It's already been released in the RPMs, but we're adding in the UI support in v62. We're updating SecurityAdvisor to do the proper thing with the new patches. Also, the patch is controlled via an 'On/Off' configuration directive in httpd.conf:
    SymlinkProtect On SymlinkProtectRoot /var/www/html
    I hope this helps!
    0
  • Spork Schivago
    Hi, We are adding in the SymlinkProtection patch to EA4. It's already been released in the RPMs, but we're adding in the UI support in v62. We're updating SecurityAdvisor to do the proper thing with the new patches. Also, the patch is controlled via an 'On/Off' configuration directive in httpd.conf:
    SymlinkProtect On SymlinkProtectRoot /var/www/html
    I hope this helps!

    cPJacon, Thank you for the response. I now know what's going on. I'll post my information on the problem in case anyone else has the same question. Currently, at least with cPanel 60, Apache is in fact patched for the Symlink Race Condition, using the Bluehost patch, but the patch is not "active" by default. The way the Security Advisor checks for the patch is by searching the Apache binary, httpd, for the SPT_DOCROOT string. If it finds it, it reports the patch as being active. The patch gets activated by the top-level Apache directives, SymlinkProtect On, in the /etc/apache2/conf/httpd.conf configuration file. There's also the SymlinkProtectRoot /var/www/html directive. The SymlinkProtectRoot directive will be ignored if the SymlinkProtect is set to Off or is missing. I believe the proper way to check to see if the Bluehost Symlink Race Condition patch is active is to run the following commands:
    grep symlink_protect /var/cpanel/conf/apache/local grep Symlink /etc/apache2/conf/httpd.conf
    Melanie showed me how to properly check. If grep doesn't report anything, then the patch is inactive. Obviously, patching the Security Advisor to just grep the various files looking for symlink_protect and Symlink, respectively, isn't really going to work. If people have commented out the link symlink_protect, grep will still report it has finding it. I think what makes it especially confusing is the documentation found here: Symlink Race Condition Protection - EasyApache 4 - cPanel Documentation It doesn't mention that the Apache Global Configuration option is only available for cPanel 62. It implies so long as you're running EasyApache 4, you should see the option to turn it on or off. Maybe the documentation should be edited to clarify that it's only for 62? EA-5670 will address the false positive, if anyone else has noticed the same issues. Just thought everyone would like to know. Thanks for taking the time to answer my questions.
    0
  • cPanelMichael
    I think what makes it especially confusing is the documentation found here:
    0
  • RWH Tech
    Let me hop in here with a real retarded question: Why /var/www/html? Wouldn't it be /home/*/public_html?
    0
  • Spork Schivago
    Let me hop in here with a real retarded question: Why /var/www/html? Wouldn't it be /home/*/public_html?

    I don't think that's a bad question. I haven't studied the patch indepthly. It's a bit hard because I have the source code for the EA3 Bluehost patch (or what I believe is very close to the EA3 Bluehost patch). I'm not sure if cPanel modified the patch to work with EA4 in anyway. I haven't studied the Bluehost patch too much, I just glanced through it. I believe this is how it works: On a default cPanel installation, before any customer accounts (or websites) are setup, the default DocumentRoot in Apache is /var/www/html With the patch, I believe it will protect whatever the SymlinkProtectRoot directive is set to in Apache's httpd.conf, unless there's virtualhosts that set a custom DocumentRoot directory, in which case it will protect those directories. So, I believe if you don't have any websites setup, because Apache is serving files from the /var/www/html directory, the cPanel team decided to set the SymlinkprotectRoot directive to /var/www/html. If you have any virtualhost entries defined in Apache's httpd.conf file, I believe /home/*/public_html is protected, where * is the username for the virtualhost entry. What's a bit odd is although the patch is supposedly disabled on my cPanel server and I currently have no other forums of protection, I downloaded a POC (Proof of Concept) source code file to try and exploit this vulnerability in Apache and it seems to have failed. I'll try looking for another POC and see what I can find.
    0
  • RWH Tech
    I don't think that's a bad question. I haven't studied the patch indepthly. It's a bit hard because I have the source code for the EA3 Bluehost patch (or what I believe is What's a bit odd is although the patch is supposedly disabled on my cPanel server and I currently have no other forums of protection, I downloaded a POC (Proof of Concept) source code file to try and exploit this vulnerability in Apache and it seems to have failed. I'll try looking for another POC and see what I can find.

    That makes sense, with Symlinkprotectroot protecting any root, along with the "default" defined one. I did some research, mainly based on other people's analysis, since I'm not a coder, and decided to run the rack911 patch on my setup. I chose it for performance reasons, but that's kind of moot because there aren't that many sites on my server. hehe It's been a long time since I did my testing, but I'm quite sure I used this and the exploit worked (exploit 1). StalkR's Blog: Exec race condition exploitations I saw the bluehost patch go up in the easyapache git, so I knew it was coming down the pipe. This is a good thing for those with CentOS 7, those with VPSs that can't update kernels, and those with small setups who don't want to shell out extra monthly cash for CL.
    0
  • Spork Schivago
    That makes sense, with Symlinkprotectroot protecting any root, along with the "default" defined one. I did some research, mainly based on other people's analysis, since I'm not a coder, and decided to run the rack911 patch on my setup. I chose it for performance reasons, but that's kind of moot because there aren't that many sites on my server. hehe It's been a long time since I did my testing, but I'm quite sure I used this and the exploit worked (exploit 1). StalkR's Blog: Exec race condition exploitations I saw the bluehost patch go up in the easyapache git, so I knew it was coming down the pipe. This is a good thing for those with CentOS 7, those with VPSs that can't update kernels, and those with small setups who don't want to shell out extra monthly cash for CL.

    Haven't gotten a lot of sleep lately because our baby's sick. I don't remember the rack911 patch right now, I'll try to remember to research it a bit tomorrow maybe. Remember though, the Bluehost patch is better than nothing, but it does not fully protect users. One of the big disadvantages, if I remember correctly, it'll only work with Apache, not PHP, etc. Last I checked, there were only a few ways to fully protect against the symlink race condition for free. Some combination of mod_ruid2, Apache Jail Shell, and maybe something else (I can't remember if there was a third requirement) The GRSec kernel security patch. For me, I think I'm going to go for the GRSec kernel patch. My hosting provider's kernel has the /proc/config.gz file present, so, I think all I need to do is download the same kernel version (4.8.6-x86_64), copy config.gz to a directory, uncompress it, and then copy the config file to the kernel's source directory but rename it .config. That should provide me with the exact same configuration my hosting providers use for their kernel. Then I'll just apply the GRSec patch and maybe fine-tune the kernel, compile and install. I think the kernel patches are the best solution. I think maybe the CloudLinux is using a kernel that has the GRSec patch enabled. I've also read using the CageFS will provide support, but I believe that's only available if you're running CloudLinux. I'll check out you're link. That's to test for the symlink race condition and not just some executable race condition? Thanks!
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post