Skip to main content

Comments

36 comments

Date Votes
  • Nirjonadda
    Tody i Updated Apache 2.4.16 to 2.4.17 then my forum home page are not loading, just say it page not found. Not working: nirmoladda.com Working: nirmoladda.com/forum.php
    0
  • sneader
    I recently updated one of our cPanel servers to update Apache from 2.4.16 to 2.4.17. After this, a customer using the Concrete5 CMS noticed that their .htaccess rewrite URLs (Pretty URLs as Concrete5 calls them) were not working. I found this thread on the Concrete5 Forums and found that my customer was not alone with this problem. Further, the thread points to a bugfix regarding REDIRECT_URL which was backported to 2.4.17: Bug 57785 " REDIRECT_URL is not suitable for use in server-generated pages In that bug report, this comment raised my eyebrows! This patch appears to break the prexisting PHP behaviour (all versions, verified as far back as PHP 5.2). When cgi.fix_pathinfo=1, PHP will under some conditions set SCRIPT_NAME directly from REDIRECT_URL (noticed running under FastCGI). I'm not going to argue correct behaviour, only point out that the existing behaviour that was relied by another project has been broken by this change.
    At this point, I have a ticket opened with cPanel to see if there is a way to downgrade Apache to 2.4.16, until this mess can be sorted out. In addition, I will not attempt to upgrade any other cPanel server with 2.4.17, until more is know about the effects of this backported patch. - Scott
    0
  • mtindor
    Wow. What a nightmare. Thanks for the warning Scott. That's a big issue considering I don't know of any easy way of reverting back to 2.4.16. Mike
    0
  • cPRex Jurassic Moderator
    Hey Scott, Thanks for opening up that ticket with us and bringing this to our attention. You are correct that there currently is not a way to downgrade minor versions of EasyApache as the only current versions available are 2.2.31 and 2.4.17. We also confirmed on your system that changing cgi.fix_pathinfo=1 to cgi.fix_pathinfo=0 in the global php.ini file and restarting the Apache service did not resolve this problem. I've opened case CPANEL-2058 with our developers and while I don't have an estimated time of when that will be resolved you can always check our changelogs at 11.52 Change Log - Documentation - cPanel Documentation for more details. ~Rex
    0
  • Trane Francks
    This problem and the mod_autoindex fiasco that breaks Phusion Passenger support for Rails apps really points out the importance of being able to downgrade Apache. Having a server upgrade break sites and then having absolutely no way of fixing them short of a full-server restore (downtime and data loss) is an extremely uncomfortable position in which to be. The same issue applies for MySQL upgrades. Being unable to rollback a borked upgrade is a recipe for disaster. IMO. Edited to add: A possible solution for the EasyApache issue might be to list n-2 versions on the initial EasyApache page. Instead of automagically always grabbing the latest version, the script could list the latest and a couple of earlier versions from which to execute, e.g.: * 3.32.3 * 3.32.2 * 3.32.1 This would be a feature that would make me 'love you long time'.
    0
  • sneader
    After updating one of our servers with Apache 2.4.17, a customer's CMS script (Concrete5) stopped processing URL rewrites. This is traced back to a change in REWRITE_URL which was backported to 2.4.17: Bug 57785 " REDIRECT_URL is not suitable for use in server-generated pages I have a separate forum post about this. - Scott
    0
  • Kent Brockman
    Hello guys. I can confirm this issue is breaking login/register pages for Magento sites and also breaking redirects for cache plugins in WordPress. Thus, I have almost 45 unusable websites right now. PLEASE I NEED A BUGFIX ASAP.
    0
  • Dhaupin
    Hmm i agree with this, why cant we rollback apache version in EA(3/4)? I was modernizing some htaccess for some stores, setting up to roll in a different MPM, and stumbled randomly upon this thread. We are still on 2.4.16 so if i would have rolled the new MPM then EA(3) would have forced 2.4.17 upon our serv, possibly resulting in many many thousands of broken rewrites....or worse, as Kent Brockman is saying.
    0
  • LBJ
    Hello guys. I can confirm... also breaking redirects for cache plugins in WordPress. Thus, I have almost 45 unusable websites right now. PLEASE I NEED A BUGFIX ASAP.

    G'day Kent, Are you running FastCGI? We encountered the known issue with Concrete5 sites on all Apache 2.4.17 servers with PHP 5.5.30, and applied the fix to config/site.php which immediately brought them back to life, but we've not seen any issues with WordPress. We run suPHP with cgi.fix_pathinfo set to the default of 1. We have many hundreds of WordPress sites, but no Magento or Rails installations. Until I read your post, I had assumed the impact was only moderate, which is what our support tickets have also suggested. Best regards, LBJ
    0
  • chposter
    Hello I can confirm all issues - Phusion passenger fails : mod_autoindex revert patch needed - Fastcgi broken in the reported way. I have not tested this but. I can not understand how a minor apache update can break so many things. How is posible that you update 16->17 and potencially all apps from your server are dead, apache? Cpanel guys, you must allow 2.4.16 in my opinion until this BIG problem is sorted out.
    0
  • JacobPerkins
    Hi, We are aware of these problems and we're monitoring these issues currently. There's a few options present that we're looking at: 1> Revert to 2.4.16. This will cause a lot of issues as we've also upgraded APR to 1.5.2, so those will both require downgrades. Also, there was a CVE assigned with the 1.5.2 update, so rolling back would bring back some vulnerabilities. 2> Attempt to custom patch and fix these issues ourselves. This would take a while to complete. 3> Wait for Apache developers to submit fixes for these problems, and then release those patches ourselves in a release. We're not really able to offer multiple versions of EA, as building in this support would take longer than just fixing the issues themselves. Thanks for these reports! Please, keep them coming as well. We know about Concrete 5 issues with REDIRECT_URL coming from PHPs SCRIPT_NAME functions, and we're also aware of the Phusion Passenger issues with mod_autoindex. We haven't come across or heard of any issues with WordPress or the like, so if you notice issues there, please ensure there's an upstream Apache bug present so they can work on fixing those issues. Thanks everyone for the information!
    0
  • sparek-3
    Can I throw out another idea? How about letting users compile Apache themselves? Set up a page where cPanel details and lists the patches they apply to Apache. Then server administrators can download Apache from apache.org, download the patches, apply the patches, run ./configure, run make, run make install and then they have whatever version of Apache they want to have installed. This would be more straight-forward than always having to go through EasyApache. Granted, it may only be for the more experienced server administrators, but it would be an option.
    0
  • sneader
    There is one other idea that nobody has mentioned. I can't take credit for it, though. I had the pleasure of having dinner with several folks from cPanel last night, and one of the attendees was none other than Mr. cPanel himself, Nick Koston. He said "You could rerun EasyApache and choose Apache 2.2." -- well, there you go!! Sure enough, this REDIRECT_URL change was not backported to 2.2! While Apache 2.2 isn't ideal, it should fix the problem at hand, without introducing any new issues (other than edge cases). Then... when Apache (hopefully) fixes this issue, and a 2.4.18 comes out, you just rerun EA, and choose 2.4.18 and off you go! - Scott
    0
  • JacobPerkins
    Absolutely, one way to get rid of these issues would be to downgrade to Apache 2.2. In regards to custom compiles of Apache, you can accomplish this on your own, by adding a pre-script for EA that removes those patches and doesn't build them in. Note that this is custom and that not including those patches may cause instability in other areas.
    0
  • Trane Francks
    For folks awaiting a Rails solution, Phusion will release a fix for the problem in Passenger 5.0.22.
    0
  • sneader
    I see that Apache is currently asking for feedback on what to do about the REDIRECT_URL change. Specifically, they are asking if they should...
      ]
    • Leave it to PHP (to fix)?
    • Introduce yet another env var?
    • Introduce a flag to switch between the two?
    • Other (please specify?)
    See: Bug 57785 " REDIRECT_URL is not suitable for use in server-generated pages (requires creation of account to add a comment)
    0
  • Kent Brockman
    Hello people, I continue finding problems with different kind of sites. Is there any update or fix to this issue? How can I roll back to Apache 2.4.16??? It used to work perfectly!
    0
  • JacobPerkins
    Hi, We're looking at the possibility of rolling Apache back to 2.4.16 and leaving APR at 1.5.2. We're testing this out now, and we hope to have this EA build released by Monday / Tuesday at the latest. We'll keep this thread updated as we know more. Thanks!
    0
  • Kent Brockman
    Great, thanks! (Can't wait) :)
    0
  • sparek-3
    Just to be clear, I don't recommend performing any of this on a production server. The purpose of this post is to detail that it can be done, not that it should be done. You might be able to roll back to Apache 2.4.16 by reading through the logs of the last successful EasyApache 2.4.16 compile:
    grep -l '/usr/local/apache/bin/apxs.*version 2.4.16' /usr/local/cpanel/logs/easy/apache/*
    The main hiccup is going to be finding a cPanel package of Apache 2.4.16. If you still have a server that is using 2.4.16 and hasn't been updated to 2.4.17 yet, then the file should be located at /var/cpanel/perl/easy/Cpanel/Easy/Apache/2_4.pm.tar.gz you may want to make a copy of that file for safe keeping. Extract that file.
    tar -zxf /my/path/to/2_4.pm.tar.gz
    change into the httpd-2.4 directory
    cd httpd-2.4
    This is where you'll now have to apply the patches as they are detailed in the log file:
    patch -p1 < ../cppatch/symlink-protection.patch patch -p1 < ../cppatch/2.4.2_cpanel_ssl_engine_rand.patch ...
    Then run the configure line, which again is in that log file. Join all of the lines into a single line. Then run the configure line.
    LD_LIBRARY_PATH=/opt/pcre/lib ./configure --disable-v4-mapped --enable-access-compat=static --enable-actions=static --enable-alias=static ...
    Then make and make install
    make make install
    Your mileage may vary. I have not upgraded my production servers to 2.4.17 yet. I did upgrade a test server to 2.4.17 and then downgraded it to 2.4.16 using this procedure. It seems to work, but it's not a production server. EDIT: And if you don't have access to an Apache 2.4.16 archive, you can create one:
    wget http://archive.apache.org/dist/httpd/httpd-2.4.16.tar.gz wget http://archive.apache.org/dist/apr/apr-1.5.1.tar.gz wget http://mirror.nexcess.net/apache//apr/apr-util-1.5.4.tar.gz tar --no-same-owner -zxf httpd-2.4.16.tar.gz tar --no-same-owner -zxf apr-1.5.1.tar.gz tar --no-same-owner -zxf apr-util-1.5.4.tar.gz tar --no-same-owner -zxf /var/cpanel/perl/easy/Cpanel/Easy/Apache/2_4.pm.tar.gz cppatch mv apr-1.5.1 httpd-2.4.16/srclib/apr mv apr-util-1.5.4 httpd-2.4.16/srclib/apr-util cd httpd-2.4.16
    Luckily the cppatch's didn't change from Apache 2.4.16 to Apache 2.4.17, so you can use the same cppatch directory from the Apache 2.4.17 2_4.pm.tar.gz file. I'm also using apr version 1.5.1, because that is what was included with cPanel's Apache 2.4.16. You might be able to use version 1.5.2, I don't know. At this point, after cd httpd-2.4.16, you are now ready to apply the patches as stated in the easyapache 2.4.16 install log and then ./configure ..., make, make install.
    0
  • JacobPerkins
    Sparek-3: Thanks for that info! Your steps do seem as if they will allow EA3 to build the old version of Apache. Granted, this is very advanced and will be overwritten once the new EA build comes along, but it's a great stop-gap to get back to Apache 2.4.16 right now. Thanks for sharing that!
    0
  • JacobPerkins
    Hi everyone, As a heads up, we'll be releasing EasyApache 3.32.4 tomorrow. This reverts Apache back down to 2.4.16, but leaves APR at 1.5.2. We've done a lot of testing and found no issues with downgrading in this manner. Once we get a patch from upstream Apache with fixes for mod_rewrite and mod_autoindex, we'll retest and send out 2.4.17 again. I'd recommend running EA tomorrow afternoon to get downgraded to 2.4.16 if you're having issues with REDIRECT_URL. Thanks again for your patience!
    0
  • Trane Francks
    Just to clarify, there won't be a mod_autoindex patch coming from the Apache crowd to fix Rails applications. Phusion Passenger will have a fix for this issue available in Passenger 5.0.22.
    0
  • Kent Brockman
    Hi everyone, As a heads up, we'll be releasing EasyApache 3.32.4 tomorrow. This reverts Apache back down to 2.4.16, but leaves APR at 1.5.2. We've done a lot of testing and found no issues with downgrading in this manner. Once we get a patch from upstream Apache with fixes for mod_rewrite and mod_autoindex, we'll retest and send out 2.4.17 again. I'd recommend running EA tomorrow afternoon to get downgraded to 2.4.16 if you're having issues with REDIRECT_URL. Thanks again for your patience!

    Hello Jacob! I just recompiled several of our servers using apache 2.4.16 and all of the problems have been automagically solved. I'm so happy now. :) Thanks for the downgrade. The problem with future versions is to recover the confidence to upgrade to those upcoming versions. The issue here was simple: cPanel should offer the last and the previous version of Apache to choose: 2.2.x 2.4.16 2.4.17 So that you can choose versions and rapidly go back to a previous working configuration. The last 10 days I've just replied to emails from angry customers asking when would be this patch ready. If cPanel could allow to choose more versions and not only the last one, the failure would last just 10 minutes, but it lasted 10 days instead. Is there anything you (cPanel staff) can do with this -ASAP- or should we open a feature request and wait until 11.58 to see it implemented? It's not that these fails happen often; in fact it's the first time since 2006 that I experience something like this but, will it be the last one? If we can opt to install a previous Apache version, the problem could be easily -and rapidly- mitigated. And... the same could be applied to PHP versions. You never know when a buggy version could appear. Hope to hear something promising about this. Very Thanks!
    0
  • Kent Brockman
    Oh and by the way, if you guys (cPanel guys) need to test 2.4.17 or others, I can help. I have good and long experience with WordPress and if it all works good with it, surely other CMS will do too.
    0
  • Trane Francks
    I can confirm that EA 3.32.4 rolling back to Apache 2.4.16 fixes the Phusion Passenger issue regarding Rails applications with the current Passenger 5.0.21 release. I completely agree with Kent that a rollback strategy is a must. Whether it be having n-1 Apache and PHP choices within the newest EA or some other mechanism to be able to call a specific EA version, I don't really care. Something short of a full server restore must be available on short notice.
    0
  • chposter
    Hello This problem looks like a little issue. However it is not: - Semantic Versioning 2.0.0 Apache follows it. Apache has followed it for years. Until this one. This relelase potentially breaks all of the apps of your server. Those changes should be in a mayor or minor release and not in a patch release. This is why CPanel downgraded Easy Apache version . The support rage would have been in the next year... at least epic. All of us who have updated, with the confidence of always, are very surprised of these httpd changes. If i have an only app, ok. If i have thousands, not ok at all. - Apache 2.2 "You can be in the 2.2.x apache to avoid the issue" . This is not always an election. APache 2.4.x was born with a set of capabilities and we are using it due to those . We have 'chosen' it. If all of my boxes are on 2.4.16. Should i downgrade to 2.2.x ? Nonsense. - CPanel and packagers This is a problem for packagers, for example CPanel and Easyapache 4 (among others). If apache starts to make these changes and Cpanel puts a new rpm (because it is a patch release!!), the pain will be big if automatic rpm upgrades are activated.
    0
  • JacobPerkins
    Hi, We can't control what Apache puts out, we only send that along to our customers. We try to not make much changes to the base Apache stack than we have to, to reduce the amount of possible breakage from changes and things like that. Unfortunately if Apache changes something upstream, it'll come down to customers. There's not much we can do about that. As for looking at 'multiple versions' of Apache and other items on the mirrors, this would require a major change and overhaul of our deployment system with EA, and as EA3 be gone in a year or so, that's not something I'd prefer to do. I was thinking about this last night quite a bit. I believe that if we can put 2 versions of EasyApache on the mirrors (the current and the previous builds), this would allow the ability to 'rollback' to a previous EA version, thus allowing the ability to 'rollback' changes from upstream. This again is a big change, but not as major as providing multiple versions of Apache inside EA3. This change would also expand the 'rollback' ability to not only Apache, but PHP, dependencies, etc. We're still discussing this here at cPanel, and any changes we make would be out in the future, as we're still working heavily on EA4. (We're trying to do as little as possible at the moment with EA3). Thanks everyone for your feedback! I'd love to keep up discussions on this if you have any further comments / concerns. Thanks again!
    0
  • Kent Brockman
    I was thinking about this last night quite a bit. I believe that if we can put 2 versions of EasyApache on the mirrors (the current and the previous builds), this would allow the ability to 'rollback' to a previous EA version, thus allowing the ability to 'rollback' changes from upstream. This again is a big change, but not as major as providing multiple versions of Apache inside EA3. This change would also expand the 'rollback' ability to not only Apache, but PHP, dependencies, etc.

    And I think that this should be applied to EA4 too. Because EA4 is the future de-facto deploy tool and these kind of things could happen sometime again. Maybe not. Hopefully not. But you should include this issue in any preemptive thinking on how EA4 may (and should) work. I can assure you none of the affected companies would like to go through this issue again. This problem must server to learn and improve cPanel :) Thanks
    0
  • JacobPerkins
    And I think that this should be applied to EA4 too. Because EA4 is the future de-facto deploy tool and these kind of things could happen sometime again. Maybe not. Hopefully not. But you should include this issue in any preemptive thinking on how EA4 may (and should) work. I can assure you none of the affected companies would like to go through this issue again. This problem must server to learn and improve cPanel :) Thanks

    Hi Kent, With EA4, we're trying to leave the latest few updates on the mirrors, and you should be able to 'pin' / yum versionlock these versions.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post