UPS-421 - Apache 2.4.52 issues?
Anybody seeing a lot of Apache dying with the latest EA4 Apache?
Upgraded to ea-apache24-2.4.52-1.1.3.cpanel.x86_64 earlier today across all of our servers, but I'm getting a lot of servers with the web server not responding.
Not sure if it's related or not - but seems to be a bit more than just a coincidence. This seems to be affecting a lot of our servers and all of our servers were upgraded to ea-apache24-2.4.52-1.1.3.cpanel.x86_64 earlier today (December 22).
Still we have some servers that were upgraded to ea-apache24-2.4.52-1.1.3.cpanel.x86_64 and they don't appear to be having this issue.
I've downgraded one server back to Apache 2.4.51 to see if the issue persists.
Just wondering if anyone else is seeing any similar behaviors after updating to ea-apache24-2.4.52-1.1.3.cpanel.x86_64
-
What MPM are you using? 0 -
Using the event MPM. Still trying to work out exactly what might be happening, because I know it's hard to diagnose anything without any evidence to pick through. Best I can tell from the Apache error_log with such events, the error log will report something like: [font="courier new">AH10291: h2_workers: cleanup, 5 idle workers did not exit after 5 seconds. (numbers may be different) Followed by, something like: [font="courier new">[plain]body.xml:2: parser error : XML declaration allowed only at the start of the document ^ body.xml:2: parser error : XML declaration allowed only at the start of the document ^ body.xml:2: parser error : Input is not proper UTF-8, indicate encoding ! Bytes: 0x8E 0x45 0x52 0x54 XXXXXA[/plain] And then Apache just stops handling new connections. It just sits there until it is restarted. I'm noticing similar such entries in the Apache error_log for servers that are running Apache 2.4.51, but Apache seems to continue trucking afterwards. I have no clue what that in the error_log is actually referring to, or if it has anything at all to do with the issue, but that seems to be where Apache 2.4.52 stops after that is reported in the error_log. Just wondering if anybody else is seeing this issue? I figure more people may pop in today with the issue, since ea-apache24-2.4.52-1.1.3.cpanel.x86_64 was release during mid-day yesterday (December 22) and most folks probably have auto-updates set to run during the overnight hours. 0 -
Hello! I am not aware of this being a known issue since the recent EA4 update. Would you be able to open a ticket using the link in my signature and provide me with the incident ID? If you cannot, your web hosting provider should be able to open one on your behalf. 0 -
I've actually run into this issue on an *ahem* other control panel that is also using Apache 2.4.52. So I'm almost convinced that something within the code change from Apache 2.4.51 to Apache 2.4.52 is causing this. Just really, really puzzled that nobody else is experiencing this issue. 0 -
[QUOTE] And then Apache just stops handling new connections. It just sits there until it is restarted.
Same here. Since yesterday it has been necessary to restart Apache several times on several servers when it is not usual to do so. When it stops accepting connections I don't see anything at /usr/local/apache/logs/error_log I use event MPM. Directdmin users are also reporting the issue:0 -
Been getting alerts from my servers all day today, about 8 times in one day when i usually have one a month. SOMETHING is up. And like @voip said, nothing in the logs at all. 0 -
Same issue here too. seeing a lot of alerts regarding apache being unresponsive ( no error in the log). Then checkservd kicks in and restart apache and it goes fine until this is repeated after a few hours 0 -
apache has a log level to know more : Home "Service Configuration "Apache Configuration "Global Configuration, and see LogLevel, crit or error may suit 0 -
You can downgrade Apache back down to Apache 2.4.51 - but you'll also need to be sure that you do not have automatic updates enabled, otherwise your server's just going to upgrade to Apache 2.4.52 on the next automatic package update run. First you need to determine what RPM version of Apache you have installed: [font="courier new">rpm -qa ea-apache24 It would appear that cPanel released two versions of Apache 2.4.52: [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64 ea-apache24-2.4.52-1.1.3.cpanel.x86_64 If you are using Apache 2.4.52, then [font="courier new">rpm -qa ea-apache24 will report one of these two versions. I would suspect that most everyone using Apache 2.4.52 on cPanel is using [font="courier new">ea-apache24-2.4.52-1.1.3.cpanel.x86_64. I think [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64 was released and then almost immediately overwritten by [font="courier new">ea-apache24-2.4.52-1.1.3.cpanel.x86_64. So the vast majority of folks went from [font="courier new">ea-apache24-2.4.51-2.4.1.cpanel.x86_64 straight to [font="courier new">ea-apache24-2.4.52-1.1.3.cpanel.x86_64. But it's technically possible someone got [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64 and never updated to [font="courier new">ea-apache24-2.4.52-1.1.3.cpanel.x86_64. The version you want to get down to is: [font="courier new">ea-apache24-2.4.51-2.4.1.cpanel.x86_64 Which would appear to be the latest version of Apache 2.4.51 that cPanel released I'm going to assume that you are running [font="courier new">ea-apache24-2.4.52-1.1.3.cpanel.x86_64 - and there may be a more eloquent way of doing this - I'm actually going to downgrade twice. The first downgrade will get me to [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64 and the second downgrade will get me to [font="courier new">ea-apache24-2.4.51-2.4.1.cpanel.x86_64 which is what I want. Probably is a more eloquent way of doing this, but by the time I could find it I could have already run the downgrade twice. If by chance you are using [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64 then you will only need to downgrade once. To insure better results, simply do a [font="courier new">rpm -qa ea-apache24 after each downgrade until you see [font="courier new">ea-apache24-2.4.51-2.4.1.cpanel.x86_64. Make a note of what version of Apache you have installed from [font="courier new">rpm -qa ea-apache24. Remember there is a [font="courier new">ea-apache24-2.4.52-1.1.3.cpanel.x86_64 and a [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64. Again, I'm going to assume that you are using [font="courier new">ea-apache24-2.4.52-1.1.3.cpanel.x86_64 but if you are using [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64 then note the difference in versions. To get a list of the packages affected, you will need to grep through your yum.log. Now... probably your yum logs have already been rotated, so what you're after is likely in /var/log/yum.log-20211223 - but for simplicity sake I'm just going to look through all of the yum logs: [font="courier new">cat /var/log/yum.log* | grep 'ea-apache24.*2.4.52-1.1.3.cpanel' | awk '{print $5}' | sed s/-2.4.52-1.1.3.cpanel.*//g | sort | uniq (again - note the RPM version used here) This is a list of all of the packages you will want to downgrade. The [font="courier new">| sort | uniq is there just in the off chance you have duplicate packages listed. You can either copy and paste this list into a package list passed to [font="courier new">yum downgrade or you can pass the output of the above directly into the package list to [font="courier new">yum downgrade: [font="courier new">yum downgrade $(cat /var/log/yum.log* | grep 'ea-apache24.*2.4.52-1.1.3.cpanel' | awk '{print $5}' | sed s/-2.4.52-1.1.3.cpanel.*//g | sort | uniq) If you'll note in this particular case yum will tell you that you are about to downgrade all of these to [font="courier new">2.4.52-1.1.1. Proceed with this downgrade. See what version of Apache you are using: [font="courier new">rpm -qa ea-apache24 This should report: [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64 If it happens to report: [font="courier new">ea-apache24-2.4.51-2.4.1.cpanel.x86_64 then you are done. If it reports [font="courier new">ea-apache24-2.4.52-1.1.1.cpanel.x86_64 then do another downgrade: [font="courier new">yum downgrade $(cat /var/log/yum.log* | grep 'ea-apache24.*2.4.52-1.1.3.cpanel' | awk '{print $5}' | sed s/-2.4.52-1.1.3.cpanel.*//g | sort | uniq) This time yum should report that it's going to downograde to [font="courier new">2.4.51-2.4.1. Proceed with this downgrade. Now when you run: [font="courier new">rpm -qa ea-apache24 This should report: [font="courier new">ea-apache24-2.4.51-2.4.1.cpanel.x86_64 And that's it. You're now using Apache 2.4.51, which does not have this not responding bug. However, ultimately, the folks over at Apache will need to fix this issue in Apache 2.4.52 or otherwise you won't get any of the benefits of Apache versions after 2.4.52. 0 -
A shorter version of the downgrade command is [CODE=bash]yum downgrade ea-apache24* liblsapi liblsapi-devel
Run the command again till you see the rpm version as ea-apache24-2.4.51 in the command [CODE=bash]rpm -qa ea-apache240 -
I also the issue on a few servers with event MPM And i don't see with other MPM 0 -
We also started getting "httpd appears to be down" notifications starting Christmas Eve. Here's our Apache server info: - Server Version: Apache/2.4.52 (cPanel) OpenSSL/1.1.1m mod_bwlimited/1.4 mod_fcgid/2.3.9
- Server MPM: event
- Server Built: Dec 22 2021 02:53:54
0 -
I have the same error, it only happens in event MPM, there is nothing in the logs 0 -
same issue here. at random servers stop to accept connections, nothing in error logs, about 10min later they accept connections again. Started the day after 2.4.52 was installed with random servers. (7 & 8 based) with mod_mpm_event enabled. 0 -
We have same problem on some servers (servers are use event mpm, but not all cPanel servers have this problem) 0 -
same problem here with deb.sury.org ondrej apache2 2.4.52 and mpm_event, but is there any other solution than downgrading apache? 0 -
I would like to add that I'm having the same issues too, this started on Christmas eve and has been every day since keeping the sites down for 15 minutes a day 0 -
Patch made the changelog, be on the look out =) 0 -
@bellwood Your diligence in keeping everyone updated is greatly appreciated! 0 -
Live in the repos: [23:24:21 root@foo:~] yum list ea-apache24.\* Installed Packages ea-apache24.x86_64 2.4.52-1.1.3.cpanel @EA4 Available Packages ea-apache24.x86_64 2.4.52-2.2.1.cpanel EA4
Update EasyApacheUpdate via WHM or CLI:[23:26:43 root@foo:~] httpd -v Server version: Apache/2.4.52 (cPanel) Server built: Dec 30 2021 14:32:42
No patch level indication on the version string, but, I'm sure a minor will be forthcoming. Thanks cPanel for getting on this.0 -
The Apache version won't change until The Apache Software Foundation officially adds the patch into the next Apache release. Presumably, cPanel just took the patch mention in the 65769 ASF bug report and applied it to their own in house Apache RPMs. I think it might be a good idea if cPanel would notarize their EasyApache changelogs - 0
Please sign in to leave a comment.
Comments
52 comments