Memcached + Apache 2.2 crashes site every 5-10 minutes
Apache 2.2/2.4 (was running 2.4 but currently back to 2.2 to try to fix below issues)
Server Version: Apache/2.2.29 (Unix) mod_ssl/2.2.29 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 Apache mod_fcgid/2.3.9
PHP 5.4
Memcached 1.4.15, running locally on the same server.
I recently had some database slowness on my site related to queries on large table (120M rows) and noticed that phpMyAdmin -> Status -> Advisor said I should no longer use the mySQL Query cache. My site was 200 requests/second and the trigger was anything over 100/second.
I installed memcached per this guide I found Easy way to install Memcached and memcache in a cPanel server | CryBit - /http://crybit.com/easy-way-to-install-memcached-and-memcache-in-a-cpanel-server/
Everything was fine, it was installed, I could see it was live in phpinfo and via this command:
memcached-tool 127.0.0.1:11211 stats
After installation, I began switching my PHP code to use memcache.
Example:
[PHP]$memcache = new Memcache;
$memcache->connect('127.0.0.1', '11211');
$sql = "SELECT ids from id_table WHERE id_type=5";
$ids = $memcache->get(md5($sql));
if (!$ids)
{
$ids = get_ids_from_database($sql);
if (count($ids))
{
$memcache->set(md5($sql), $ids, MEMCACHE_COMPRESSED, 60);
}
}[/PHP]
I verified data was being loaded from the cache in my PHP page, and I also could see data being stored using the memcached-tool command noted above .
Soon after I began using memcached, Apache would lock up and go to 256 connections (my configured max through WHM) every 30 minutes or so. This time would vary, it was seemingly random. I could also see database queries massively backing up when viewed via:
mytop -d mysql
I thought it was database issues so I began caching more and more stuff in my code, converted some high update/select tables to INNODB, etc. This actually seemed to help as my site was faster but as I cached even more, my site became unusable as Apache would overload at shorter and shorter increments, down to only working from 5-10 minutes today. I finally decided to disable all memcaching just as a test and boom, site came back up everything is ok, barring the occasional high load/high # of Apache connections that I think is normal for my site, especially when I still have the mySQL cache off.
I would like to use memcached as I have a bunch of queries that are long-running, often taking 3-30 seconds, and occasionally my site will still run slow due to these queries.
What can I check and/or change to get memcached working reliably?
Edit #1:
After looking at a lot of PHP code samples, I see very few people doing this in examples:
[PHP]$ids = $memcache->get(md5($sql));[/PHP]
This example is more common, where some text is appended:
[PHP]$ids = $memcache->get('sql:' . md5($sql));[/PHP]
Is there an issue with just using md5 alone to generate a key?
Edit #2:
My site is a medium sized site - it has 800-1000 people online most of the time. I wonder if calling md5() hundreds/times/second is an issue? The major pages that I initially optimized had 5-10 memcached and md5 calls each.
Edit #3:
I took out some md5() calls as a test and just used a simple key like 'users' and my site still began overloading fairly quickly.
As another test, I also removed MEMCACHE_COMPRESSED and replaced it with a 0 to disable any compression in case that was a speed issue. The data I am caching is an array and it was NOT cached. Possible bug?
Edit #4:
Is mod_fcgid an issue? I mean overall but here are my settings from Pre VirtualHost Include, All Versions, in case there is an issue with the settings:
FcgidMaxProcesses 300
FcgidMaxProcessesPerClass 100
FcgidIOTimeout 300
FcgidMinProcessesPerClass 1
FcgidIdleTimeout 300
FcgidIdleScanInterval 120
FcgidBusyTimeout 300
FcgidBusyScanInterval 120
FcgidErrorScanInterval 10
FcgidZombieScanInterval 3
FcgidProcessLifeTime 3600
FcgidMaxRequestLen 1073741824
FcgidOutputBufferSize 0
-
For reference, in EasyApache I have these selected, among other things: MPM Prefork eAccelerator 0.9.6.1 Mod FastCGI v2.3.9 Mod Security 2.8.0 Just in case any of those are causing some conflict or should be altered to have memcached work. 0 -
Resolution to this issue: There was no issue with memcached. Originally my site was overloading because one of my tables had grown so large (120M rows) that the mySQL queries against it could not be processed in a timely fashion. Even though I had used memcached to cache a lot of my "busy" queries, Apache and my site were still overloading because the mySQL Query Cache was off. A combination of redesigning the large table design and my queries against it, turning the Query cache back on, some mySQL database parameter tuning, and using memcached for my high-volume queries has my site running smooth now with a low load. Notes: My server had auto-updated Memcached to version 1.4.21 while I was working on these issues but I don't believe this had any effect. I am also back on Apache 2.4. 0 -
I am happy to see the issue has been resolved. Thank you for updating us with the outcome. 0 -
Just out of interest, how did you install memcached on Apache 2.4, as the official guide states Memcached is not compatible with this version of Apache? 0 -
Just out of interest, how did you install memcached on Apache 2.4, as the official guide states Memcached is not compatible with this version of Apache?
Memcached site: memcached - a distributed memory object caching system Where does it says that? It's working fine for me since installing it above.0 -
The Apache module mod_cache is not the same as memcached. 0
Please sign in to leave a comment.
Comments
7 comments