Apache Processes
Hi,
I re-open this ticket because this issue has been tormenting me for quite some time.
My host tells me that they are normal apache processes, and everywhere I read that they are normal.
I have been looking at this issue very closely and in my case I could notice that:
- When upgrading from centos 5.x to 6.x, these processes stopped for a few hours, then started to return 1 by 1, and now I have a dozen always open
- if I kill them all and restart http service, then my load goes down for a while, up to a couple of hours, then returns.
- they seem to be responsible for high load, but they usually show 0% usage of memory and CPU (or is it Maldet that uses so much resources to keep up with what they are doing)
- from time to time I see one that is run by root, but most of the time they are run by nobody which is I think the normal way, so why the root one?
but more concretely this is what bothers me:
When I track these processes, some show this kind of result:
Going on a website that has no php script and look for .htaccess files... fine
going to another website that is not directly linked to that 1st one
In the process mentioned above, it seems to be looking for .htaccess files everywhere I have attached a truncated process tracking output, of course I don't understand a word in it. it just goes on and on and on for hours or days. In other such processes it's looking for files that don't exist, here index.xhtml and index.shtml... extensions I never use:
When I kill them, restart apache, they return instantly. So here are my questions: 1/ What are these processes actually doing apart from looking for .htaccess or vulnerable files everywhere? 2/ why doesn't anything on the system controls processes that last so long and work their way from account to account? I have CSF setup pretty tight but it doesn't seem to matter. 3/ is there a way to track what IP is actually using this process and block it? 4/ why are these processes increase my load but don't seem to consume cpu and ram? 5/ how can I setup cpanel to recognise these dodgy processes and kill them fast? 6/ Is there a way to track eventual script that is exploited or exploiting the system this way? (all my scripts are up to date, but maybe I am carrying a parasite for some time and I don't know about it. I have used freelancers in the past and it's not impossible that one of them left a small gizmo in a corner. ... That's a long post for a 1st post... but this issue has been increasingly bothering me for several years now and I'd really like to hear what other people think about it.
"lstat("/home/XXXXXX/public_html/culture_reggae_afro", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/home/XXXXXX/public_html/culture_reggae_afro/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
lstat("/home/XXXXXX/public_html/culture_reggae_afro/artists", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/home/XXXXXX/public_html/culture_reggae_afro/artists/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
lstat("/home/XXXXXX/public_html/culture_reggae_afro/artists/fj", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/home/XXXXXX/public_html/culture_reggae_afro/artists/fj/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
lstat("/home/XXXXXX/public_html/culture_reggae_afro/artists/fj/him.jpg", {st_mode=S_IFREG|0644, st_size=36495, ...}) = 0
"
Going on a website that has no php script and look for .htaccess files... fine
"/home/hotelcom/public_html/crans-montana_directory/submit_useful_link.html", {st_mode=S_IFREG|0644, st_size=14714, ...}) = 0
open("/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
open("/home/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
lstat("/home/YYYYYYY", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
open("/home/YYYYYYY/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
lstat("/home/YYYYYYY/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
open("/home/YYYYYYY/public_html/.htaccess", O_RDONLY|O_CLOEXEC) = 88
fstat(8"going to another website that is not directly linked to that 1st one
"stat("/home/ZZZZZZ/public_html/components", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/home/ZZZZZZZ/public_html/components/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
lstat("/home/ZZZZZZZ/public_html/components/com_jvotesystem", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/home/ZZZZZZZ/public_html/components/com_jvotesystem/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
lstat("/home/ZZZZZZZ/public_html/components/com_jvotesystem/ajax.php", {st_mode=S_IFREG|0644, st_size=40315, ...}) = 0
op"In the process mentioned above, it seems to be looking for .htaccess files everywhere I have attached a truncated process tracking output, of course I don't understand a word in it. it just goes on and on and on for hours or days. In other such processes it's looking for files that don't exist, here index.xhtml and index.shtml... extensions I never use:
"....
lstat("/home/UUUUUUUU/public_html/country_info/asia/east_asia/china/lhasa/index.shtml", 0x7fff6c9da440) = -1 ENOENT (No such file or directory)
access("/var/cpanel/bwlimited/domain.com", F_OK) = -1 ENOENT (No such file or directory)
stat("/home/UUUUUUUUo/public_html/country_info/asia/east_asia/china/lhasa/index.xhtml", 0x7fff6c9da440) = -1 ENOENT (No such file or directory)
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/UUUUUUUU", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/UUUUUUUUU/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/UUUUUUUU/public_html/country_info", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/UUUUUUUUU/public_html/country_info/asia", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/UUUUUUUU/public_html/country_info/asia/east_asia", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/UUUUUUUU/public_html/country_info/asia/east_asia/china", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/UUUUUUUU/public_html/country_info/asia/east_asia/china/lhasa", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/UUUUUUUUU/public_html/country_info/asia/east_asia/china/lhasa/index.xhtml
..."
When I kill them, restart apache, they return instantly. So here are my questions: 1/ What are these processes actually doing apart from looking for .htaccess or vulnerable files everywhere? 2/ why doesn't anything on the system controls processes that last so long and work their way from account to account? I have CSF setup pretty tight but it doesn't seem to matter. 3/ is there a way to track what IP is actually using this process and block it? 4/ why are these processes increase my load but don't seem to consume cpu and ram? 5/ how can I setup cpanel to recognise these dodgy processes and kill them fast? 6/ Is there a way to track eventual script that is exploited or exploiting the system this way? (all my scripts are up to date, but maybe I am carrying a parasite for some time and I don't know about it. I have used freelancers in the past and it's not impossible that one of them left a small gizmo in a corner. ... That's a long post for a 1st post... but this issue has been increasingly bothering me for several years now and I'd really like to hear what other people think about it.
-
Hello :) I have moved this post to our "Security" forum, as those questions are more appropriate here. Thank you. 0 -
A couple things here. For gathering data and seeing what IP(s) are in question, I'd have a look at the output of "httpd fullstatus" as well as "lsof -i :80". These should help you track things a little better and help you associate IP addresses to requests and PIDs. Also review the domlogs (domain access logs) for the domain(s) in question. It's normal for apache to look for .htaccess anywhere under the user account. If you open a file that's X subdirectories deep, Apache will likely look for any .htaccess files in X subdirectories. For example if I make a web request for /home/UUUUUUUUU/public_html/country_info/asia/east_asia/china/lhasa/index.xhtml, Apache is probably going to look for: /home/UUUUUUUUU/.htaccess /home/UUUUUUUUU/public_html/.htaccess /home/UUUUUUUUU/public_html/country_info/.htaccess (and so on). Also It is normal to have one httpd process as root and all the child processes as the Apache user ("nobody" on a cPanel system). You may also see /usr/local/cpanel/bin/splitlogs run as root. I hope this sheds a little light. 0
Please sign in to leave a comment.
Comments
2 comments