Is this xmlrpc brute force amplification attack?
Recently, I noticed that the "nobody" process was using 100% CPU for a long time on many cpanel servers. After lsof the process, I see the server was responding to the request from IP 50.93.203.147(and many other different IPs, most were VPS or Dedicated server). From the Apache access log, I see this IP issued only a few xmlrpc requests, but the request seems be different from before, as the POST size is larger and it always starts with "POST / HTTP/1.1" 2 or 3 mins before the "POST /xmlrpc.php HTTP/1.1". I guess the attacker passed a bunch of username and password into the 1st POST, then use xmlrpc to do brute force in 2nd POST continuously in one connection. Because they only do 1 or 2 connections at the same time, it bypass the old mod_sec rule for xmlrpc attack. Is there a way to stop this? Thank you.
Brute Force Amplification Attacks Against WordPress XMLRPC - Sucuri Blog
# lsof -p 182704
Here's the Apache log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd 182704 nobody cwd DIR 253,0 4096 2 /
httpd 182704 nobody rtd DIR 253,0 4096 2 /
httpd 182704 nobody txt REG 253,0 1809242 24909121 /usr/local/apache/bin/httpd
httpd 182704 nobody mem REG 253,0 27424 72614015 /lib64/libnss_dns-2.12.so
httpd 182704 nobody mem REG 253,0 65928 72614229 /lib64/libnss_files-2.12.so
httpd 182704 nobody mem REG 253,0 206640 72613996 /lib64/libidn.so.11.6.1
httpd 182704 nobody mem REG 253,0 439965 45876797 /opt/curlssl/lib/libcurl.so.4.3.0
httpd 182704 nobody mem REG 253,0 2457613 24910725 /usr/local/apache/modules/mod_security2.so
httpd 182704 nobody mem REG 253,0 4319965 45876366 /opt/xml2/lib/libxml2.so.2.9.2
httpd 182704 nobody mem REG 253,0 90880 72613965 /lib64/libgcc_s-4.4.7-20120601.so.1
httpd 182704 nobody mem REG 253,0 596272 72614059 /lib64/libm-2.12.so
httpd 182704 nobody mem REG 253,0 987096 24908802 /usr/lib64/libstdc++.so.6.0.13
httpd 182704 nobody mem REG 253,0 64956 24910723 /usr/local/apache/modules/mod_suphp.so
httpd 182704 nobody mem REG 253,0 33138 24910724 /usr/local/apache/modules/mod_bw.so
httpd 182704 nobody mem REG 253,0 9610 24910720 /usr/local/apache/modules/mod_bwlimited.so
httpd 182704 nobody mem REG 253,0 79008 24905557 /usr/lib64/liblve.so.0.9.0
httpd 182704 nobody mem REG 253,0 46048 24910722 /usr/local/apache/modules/mod_hostinglimits.so
httpd 182704 nobody mem REG 253,0 122040 72614113 /lib64/libselinux.so.1
httpd 182704 nobody mem REG 253,0 110960 72614235 /lib64/libresolv-2.12.so
httpd 182704 nobody mem REG 253,0 10192 72614091 /lib64/libkeyutils.so.1.3
httpd 182704 nobody mem REG 253,0 43728 72614033 /lib64/libkrb5support.so.0.1
httpd 182704 nobody mem REG 253,0 10312 72614067 /lib64/libfreebl3.so
httpd 182704 nobody mem REG 253,0 19536 72614048 /lib64/libdl-2.12.so
httpd 182704 nobody mem REG 253,0 174840 72613927 /lib64/libk5crypto.so.3.1
httpd 182704 nobody mem REG 253,0 14664 72614244 /lib64/libcom_err.so.2.1
httpd 182704 nobody mem REG 253,0 946048 72613932 /lib64/libkrb5.so.3.3
httpd 182704 nobody mem REG 253,0 277704 72613897 /lib64/libgssapi_krb5.so.2.2
httpd 182704 nobody mem REG 253,0 1920936 72613905 /lib64/libc-2.12.so
httpd 182704 nobody mem REG 253,0 142640 72614050 /lib64/libpthread-2.12.so
httpd 182704 nobody mem REG 253,0 40400 72613974 /lib64/libcrypt-2.12.so
httpd 182704 nobody mem REG 253,0 43880 72614238 /lib64/librt-2.12.so
httpd 182704 nobody mem REG 253,0 294128 24909036 /usr/local/apache/lib/libapr-1.so.0.5.1
httpd 182704 nobody mem REG 253,0 165264 72613984 /lib64/libexpat.so.1.5.2
httpd 182704 nobody mem REG 253,0 232253 24909093 /usr/local/apache/lib/libaprutil-1.so.0.5.4
httpd 182704 nobody mem REG 253,0 88600 72613952 /lib64/libz.so.1.2.3
httpd 182704 nobody mem REG 253,0 690887 45876294 /opt/pcre/lib/libpcre.so.1.2.4
httpd 182704 nobody mem REG 253,0 1963296 24905926 /usr/lib64/libcrypto.so.1.0.1e
httpd 182704 nobody mem REG 253,0 441256 24908718 /usr/lib64/libssl.so.1.0.1e
httpd 182704 nobody mem REG 253,0 154664 72613899 /lib64/ld-2.12.so
httpd 182704 nobody mem REG 0,4 495107656 (deleted)/dev/zero (stat: No such file or directory)
httpd 182704 nobody mem REG 0,4 496619447 (deleted)/dev/zero (stat: No such file or directory)
httpd 182704 nobody mem REG 0,4 496619446 (deleted)/dev/zero (stat: No such file or directory)
httpd 182704 nobody mem REG 0,4 496619451 (deleted)/dev/zero (stat: No such file or directory)
httpd 182704 nobody 0r CHR 1,3 0t0 4350 /dev/null
httpd 182704 nobody 1w CHR 1,3 0t0 4350 /dev/null
httpd 182704 nobody 2w REG 253,0 579156 28185318 /usr/local/apache/logs/error_log
httpd 182704 nobody 3w REG 253,0 108426411 28185555 /usr/local/apache/logs/modsec_audit.log
httpd 182704 nobody 4r CHR 10,57 0t0 12393 /dev/lve
httpd 182704 nobody 5w REG 253,0 15691127 28186096 /usr/local/apache/logs/modsec_debug.log
httpd 182704 nobody 6u IPv4 495104908 0t0 TCP *:http (LISTEN)
httpd 182704 nobody 7u IPv4 495104914 0t0 TCP *:https (LISTEN)
httpd 182704 nobody 8r FIFO 0,8 0t0 496619412 pipe
httpd 182704 nobody 9w FIFO 0,8 0t0 496619412 pipe
httpd 182704 nobody 10w REG 253,0 229383576 28185359 /usr/local/apache/logs/access_log
httpd 182704 nobody 11u 0000 0,9 0 4348 [eventpoll]
httpd 182704 nobody 12w FIFO 0,8 0t0 496619414 pipe
httpd 182704 nobody 13u IPv4 497074376 0t0 TCP our_ip:http->50.93.203.147:46508 (CLOSE_WAIT)
httpd 182704 nobody 14w FIFO 0,8 0t0 496619417 pipe
httpd 182704 nobody 15w REG 253,0 0 28182908 (deleted)/usr/local/apache/logs/ssl-cache.60941
httpd 182704 nobody 16w REG 253,0 0 28185319 (deleted)/usr/local/apache/logs/rewrite-map.60944
httpd 182704 nobody 18w FIFO 0,8 0t0 496619449 pipe
httpd 182704 nobody 19r FIFO 0,8 0t0 496619450 pipeHere's the Apache log
50.93.203.147 - - [02/Nov/2015:13:10:52 -0800] "POST / HTTP/1.1" 200 domain.com 12744 "-"
50.93.203.147 - - [02/Nov/2015:13:13:08 -0800] "POST /xmlrpc.php HTTP/1.1" 200 domain.com 6607 "-"-
This should be addressed in the WP core in the next update: #34336 (Disable XML-RPC system.multicall authenticated requests on the first auth failure) " WordPress Trac In the mean time I have written some modsec rules but they might break some things like jetpack. Although I have tried to avoid this breaking any legitimate code or plugins, I offer no guarantee. As far as I can tell jetpack and other plugins very rarely use the multicall feature so these are probably safe but I have only used them on personal servers and not my customers servers. The first rule basically disables "system.multicall" entirely for properly formatted requests. The 2nd rule disables system.multicall for malformed requests that contain the two most commonly abused functions. #would work for posts with the proper content type specified but may break legit apps :/ SecRule REQUEST_HEADERS:Content-Type "^text/xml$" "phase:1,t:lowercase,nolog,pass,ctl:requestBodyProcessor=XML,id:3997" SecRule REQBODY_PROCESSOR "!^XML$" "nolog,pass,skip:1,id:3998" SecRule REQUEST_URI "xmlrpc.php" "t:lowercase,id:3999,deny,status:500,chain" SecRule XML:/methodCall/methodName/text() system.multicall #testing this because some POSTs are coming in as "application/x-www-form-urlencoded" not "text/XML" SecRule REQUEST_URI "xmlrpc.php" "deny,id:4000,chain" SecRule REQUEST_BODY "system.multicall" "chain" SecRule REQUEST_BODY "wp\.(getCategories|getUsersBlogs)"
I recommend only using these where you are actually being attacked, and then removing them after the next WP core updates are installed on the majority of your users sites. If you do test them on servers with many sites I would be interested in receiving your feedback regarding if you get any complaints or false positives.0 -
This should be addressed in the WP core in the next update: #34336 (Disable XML-RPC system.multicall authenticated requests on the first auth failure) " WordPress Trac In the mean time I have written some modsec rules but they might break some things like jetpack. Although I have tried to avoid this breaking any legitimate code or plugins, I offer no guarantee. As far as I can tell jetpack and other plugins very rarely use the multicall feature so these are probably safe but I have only used them on personal servers and not my customers servers. The first rule basically disables "system.multicall" entirely for properly formatted requests. The 2nd rule disables system.multicall for malformed requests that contain the two most commonly abused functions.
#would work for posts with the proper content type specified but may break legit apps :/ SecRule REQUEST_HEADERS:Content-Type "^text/xml$" "phase:1,t:lowercase,nolog,pass,ctl:requestBodyProcessor=XML,id:3997" SecRule REQBODY_PROCESSOR "!^XML$" "nolog,pass,skip:1,id:3998" SecRule REQUEST_URI "xmlrpc.php" "t:lowercase,id:3999,deny,status:500,chain" SecRule XML:/methodCall/methodName/text() system.multicall #testing this because some POSTs are coming in as "application/x-www-form-urlencoded" not "text/XML" SecRule REQUEST_URI "xmlrpc.php" "deny,id:4000,chain" SecRule REQUEST_BODY "system.multicall" "chain" SecRule REQUEST_BODY "wp\.(getCategories|getUsersBlogs)"
I recommend only using these where you are actually being attacked, and then removing them after the next WP core updates are installed on the majority of your users sites. If you do test them on servers with many sites I would be interested in receiving your feedback regarding if you get any complaints or false positives.
Thank you for this information. I will give it a try on my server and let you know if I have any result. :)0 -
Good deal. I know the rules work and have blocked attacks with both of them, but just curious if there are false positives on busier servers. Cheers. 0 -
Hello :) Thank you for taking the time to provide these Mod_Security rules as solution until the issue is addressed. 0 -
Hi could anyone please tell me how to block xmlrpc attack I use Free Modsecurity rules - Comodo Web Application Firewall but it does not stop them and the rule to stop them is enabled 0 -
Hello :), If you are getting this issues with only one site then you can add following code on your .htacces file to prevent this. Order Deny,Allow Deny from all0 -
Hi no it's loads of sites. 0 -
Hi how long until the next Wordpress update does anyone know. 0 -
Hi how long until the next Wordpress update does anyone know.
WordPress version 4.4 is available: Version 4.4 " WordPress Codex It includes the resolution for the bug referenced earlier in this thread: #34336 (Disable XML-RPC system.multicall authenticated requests on the first auth failure) " WordPress Trac Thank you.0 -
Hi thanks so i need to make sure all customers upgrade to new 4.4 and is this built in or do they have to enable it please. 0 -
Hi i found out today that if you delete ip.pag file in var/tmp and restart web server it will create a new one and stop all attacks. This works with comodo mod security rules. My file was huge. 0 -
Hi thanks so i need to make sure all customers upgrade to new 4.4 and is this built in or do they have to enable it please.
In 4.4, in a multicall request if auth fails the subsequent calls in that request will be denied. This is in the code and is nothing that has to be manually enabled. Once upgraded to 4.4 the site is protected from multicall requests being able to try multiple auth attempts.0 -
The problem is that multicall requests are not the only trouble. In fact multicall request give the attacker better chance to guess your passwords with less number of requests and most of the times, these type of requests are not the ones breaking down your server. What is the real trouble is bruteforcing while using single request per user/password try, which is the case with the single xmlrpc.php calls like "wp.getUsersBlog" or "wp.getUsersCategories", because this way the attacker needs to issue much more requests to your servers (and they do that), which immediately leads to spike in your resource usage. In short, the bugfix of Worpress guys which is mentioned as: #34336 (Disable XML-RPC system.multicall authenticated requests on the first auth failure) " WordPress Trac
, wont fix your xmlrpc.php ddos load spikes problems at all ! The effective protection to this attack is to try drop the request before they reach Wordpress, which could be done with ModSecurity. If you want to globally deny xmlrpc.php hits, you can do this with the following mod_Security rule:SecRule REQUEST_URI "@pm xmlrpc.php" "phase:1, id:22, deny, status:406, msg:'Blocked xmlrpc.php hit'"
Consider the following: - If some software relies on xmlrpc.php it will stop working (JetPack wp plugin is one example) - The current rule id is "22", make sure you have no other rules with same id (or change it to a free one) - Every request to xmlrpc.php or xmlrpc.php containing url, will end up with error code 4060 -
The multicall requests cause significant load. WP uses a processing intensive hashing algorithm. Being able to check multiple passwords with one POST request causes that single request to make the server hash hundreds of passwords. This is by design because "hash cracking" will take longer should a database be leaked. You do not need to block xmlrpc entirely. This is an overly heavy solution and should only be used temporarily on sites under heavy attack (if at all). If you're going to use ModSecurity you can specifically block just the system.multicall requests, as well as "wp.getUsersBlog" or "wp.getUsersCategories" by doing post payload inspection. This allows things like Jetpack (which are extremely popular) to still function. You can also filter xmlrpc.php requests that are missing user agents or HTTP referers, which drops a significant portion of both multicall attacks and single calls from bogus bots. Some requests can also be caught because they contain mismatched or missing headers (such as containing a content-length header with no content-type specified). The goal of a good modsecurity rule should be to only block bad requests while still allowing software to function as intended. Rules that break legitimate features are one of the reasons customers so often demand modsecurity be disabled, causing them to forfeit their own protections. 0 -
You do not need to block xmlrpc entirely. This is an overly heavy solution and should only be used temporarily on sites under heavy attack (if at all).
For anyone unsure of what xmlrpc actually is: Should You Disable XML-RPC on WordPress? - Wordfence0 -
But how many people actually use anything related to xmlrpc.php? I don't know about everyone else, but I'd factor a guess (conservatively) that 50% of the WordPress installs we have on our servers are single "install and never touch again" installations. At which point xmlrpc.php just becomes a venue that malicious users can easily exploit (or attempt to exploit) to hack into the installation and then do their bidding. There definitely are users that fully utilize everything regarding WordPress and xmlrpc.php, but I'm just not sure how many really are. My preferred method, for any model, is the principle of least privilege. Don't enable more privileges than are needed and if they are needed then those select users can enable them as needed. 0 -
Agree, just adding context to the thread. ;) 0 -
Further to this fix for WordPress 4.4. Are they going to fix this for WordPress 4.3? 4.2? 4.1? 4.0? 3.9? 3.8? 3.7? See this is the problem with trying to support so many versions, like WordPress has tried to do. Up until 4.4, they were releasing new versions for 4.2, 4.1, etc. alongside 4.3 updates. And now that just suddenly stops? 0 -
I would probably recommend using some form of xmlrpc.php blocking (ModSecurity is probably a good one) by default and exempting those VirtualHosts that really need access to xmlrpc.php. This is why ModSecurity is probably a good avenue for doing this, because you can exempt certain ids per VirtualHost. My real recommendation would be for WordPress to do an xmlrpc disable by default on new installs. But since I don't see that happening, taking matters into your own hands might be the only solution. 0 -
Least privilege is certainly a good security principle. It's much easier to enforce on small scale. I manage ModSecurity rules that are deployed to thousands of customers servers (and many of those may have hundreds of installations of wordpress each), so if I break anything for even 1% of people it's not a good day. If you just manage (for example) under 100 sites, you can probably get away with disabling it for a good percentage of them or even by default. Personally, I've had very good luck blocking the bulk of these attacks with more targeted modsec rules that inspect the aspects of the specific request to xmlrpc.php and reject the bogus ones, while leaving it functional for the people who use it. 0 -
Hi quizknows, The multicall requests cause significant load. WP uses a processing intensive hashing algorithm. Being able to check multiple passwords with one POST request causes that single request to make the server hash hundreds of passwords. This is by design because "hash cracking" will take longer should a database be leaked.
First of all, I want to make it clear, that I also stated, blocking whole xmlrpc.php would break software which uses it. But if you maintain just one site (or a several ones), and none of them uses the xmlrpc.php functionality, this could be handy solution. About the multicall requests, yes they cause load, but especially in my case, they were < 1% compared to the overall xmlrpc.php targeted traffic, which used wp.getUserBlogs and others like this.You do not need to block xmlrpc entirely. This is an overly heavy solution and should only be used temporarily on sites under heavy attack (if at all). If you're going to use ModSecurity you can specifically block just the system.multicall requests, as well as "wp.getUsersBlog" or "wp.getUsersCategories" by doing post payload inspection. This allows things like Jetpack (which are extremely popular) to still function. You can also filter xmlrpc.php requests that are missing user agents or HTTP referers, which drops a significant portion of both multicall attacks and single calls from bogus bots. Some requests can also be caught because they contain mismatched or missing headers (such as containing a content-length header with no content-type specified). The goal of a good modsecurity rule should be to only block bad requests while still allowing software to function as intended. Rules that break legitimate features are one of the reasons customers so often demand modsecurity be disabled, causing them to forfeit their own protections.
If you are talking from the point of view of hosting provider, you are definitely right. Rules from protecting hosting company customers by these attacks are much more complex (I have also written such). But still, if you are sure your site doesn't use xmlrpc.php at all, I don't see any problems for blocking it entirely. Also, I don't think there is big difference between blocking whole xmlrpc.php access or just blocking wp.GetUsersBlog and wp.GetUsersCategories(I have cases in my mind, where it is even worse than blocking whole xmlrpc.php) . If you (or your software) uses xmlrpc.php there is big chance to also use system.multicall or getUsersBlog or ...whatever it is, also jetPack uses system.multicall too. And again, it strongly depends on user/client demands and usage. Cheers.0 -
Jetpack does use system.multicall but it seemed to be an extremely rarely used feature. I did not block that feature except on hosts where I was specifically requested to. Doing header analysis (user agent or lack there of, content type header, etc.) seems to be enough to block the majority of this. Of course, system.multicall is still a very small percentage of the xmlrpc.php attacks overall. I could probably parse out some data to get exact numbers, but the individual call requests are certainly much more prevalent. Also it has to be accounted for that a single multicall request can be the equivalent of several hundred (or even a thousand) individual calls, so the sheer number of POSTs using that method is quite misleading to us all. I also agree for a single user, if they know they're not using it, disabling xmlrpc.php is fine. Cheers mate. 0 -
Hi guys, Due to this xmlrpc attack issue, I've been trying to find a way to block the malicious xmlrpc attacks but still allow people who use jetpack to use it. I'm using the following code: ## Protect wordpress xmlrpc from attack Order Deny,Allow # Whitelist Jetpack/ Automattic CIDR IP Address Blocks Allow from 192.0.64.0/18 Allow from 209.15.0.0/16 Allow from 66.155.0.0/17 Deny from all
But in the access logs I see the following:192.0.99.18 - - [21/Jan/2016:08:03:22 -0800] "POST /xmlrpc.php HTTP/1.0" 403 1097 "-" "The Incutio XML-RPC PHP Library" 192.0.81.13 - - [21/Jan/2016:08:08:21 -0800] "POST /xmlrpc.php?for=jetpack&token=z9T%25irLBEzDrJgcYG%5EFZOwR9%24%266SBdu3%3A1%3A1×tamp=1453392490&nonce=YLdt2St1cK&body-hash=B1DQHoOeqzuFhFZSfMF2Cgh0Gpg%3D&signature=Dlh1unvh0TNSiWmzOE4XtjvhC3A%3D HTTP/1.0" 403 1097 "-" "Jetpack by WordPress.com" 192.0.81.13 - - [21/Jan/2016:08:08:24 -0800] "POST /xmlrpc.php HTTP/1.0" 403 1097 "-" "The Incutio XML-RPC PHP Library" 192.0.81.13 - - [21/Jan/2016:08:13:51 -0800] "POST /xmlrpc.php?for=jetpack&token=z9T%25irLBEzDrJgcYG%5EFZOwR9%24%266SBdu3%3A1%3A1×tamp=1453392820&nonce=VVpHthM4e6&body-hash=B1DQHoOeqzuFhFZSfMF2Cgh0Gpg%3D&signature=xiqHOjOJ6fZ0ESY7CIbo1PAj%2FS0%3D HTTP/1.0" 403 1097 "-" "Jetpack by WordPress.com"
Those ips 192.0.99.18 ect are in the subnet I have allowed, but they get 403 message in the logs. Does that mean Jetpack is not working? Thanks0 -
Try one of the following: 1. Change the Order statement to: "Order Allow,Deny" Or 2. Move the "Deny from all" at the beginning 0 -
add this to the apache config, this works for us, and will allow jetpack Order Deny,Allow # Whitelist Jetpack/ Automattic CIDR IP Address Blocks Allow from 192.0.64.0/18 Allow from 209.15.0.0/16 Allow from 66.155.0.0/17 Deny from all 0 -
Your "Order" is backwards. It should be Allow,Deny, or you should move your deny rule above your allow rules to follow the order you are specifying. 0 -
Thanks guys. I've made the change and its all good. 0
Please sign in to leave a comment.
Comments
27 comments