Blocking many IP addresses
With GenAI changing the way some users interact with websites, we are seeing an increasing number of IP addresses crawling the website. While these are easy to spot, and apart from the unnecessary workload on the server, there are too many to use tools like "IP Blocker", given the impact on Apache and a large htaccess file slowing up a website/affecting core vitals. How can a large number of IP addresses best be blocked? Currently, we use IP Tables as the filewall and it has the same constraint as htaccess in that a large number affects performance, using a sequential scan.
Given that Nginx is used for caching, etc., is that a better place to reject any requests to the website? I believe it has a better approach to blocking IP addresses and is more efficient.
Any thoughts in terms of the best approach is appreciated.
Thank you.
PS, we are talk 10s of thousands of addresses.
-
Hey there! Do you have certain criteria that you use to block these IPs? Is this something that could be done in an automated fashion through a more advanced firewall? We're still recommending CSF if you need fancier tools for managing the firewall on cPanel machines (and we'll have our own release of that soon as well) but if you could let me know how you're detecting these IPs and what is currently working for you I may be able to provide better details.
0 -
Well, it's hard to come up with an exact criterion that's foolproof, maybe where x pages have been called in y seconds would do it, if CSF would cope with that? Basically, the bots are quicker to ask for the next page, whereas a human needs to recognise, read and decide what the next page is they want.
Can CSF handle that?Thank you!!
0 -
Unfortunately no, this sounds like you would need a more advanced, possibly even external hardware firewall to deal with that level of traffic and filtering.
Have you reached out to your host to see if they have anything to offer?
0 -
Well, we have a dedicated server. I could add an external firewall, which I could configure, but it just seems an excessive way to handle the situation. As these sort of crawlers are on the increase, you may like to consider a solution. Of the options I have, I think Nginx is probably the best of a possibly bad bunch. Agree?
Thank you!0 -
The issue with any server-side tool is that the server itself will still need to process the traffic. If you're seeing tens of thousands of IPs causing an issue that could increase the server load even just by reading the configuration files each time it checks an IP address.
0 -
I agree, but it will/should be less than processing a whole web page + database queries, etc. The only real impact is if it delays the website's core vitals beyond acceptable parameters, but that requires testing.
Agree?
0 -
Maybe - it's really hard to say as this isn't something that's part of cPanel, so I can't give an official answer.
0 -
I may test it. If I do and fine it's OK, will advise, especially if it makes any real difference on time to first byte, which is where I'd expect it to. Thank you.
0
Please sign in to leave a comment.
Comments
8 comments