Skip to main content

How to prevent scrapers using multiple IPs

greektranslator
At times I get massive spikes of "visitors" coming from very wide IP ranges. For example: xxx.yyy.145.193 xxx.yyy.159.123 Where xxx.yyy are the same numbers. Would manual banning such huge ranges be meaningful? Is there any other easier way to counteract them? I do have CSF installed.

Comments

8 comments

Date Votes
  • cPRex Jurassic Moderator
    Hey there! Do you specifically know if they are bots from a certain provider? If so, you could setup a robots.txt to keep them from indexing your content, or you could just block the entire IP range. There's nothing wrong with blocking a whole range, but it does open up the possibility of you blocking user's IPs that were not intended.
    0
  • greektranslator
    "but it does open up the possibility of you blocking user's IPs that were not intended" - yes, that is what I was worried about. Also, such a huge range blocking would affect server performance?
    0
  • cPRex Jurassic Moderator
    CSF lets you use CIDR notation so you don't have a massive file with all the IPs listed, so that saves on resources significantly.
    0
  • keat63
    Perform a quick google search on the IP's, chances are xxx.yyy are from the same country and you may find that you have no dealings with such country. In which case just block the whole country. Problem solved. I have half a dozen parts of the globe blocked.
    0
  • greektranslator
    CSF lets you use CIDR notation so you don't have a massive file with all the IPs listed, so that saves on resources significantly.

    Still, the system has to check for numerous IPs nonetheless or using the ranges implies that this does not happen?
    Perform a quick google search on the IP's, chances are xxx.yyy are from the same country and you may find that you have no dealings with such country. In which case just block the whole country. Problem solved. I have half a dozen parts of the globe blocked.

    Yeah, the source was Singapore :) Still, someone could launch the attack via a proxy service. The point is, I am looking for an automated solution that will recognize irregular visit patterns from a group of similar IPs. Is there something like this?
    0
  • cPRex Jurassic Moderator
    You'd have to contact CSF to see how they handle the CIDR ranges internally - I'm not sure what processing goes into that on their end. While this is an older post, a few users commented that they have blocked many ranges with no performance issues:
    0
  • keat63
    Personally, I feel that these things go in waves. Block Singapore, forget about. Unblock Singapore in 3 months time and you'll probably find that it's gone away.
    0
  • vicycletwo
    Rather than blacklisting bots, maybe you should whitelist them. If you don't want to kill your search results for the top few engines, you can whitelist their user-agent strings, which are generally well-publicized. The less ethical bots tend to forge user-agent strings of popular web browsers. The top few search engines should be driving upwards of 95% of your traffic.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post