bandwidth usage causing problems

linkup

• April 25, 2024 20:35

Hello,  I run a small server with about 50 domains.  Of the 50 domains, four of them are constantly being hammered, presumably by bots.

1.  worst one is the domain of a retired university professor and probably has next to no real traffic, but his domain is being pounded, mainly from Spain, mainly semrushbot and bingbot

2. domain is a family domain, hardly any "real" traffic, being hammered from Canada

3.  another small domain, hardly any real traffic, being hammered from the US by dotbot

4.  another small domain, hardly any real traffic, getting hammered from the US by nbot.  What is odd about this one, when you look at where the traffic is coming from, it is the SERVER IP???? therefore I can't even block it for a temporary respite.

I am guessing the other domains have some amount of the same problem but blocking IP addresses doesn't seem to help anything because, as you know, they just switch to another IP.  The last two sites are Wordpress, and I have attempted to do some blocking in WP with no success.

The server is running CentOS v7.9.2009 and cPanel 110.0.31

I know the server needs to be updated, but is that related to the problem or not?  Is there  something else that can be done other than constantly blocking IPs, other than the 4th one which I can't even block via the IP?

Thanks

• 

  DennisMidjord

  • April 25, 2024 21:46

  SemrushBot is a known bad bot.
  What you can do to block these bots is block the user agent.

  Login to WHM and navigate to Service Configuration > Apache Configuration.
  Access "Include Editor".
  In "Pre Main Include" for all versions, enter the following:
  ```
  <Location />
      SetEnvIfNoCase User-Agent SemrushBot badbot
  
      Order Allow,Deny
      Allow from All
      Deny from env=badbot
  </Location>
  ```
  Add a new SetEnvIfNoCase line for each user agent you want to block. It will result in a 403 whenever they access any webpage on your server, resulting in much less bandwidth being used..

  1
• 

  linkup

  • April 25, 2024 23:14

  Thanks very much...I am sure that will help...turns out the worse offender after evaluating four domains is one called "nbot".  That bot does not appear on a good or bad lists I found but on a small site generated 37GB of bandwidth.  Another high user is simply called "feed" which had 830K hits on a site with 13gb of bandwidth.  Bothers me more not knowing what they are...

  Really appreciate it.

  0
• 

  cPRex Jurassic Moderator

  • April 26, 2024 13:37

  Thanks, DennisMidjord!

  0

Please sign in to leave a comment.