Bandwidth usage issues on various accounts at different times.
It seams like every month I'll have an account or two that far exceed their bandwidth limit. I get the standard e-mails from the server:
[domain.org] The account “mydomain” with primary domain “domain.org” has reached its bandwidth limit (29.57 GB/29.3 GB).
These sites typically run about 3-5GB per month, some months they may see 5-7GB. But then once in a while, they get run up to 15, 20, 25, 35, 45, etc....
I have root, access. so any diagnostics at the server should be possible.
When I review the stats (Bandwidth, Webalizer, and Awstats) I can't seem to find the issue.
For this particular issue, Bandwidth shows 4.3GB for yesterday on the server, a normal day is 1.1GB. It's 99.999% HTTP.
Webalizer shows 50 MB for non-ssl on this account yesterday and 4GB for ssl on this account yesterday and 5GB the day before. Typical is 200MB a day.
Awstats shows similar numbers, but also shows nearly all requests coming from IP: 47.79.X.X
My host has been unable to assist with figuring this out.
I want to know is what are the best steps to stop this? If I look up 47.79.4.196 it comes back as being on the Alibaba Cloud and is reported as spam.
If blocking them is the best solution, how do I go about doing that when there are so many varying IPs?
Thanks.
-
Look in Raw Access logs, I think you will find out it's Web crawler bots.
0 -
If I am reading correctly, (at least from that IP range) they are performing a "get" on every single entry in our public calendar. Would this be a crawl?
0 -
It's extremely common for bots to use up a large amount of bandwidth, as quietFinn said. Even though it doesn't seem like much, it adds up to way more than you think should be possible from a crawler, and would explain what you're seeing.
It may be worth looking into ModSecurity rules to handle this type of traffic as well: https://support.cpanel.net/hc/en-us/community/posts/19141032118679-Help-with-Globally-blocking-Bad-Bots
0
Please sign in to leave a comment.
Comments
3 comments