Intermittent issue with ISP being blocked from server
I'm having a rather odd issue, and I can't tell if it's on my end or not.
I've had reports from two people that use Verizon Wireless that they can't reach any of the sites on my server. This began on Friday.
It's worth noting that on Friday, someone local had tried to hack in to my server, and I had blocked several IPs in the firewall (CSF). So, I've been operating under the theory that I had inadvertently blocked Verizon.
But here's where it gets weird:
1. The first person was using an Aircard. They changed the DNS Server on their laptop to OpenDNS, and it solved the problem.
2. The second person has been working with me to track down the problem. They sent their IP, and I whitelisted their entire /8 range, but that didn't help. When I disabled the firewall entirely, though, it instantly came back up.
So, they downloaded an app to run traceroute from their phone to my server, giving me 24 hops along the way (which I noted were VERY slow; in some cases, over 3.5 seconds on a single hop). I whitelisted each of those hops at the /16 range, and everything started working again... until a few hours later, when it was denied again.
They sent another traceroute report, and it had the same IPs, so nothing here changed. I saw, though, that it was failing when the traceroute got to the IP 0.0.0.0, which isn't really an IP; it implies a DNS problem.
Since thing, we've duplicated the same thing numerous times; I can whitelist an IP or remove an IP from the blacklist, and the phone will connect, but then I change it back to the original state, and it STILL connects. After several hours, though, it starts to fail again without warning.
Simply restarting the firewall doesn't seem to help; it only changes when I add or remove something to the white/blacklist. Coincidence? Maybe, but I've duplicated it 3 times.
This last time, they tried running a traceroute when it failed, and had a new error:
But a scan with MXToolbox and DNSStuff show that the DNS is perfect, except for minor warnings (SOA refresh and expire too high). A scan with DNSInspect fails, though, until I turn off the firewall; then, it comes back the same as DNSStuff. The only additional statement (a "notice", not a "warning") is that they found name servers without AAAA records (whatever that means). I should also mention that on Tuesday of last week, I did change /etc/named.conf to include:
This was based on a suggestion from DNSInspect. With all of that, I'm at a little bit of a loss. Am I dealing with a firewall problem on my end, or a DNS problem on Verizon's end? Or something on my end that's CAUSING a DNS problem on Verizon's end?
IP or hostname is invalid
Only IPv4 address with working reverse dns are supported
or hostnames with working dns
But a scan with MXToolbox and DNSStuff show that the DNS is perfect, except for minor warnings (SOA refresh and expire too high). A scan with DNSInspect fails, though, until I turn off the firewall; then, it comes back the same as DNSStuff. The only additional statement (a "notice", not a "warning") is that they found name servers without AAAA records (whatever that means). I should also mention that on Tuesday of last week, I did change /etc/named.conf to include:
options {
...
version "unknown";
};
This was based on a suggestion from DNSInspect. With all of that, I'm at a little bit of a loss. Am I dealing with a firewall problem on my end, or a DNS problem on Verizon's end? Or something on my end that's CAUSING a DNS problem on Verizon's end?
-
Hello :) Have the users tried updating the resolvers used on their systems to see if that makes a difference (e.g. 8.8.8.8)? Have you tried removing the customization you made to /etc/named.conf to rule it out as the cause of the problem? Thank you. 0 -
The first user changed their DNS resolver to OpenDNS, and it did solve the problem. The second user is using an iPhone, though, and I don't think that changing the DNS is an option unless on WiFi. With two users reporting a problem, though, the concern is that I may have inadvertently blocked a lot of people, but only 2 have reported it; especially if the block is at the firewall level, in which case they might not even be able to email me to report it. So I need to find the problem on my end instead of changing each user's DNS one at a time. I did try removing the named.conf change last night, but since I had made it on Tuesday and the problem didn't start until Friday, I'm not optimistic. The user hasn't reported back today as to whether they're still locked out. 0 -
You could temporarily disable the manual IP address blocks in your firewall and keep the default rules active to help determine if it's part of the problem. Thank you. 0 -
I did try that, and the result was very confusing. When the user reported that they couldn't connect again, I removed all of the IPs from the DENY list, and they could then connect. I added 10 back in, restarted the firewall, and they could still connect. So, I added 10 more, and so on. Eventually, I had added all of the IPs back, and they could STILL connect! Even though it was exactly like it was before, when they couldn't connect. About 12 hours later, they couldn't connect again, even though their traceroute showed the same IPs, and no new IPs were in the blacklist (temp or permanent). 0 -
You may need to consult with your data center or a system administrator to have them review your configuration and determine if there are any network issues at fault. Thank you. 0
Please sign in to leave a comment.
Comments
5 comments