IP Address Crawlable

tribulant

• September 18, 2017 02:40

Good day, It seems like Google is crawling our website on it's IP address, hostname, etc. at Amazon EC2 eg: example.com/blog/ ec2-54-88-xx-xxx.compute-1.amazonaws.com/blog/ 54.88.xx.xxx/blog/ How can we make sure only the domain is crawled and accessible?

• 

  rpvw

  • September 18, 2017 08:22

  If your domain name resolves to an IP (and it must for anyone to be able to call the website using the domain name) the DNS is probably correct. I am intrigued as to how the DNS is resolving to the /blog website using the hostname and the IP in the first place. Perhaps this would better be addressed by your web host. There is no way of really influencing what the various robots can crawl other than by using a robots.txt file (or include crafted meta tags into the headers of the parsed web page) to exclude certain folders/files from being searched. As far as I know, if you put an IP into this file, it would be ignored, and you need to remember that not all the robots respect the robots.txt or meta tag conventions anyway ! Hope this helps.

  0
• 

  tribulant

  • September 19, 2017 09:32

  If your domain name resolves to an IP (and it must for anyone to be able to call the website using the domain name) the DNS is probably correct. I am intrigued as to how the DNS is resolving to the /blog website using the hostname and the IP in the first place. Perhaps this would better be addressed by your web host. There is no way of really influencing what the various robots can crawl other than by using a robots.txt file (or include crafted meta tags into the headers of the parsed web page) to exclude certain folders/files from being searched. As far as I know, if you put an IP into this file, it would be ignored, and you need to remember that not all the robots respect the robots.txt or meta tag conventions anyway ! Hope this helps.

  
  Thanks I will continue to investigate and post back here once I have a solution to this. In the meanwhile, I hope that Google doesn't penalize for this.

  0
• 

  rpvw

  • September 19, 2017 10:32

  Is your domain name the same as your hostname ? The fact that your website resolves using the AWS PTR confuses me, but then I have little experience with cloud based servers, and maybe this is normal for Amazon In any case, you may want to review the

  0
• 

  kssuhesh

  • September 19, 2017 11:12

  Hello, You should add a 301 redirect to one url, once you 301 redirect everything, all the urls currently indexed with the IP will eventually drop out in favor of the same urls on the domain to which you redirected them. You can use the redirect in the following link for doing the same. - Removed -

  0
• 

  cPanelMichael

  • September 19, 2017 15:09

  Hello, You can follow the steps referenced below (partly taken from our interface (WHM Home >> Service Configuration >> Apache Configuration >> Include Editor). 2. Select the Pre Virtual Host Include option. 3. Select the Apache version from the menu. We recommend that you select All Versions. 4. Enter the following text in the available text box:
  ServerName HOSTNAME DocumentRoot /usr/local/apache/htdocs ServerAdmin EMAIL suPHP_UserGroup nobody nobody ServerName HOSTNAME DocumentRoot /usr/local/apache/htdocs ServerAdmin EMAIL suPHP_UserGroup nobody nobody SSLEngine on SSLCertificateFile SSLCERTIFICATEFILE SSLCertificateKeyFile YOUR-SSLCERTIFICATEKEYFILE
  Note: This example uses the following values: [LIST]

  IPADDRESS represents your account's dedicated IP address.

  HOSTNAME represents your server's hostname.

  EMAIL represents your contact email address.

  SSLCERTIFICATEFILE represents the full file path to your SSL certificate.

  SSLCERTIFICATEKEYFILE represents the full file path to your SSL certificate's key. 5. Click Proceed 6. Click Update. Thank you.

  0

Please sign in to leave a comment.