Why is site not being indexed by Google, and not showing on a crawl test??

[RobertFisher]

On a site we developed of which .com is forwarded to .net domain, we quit getting crawled by google on about the 20th of Feb. Now when we try to run a crawl test on either url, we get     There was an error fetching this page. Error description     For some reason the page returned did not describe itself as an html page. It could be possible that the url is serving an image, rss feed, pdf, or xml file of some sort. The crawl tool does not currently report metrics on this type of data. Our other sites are fine and this was up to this date. We took out noodp, noydir today as the only thing we could think of. Site is on WP cms.

[Guest]

This post is deleted!

[G-Force]

Hi Robert Fisher,

This problem probably come from the headers of the file and not from the content itself.  You might want to look at the headers returned by your URL using one of the following tools :

http://www.seoconsultants.com/tools/headers
http://www.rexswain.com/httpview.html
http://web-sniffer.net/
http://www.g-force.ca/referencement/entetes

When you got the headers, I suggest you post it here so we can look into it.

Best regards,
Guillaume Voyer.

[StalkerB]

Probably going to need the URL on this one.

I presume you can access the site as a user? What's in your robots.txt file? You using the SEOmoz tools?

[RobertFisher]

The site is www.innerloophomesreport.net, .com. Thanks.

[StalkerB]

Site last cached 2nd March

Your site is indexed.

Header's returning 200 codes.

Site can be crawled fine, Xenu finds about 27 pages.

Lynxviewer gets through the page alright.

Only thing I can think of is that robots.txt looks needlessly complicated but should be alright, I would consider stripping it all out and re-running the test, if you get the same error then it's not that, if it is then narrow down what it could be.

If no joy, let me know and I'll have another look.