Possible Crawling Problem with Screaming Frog and Moz Crawlers

[CJ5]

So I'm not sure if what I'm seeing is a problem or not.

As of about two weeks ago the Moz crawler has only been able to see www.mysite.com, and none of the links, content, title, ect associated with the page. Essentially the report has one line, what should be the homepage, but it's not able to pull any information from the page but does show a 200 http status code. The report shows nothing blocked by robots or any errors.

When I use screaming frog to crawl the site about 75% of the time it just reports one line www.mysite.com with a 200 status code, but again the crawler is not able to actually see the html. The other 25% of the time it works perfectly fine, crawls all pages and sees all meta info and content.

There are no errors in Google WMT and everything looks ok there. We have seen a traffic drop the last two weeks but I don't know if this is the reason for it.

I can't publicly post the page but if someone has an idea of what might be going on I'd be happy to PM them.

Thanks

[Ray-pp]

If you PM me the domain I can take a look myself.

Does the robots.txt have anything funny in there?

[MichaelEka]

Two possible simple key-in items to consider: make sure the URL is inputted w/ the full url (not just mysite.com) and/or ensure to click any options for including root or sub-domains so its not just looking at a single page.

[CJ5]

Thank you for the response.

I've ran two MOZ crawl reports today, one with mysite.com and one www.mysite.com. Both returned 1 result for mysite.com and www.mysite.com respectively, with a 200 status code, but no meta data. I know that I've successfully crawled www.mysite.com about a month ago with no problems. I have made small changes here and there but nothing is jumping out at me as wrong.

Screaming Frog is currently crawling my site successfully about 1/10 tries. The successful tries it sees 163 Total URL Encountered (its a small site) and the other 9/10 times it shows exactly 1 URL (the one i entered) and no meta data. There doesn't seem to be any pattern when it successfully crawls and when it doesn't make it past the first page.

Google WMT is currently showing No Data Available for both internal links and links to your site which is a little concerning. Everything else in WMT looks ok.