Screaming frog, Xenu, Moz giving wrong results

[GPainter]

Hello guys and gals,

This is a very odd one, I've a client's website and most of the crawlers I'm using are giving me weird/ wrong results.

For now lets focus on screaming frog, when I crawl the site it will list e.g. meta titles as missing (not all of them though), however going into the site the title is not missing, and Google seems to be indexing the site fine. The robots.txt are not affecting the site (I've also tried changing the user agent).

The other odd thing is SF gives a 200 code but as a status tells me "connection refused" even though it's giving me data.

I'm unable to share the clients site, has any one else seen this very odd issue? And solutions for it?

Many thanks in advanced for any help,

[PatrickDelehanty]

Hi Chris

This is hard to answer without an example - do you have a URL or two that you can share with the community?

Thanks so much!

[Andy.Drinkwater]

I have shared this question with Dan from Screaming Frog for you

-Andy

[screamingfrog]

Cheers for the tweet Andy.

Chris, I'd recommend running through this list here in our FAQ -

http://www.screamingfrog.co.uk/seo-spider/faq/#17

In particular, I'd recommend reducing the crawl speed & adjusting the user-agent.

You can also pop it through to our support (http://www.screamingfrog.co.uk/seo-spider/support/) & we can advise. All confidential etc.

Cheers

Dan

[PatrickDelehanty]

^ That's customer service!

[MattRoney]

Thanks Dan!