Crawl Diagnostics

[Studio33]

My site was crawled last night and found 10,000 errors due to a Robot.txt change implemented last week in between Moz crawls.  This is obviously very bad so we have corrected it this morning.  We do not want to wait until next Monday (6 days) to see if the fix has worked.  How do we force a Moz crawl now?

Thanks

[olsenpartnership]

"Submit URL to Index," which allows you to submit new and updated URLs that Google themselves say they "will usually crawl that day"

[Studio33]

Thanks but I require a Moz crawl first.

[LynnPatchett]

Hi,

I think this will do what you are after: http://pro.moz.com/tools/crawl-test, limited to 3000 pages but should give an idea if the fix is working as you expect.

[Chris.Menke]

I'm pretty sure that you're not able to force a refresh of your campaign stats in between your normal weekly crawl.  This tool will crawl the site but it doesn't refresh your campaign.  Specifically, what errors were found that you're trying to get rid of?

[Studio33]

Thanks

9000 duplicate content and duplicate page titles caused by my blog.  I have added

User-agent: *Allow: /Blog/post/Disallow: /Blog

to the Robot.txt to just allow the main site and the Blog posts

Is this a good way to fix it?

[LynnPatchett]

Hi,

As Chris says I don't think there is a way to force a refresh on your campaign crawls, but that crawl test tool should be able to give you an indication if the relevant pages are still producing duplicate content issues or if the fix seems to be reducing them.

That being said, I don't think that robots.txt is the best way to approach duplicate content issues generally. Check out this guide for best practices. It is also worth noting that many times duplicate content issues can be solved by simply removing or adjusting the various differently formatted links that are producing them in the first place (though this depends a lot on which cms you are using and what the root cause of the duplicate content is).

[DougRoberts]

Why so many duplicates? As it's a blog I suspect it's something to do with tags and/or categories.

Instead of trying to hide the problem using the robot.txt file can tackle the root cause directly?

[Studio33]

Its a dotnetblogengine.com blog its open source but not sure where to start