Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Do you know any tool(s) to check if Google can crawl a URL?

3 3 863

Log in to reply

esiow2013 last edited by

Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.

In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.

Thanks in advance!
1 Reply Last reply
Reply Quote 0
masdigitalmedia last edited by

You could consider this online tool: http://web-sniffer.net
1 Reply Last reply
Reply Quote 1
danwebman last edited by

I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
1 Reply Last reply
Reply Quote 1

Browse Questions

View

From

Sorted by

With category