Questions
-
How Do I Scan My New Site & Grade My Work With The Robots Turned Off? For Pre-Inspection before I launch my Site?
You could block the site for all bots except rogerbot ( Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot). Alternative would be to use a local crawler like Screaming Frog (and modify the spider settings so that it ignores the robots.txt) rgds, Dirk
Getting Started | | DirkC0