How do I disallow MOZ from crawling URL's with parameters?
-
I'm seeing hundreds of pages with issues on my crawl reports. When I research the URL's, I see they are all duplicates of the same URL's, with parameters following the question mark symbol.
I have already added the URL Parameter rules to Google Webmaster Tools, so Google knows how to handle them.
Does MOZ have a similar setting that I can edit?
-
Hi there
Don't forget your canonical tags! You should read through Google's resource on duplicate content as well.
You can block this access with your robots.txt by using rogetbot as your user agent.
You can read more from this Q+A thread as well.
Hope this helps! Good luck!
-
Yes definitely don't forget canonical tags where possible. But I think those pages are being auto generated via searches on your website or some other means. Most good Seo plugins or extensions prevent these pages from being indexed. Second to that, find the parent page and use use a wildcard block command in your robots.txt file. It should look something like this:
User-agent: *
Disallow: /your-inner-page/*That command will block every url variation after www.yoursite.com/your-inner-page/ from being indexed or crawled but still allow your real page url of www.yoursite.com/your-inner-page/ to be crawled and indexed.