Meta Noindex report: exclude urls and delete all warnings in once
-
Dear community,
I have a question about "Site Crawl" > "Crawler Warnings" > "Meta Noindex" reports. I see 3.6K errors with all the same url base: /register?url={{xxxxxxxx}}
The page have the following robots meta tag: 1. Can i exclude some urls from crawling by moz-spider bot? If yes, how?
2. Can i use the 'mark as fixed' or 'ignore' functionality for all urls in once? Now, i have to execute the action 36 times to 'ignore' all 3600 errors.Hope, someone can help me

-
Hi,
1. You can exclude the Moz crawler (Rogerbot) from various pages: https://moz.com/help/moz-procedures/crawlers/rogerbot
You need to use your robots.txt file, rather than trying to do it on page2. Once you've done the above, you can request a recrawl and they should all disappear