Is there a way to control a crawl?

[JoshAsteteWebFullCircle]

I need to decrease our campaigns to 1,000 crawls we are currently at 10000 per

[TakeshiYoung]

Are you asking about crawls or indexation?

I can't think of any reason why you would want to decrease crawls, but you could probably do so by getting backlinks removed, blocking sections of your site from being crawled through robots.txt or nofollow links, and decreasing the change frequency in your xml sitemaps.

[MattRoney]

Hi Josh!

Thanks for posting in Q&A! Just to clarify, are you referring to your _pages _crawled in Moz Analytics?

[Travis_Bailey]

Kind of what I was wondering. I think this is a product deal, since they referenced 10000.

[JoshAsteteWebFullCircle]

This post is deleted!

[JoshAsteteWebFullCircle]

Yes I am talking about pages crawled.

[DavidLee]

Hey Josh!

With the latest increase of crawl limits for all campaign, unfortunately the lowest is set for 10,000. If you have less than 10,000 pages then we will only crawl the total amount of pages you have. Takeshi has great tips on excluding certain pages from being crawled in robots.txt

Our user-agent is rogerbot so an example would be

User-agent: rogerbot
Disallow: /categories/

Hope this helps!