Welcome to the Q&A Forum

Hey Josh! With the latest increase of crawl limits for all campaign, unfortunately the lowest is set for 10,000. If you have less than 10,000 pages then we will only crawl the total amount of pages you have. Takeshi has great tips on excluding certain pages from being crawled in robots.txt Our user-agent is rogerbot so an example would be User-agent: rogerbot Disallow: /categories/ Hope this helps!