Limiting my MOZ crawled pages

[MrWoc]

Hi there

Is there any way i can limit the amount of pages crawled on my site ? as it says i have used up all my crawl allocation and asks me to upgrade ?

J

[KeriMorgret]

We actually just upgraded to 50,000 URLs per site. Are you hitting this new limit?

[LesleyPaone]

I think there are a couple answers to this question. The first would be when was your last crawl? Last week they up the amount of crawled pages per campaign by 5 fold to 50k pages, http://moz.com/community/q/10-000-page-crawl-limit-too-low-well-we-ve-increased-it-to-50-000  The second would be in your campaign settings you can limit the number or crawled pages, http://moz.com/help/guides/getting-started/campaign-settings

The third is a little bit more tricky, but it can be helpful. In your robots.txt you can use useragent: rogerbot to block directories and other parts of your site. Like say you have a huge blog and a huge product section, you can block the product section one week from the crawl, then the next week block the blog and allow the products.

[KeriMorgret]

A fourth possibility is that there are some URLs that are duplicates and should be fixed, which will help you in the search engines as well as reduce your number of URLs crawled.

• Are both www and non-www indexed?
• Are there a bunch of URL parameters indexed, such as six different ways to sort items?
• Are tracking parameters enabled and creating 50 versions of the same page?

[LesleyPaone]

I looked at your profile and google'd your site. It seems you have 470k indexed pages, which I assume is why you are hitting the limit. My first thought would be to exclude the profile pages and crawl the core site pages where you have control of. But, I don't know if that is possible with your url structure. If something like that can be pulled off I would try it, because you really have no control over user generated content pages, they are pretty much going to "fall where they may" in the rankings.

[MrWoc]

Hi There

Many thanks for the replies

I looked in campaign settings and couldn't see how to limit the number of pages crawled ? i looked at the page you referred me to and couldn't find it.

So your saying it would probably be better to stop the profile pages being indexed at all Lesley ? I am not sure if i can do this, but will try, as I have been thinking these pages are quite low quality anyhow, do you have any tutorials you could refer me to ?

J

[KeriMorgret]

If the profiles were in their own directory, like domain.com/profile/user1, it'd be easy to exclude /profile/ in robots.txt. However, your profiles look like they're off the root. In your software system, is there any way to add a noindex tag to profile pages, or make any type of control over indexing of types of pages?

Is there one main section (folder) of your website that has content that you want optimized? You could also set up a campaign for just that folder, such as domain.com/myimportantcontent/.