SEOMOZ Crawling Our Site

[BeerCartel75]

Hi there,

We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist.

Thanks,

Geoff

[AdamThompson]

Hi Geoff,

There are multiple reasons your site could have "extra" pages: tag pages, category pages, print this pages, duplicate pages, etc.

To see what pages SEOmoz is crawling:

1. Go to http://pro.seomoz.org/campaigns
2. Click View This Campaign for your site
3. Click Crawl Diagnostics
4. At the top right, Export as CSV
5. You'll get a spreadsheet listing all the URLs that SEOmoz crawled

Hope that helps!

~Adam

[MattAntonino]

In addition to what Adam said, you can just search your site on Google and I come up with over 14,000 results:

Both ways should give you a good insight into what's being found.

unifor.jpg

[BeerCartel75]

Thanks Adam.

Have had a look and it looks like the crawlers are cycling through numerous unnecessarily crawled pages for each product based on the top and bottom nav bar on our website (i.e. it adds FAQ and Shipping Info for each product + all the categories we have created on the website). Attached is an image link of some lines from the CSV file.

I imagine this is what is making it chew up bandwidth. Any insight on how to avoid / change this would be much appreciated.

Thanks

Geoff

4RIXL.jpg

[AdamThompson]

If SEOmoz is crawling all those duplicate/unnecessary URLs, the search engines probably are, too.

Best solution: Change the navigation on your site so that each page is linked to using a single URL.

[BeerCartel75]

Hi Adam,

Thanks - how do you change the navigation so each page is linked with a single URL?

Thanks

[AdamThompson]

That's a question best answered by a programmer and/or someone who is familiar with the code of your site. Sorry I can't help more!