Website pages missing from seomoz crawl

[ovieira]

Hi!

I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?

Thanks!

[Mulith]

Hi Ovieira,

This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.

The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.

Mulith

[KeriMorgret]

Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email help@seomoz.org they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.

[ovieira]

Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.

The crawler http://www.xml-sitemaps.com/  also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).

I need to solve this and don't know how?!

Thanks for your help do.

Regards.

[ovieira]

Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.

Here's my website homepage URL - http://bit.ly/TGjpVx

And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.

Regards,
OV

[ovieira]

Anyone?

[KeriMorgret]

Hi! It's probably best to email help@seomoz.org about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!

[Mulith]

Another possibility could be your robots.txt file, is it blocking some directories?

[ovieira]

I don't think so:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

# Goole Bot
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

Regards.

[ovieira]

I'll do that Keri. Thanks!

[Mulith]

Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.