How to crawl the whole domain?

[EduardoCoen]

Hi,

I have a website an e-commerce website with more than 4.600 products. I expect that Seomoz scan check all url's. I don't know why this doesn't happens.

The Campaign name is Artigos para festa and should scan the whole domain festaexpress.com. But it crels only 100 pages

I even tried to create a new campaign named Festa Express - Root Domain  to check if it scans but had the same problem it crawled only 199 pages.

Hope to have a solution.

Thanks,
Eduardo

[HiveDigitalInc]

Not sure, but you could try Microsoft's IIS tool to spider your site. It is possible that your site has issues that make it difficult to spider, hence why SEOMoz's bot isn't working. You could also try something like Xenu Link Sleuth or HTTrack.

[RyanKent]

Hi Eduardo.

The way crawlers work is the begin on your home page and "crawl". They look at all the links on your home page and follow each one to the next page, then the next until your whole site has been captured.

Why are only 100 pages being crawled?

Most likely either because your site is not very well linked, or because you don't have a good navigation system, or because your navigation and links are presented in a format such as flash which the crawler cannot read.

Another possibility would be if the crawler is being blocked or hindered by your robots.txt file.

[EduardoCoen]

Still have the same problem. Isn't that an issue with SEOMoz?
The domain is www.festaexpress.com has no flash and is crawled by google with no issues.

Regards,
Eduardo.

[KeriMorgret]

Hi Eduardo,

I'm sorry you're still having problems. At this point, it'd be best for you to send an email to help@seomoz.org and have our help team look at it for you. They'd be the ones who could give you the most advice for diagnosing this.

Keri

[EduardoCoen]

Hi Kery, thanks, I just sent to them.

Regards,
Eduardo.