How Is a Page Crawled by Moz When Moz Says 'No Links'?

[xerox432]

As above, really. I've crawled a new client's site to find the Moz crawler has identified a handful of 404 errors.

The Moz crawler says these pages have '0 linking domains', and OSE has no data for these pages.

So how are these pages being crawled by Moz and what should I advise my client?

[crackingmedia]

Hi, if you put site:xxx.com into Google where xxx.com is your client's domain name, does Google report anything?

Peter

[KeriMorgret]

We have two completely different crawling systems. We have on crawler for your website, where you enter in your URL and we crawl your site looking for problems. We have a different crawler for OSE that goes and looks at the web in general and crawls for links. The two crawlers do not talk to each other, so it is actually understandable that OSE may not have data on the site. OSE doesn't have the same index size as Google, so we don't see all the links that are actually out there (though we're working all the time on improving this!).

[xerox432]

Hi Peter,

I searched for their domain name and about the expected number of results came up.

I searched for these individual URLs that Moz is reporting using the site: command, and they were not returned.

They are also neither in the site's sitemap (HTML page or XML file).

So how is SEOmoz Crawler finding them?

[xerox432]

Hi Keri,

Thanks for getting back to me (thanks to Peter above, also).

So what is the best way of me finding where these 404 pages are linked from?

I am 99% sure they are internal links that I will ask the client to remove, but would like to avoid manually checking every link on the site if possible.

[KeriMorgret]

If you download the CSV report, we list the referring page there. We know it isn't obvious and we're working on making this a little easier to find.

[crackingmedia]

Hi, no problem. Keri has answered your query re the Moz crawler not finding them.

Peter

[xerox432]

That's great, thanks Keri.

[xerox432]

Thanks again, Peter.