Specific external links

[Greatmats]

So...we have an A+ rating with the BBB and have for some time.  We have a link from their page.  We also use sites like pricegrabber and reseller rankings and have links from those.

What I can't figure out is why links from the BBB and Pricegrabber don't even show up in open site explorer but the reseller ratings links are everywhere.  The BBB one is easily the oldest and the other two are roughly the same age (i.e. at least a couple years old.)

[HealthLotteryMan]

Hi,

Firstly it could be that Open Site Explorer has not had time to pick them up, if it's years old then let's not forget the OSE manually crawls the internet and is not Google's crawler meaning there will be sections of the web which won't be crawled by Moz.

If it's old then there may be no links pointing to these pages which would not give OSE a reason to crawl them, certainly if it's years. Don't worry about it, a missing link on OSE is one less your competitors can find I had a similar issue with a few links from BOTW but in time it updated and works fine but Open Site Explorer is not flawless.

[Oren.]

Perhaps a Mozzer can verify this, but I remember someone saying they Moz doesn't crawl some big sites that aggregate lots of data (twitter, BBB, yelp, domain listings, etc. etc.) because of the size of their servers, and the likelihood that the user put it there themselves.  They have smaller servers than say, Google or Bing.

[HealthLotteryMan]

I wonder if Moz understands and follows the Robots.txt file? I know at one point they never, back in the day when I used Moz most of my errors came from URL's which were hidden by robots which would not of caused errors for Google.

& what your saying Oren makes perfect sense if it's correct. Why would they want to crawl aggregated content.

[Harbor_Compliance]

I share your frustration in wanting to be able to track links in OSE.  Someone at Moz told me a while back that OSE only reaches 20-25% of links out there.  I've just come to tell myself that if I know the link exists then I don't need to see it in OSE.

[Cyrus-Shepard]

The Open Site Explorer indexed is completely refreshed every few weeks, and currently consists of 105 billion links. While that's quite a large data set and mostly comparable with other indexes in the space, it's still only a fraction of the links on the web and only a portion of the links Google undoubtedly knows about.

The way OSE finds links is that it prioritizes pages with lots of inbound links and links it found during previous crawls. Because BBB pages are often found deep-deep-deep under layers of navigation and usually don't have a lot of external links pointing to them, they aren't easy pages for crawlers to find on a regular basis. BBB has an especially difficult site architecture, and sadly this issue comes up a lot with that particular site

We're working on including more historical data in the index, so that even if we didn't crawl the link in the past two weeks, we'll still include it if we are reasonably confident it still exist. This will help alleviate some of these concerns.

The good news (or bad depending on how you look at it) is that while OSE doesn't show every link, it tends to show the majority of the most valuable links.

Regardless, thanks for the feedback and best of luck with your SEO.