Enhancement

[dt1807]

SEOMoz,

I would like to know if you are considering to update your crawler so that it discounts links which don't have a relevancy rating with Google. I'm doing competitive analysis, and I still see a large amount of links that are meaningless in 2012/13 best practices.

Wondering if you are tackling this issue?

Thanks

[Dr-Pete]

Broadly, yes - we're very interested in this issue. Unfortunately, it's incredibly complicated. A couple of key points:

(1) We have no idea which links Google discounts. There's no API or signal or crawl trick that will tell us, and there's really no way to measure the impact of any single link. So, we're forced to try to deduce the signals they use. Unfortunately, even those signals can take a mountain of data to process, and so it gets computationally more and more complex.

(2) Our crawl does bias toward what we believe are higher-quality links. In other words, we try to focus on quality over quantity in our data. So, in that sense, we do discount them, to some degree. This is mostly based on authority-based metrics, though, not on "spamminess".

(3) Dr. Matt Peters here at SEOmoz is actively working on a project to classify link spam and try to emulate Google's behavior. That's been a very difficult project, but the goal is to measure spam more effectively and eventually try to integrate what we learn back into the crawl. I can't currently give you an ETA on that project, but it is something the Data Science team is working hard on.