Drastic Monthly Fluctuations in Page Link Metrics

[TVape]

We have experienced very drastic changes in our root domain numbers and as a result, have seen an odd impact on our DA. The data does not seem at all reliable at all. We went one month with over 50 recorded root domains and the next month that dropped over 75%.

It doesn't make sense to be paying for a monthly pro account when the data is so clearly unreliable. What is going on? Looking for a good answer before closing our account!

[DavidLee]

Hello!

Sorry to hear your metrics have fallen. While I can't provide any consulting advice I can clarify how we index sites for Mozscape.

• We grab the most recent index.
• We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
• We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).

Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index.

We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always collecting data. When we need to put the index together, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important. You can see our most recently updated schedule here: http://moz.com/products/api/updates

Mozscape focuses on a breadth-first approach. Therefore we almost always have content from the homepage of websites, externally linked-to pages, and pages higher up in a site's information hierarchy. However, deep pages that are buried beneath many layers of navigation are sometimes missed and it may be several index updates before we catch all of these.

If our crawlers or data sources are blocked from reaching those URLs, they may not be included in our index (though links that point to those pages will still be available). Finally, the URLs seen by Mozscape must be linked-to by other documents on the web or our index will not include them.

The index is based on freshness so domains with higher page/domain authority tend to be re-indexed each month but lower ranked pages can fall off or not be re-crawled if there are not consistently new links being built on high quality sites. We never re-crawl the same sites each index unless strong links take us to your site. It starts over completely from scratch.

Hope this helps and I will set this as a discussion so other community members can offer any advice.

[TVape]

thank you very much, but the answer is so technical that it does not allow me to understand much. The interesting thing from our perspective is that we had barely any links from above DA sites 70 before and we had 31 DA and now we have a lot of links from DA sites above 70 and we actually dropped.

When we posted this question, suddenly within a day many of our links appeared on your tracking system. This may indicate something happening on your end as a result of us posting this question. Many links are still missing, inlcuding one link from inc.com, which is odd.

I have also seen many sporadic changes at times in the way your metrics evaluates us. It makes it difficult to use your data for anything.

two questions that arise from the minimal amount i understood:

1- inc.com is a very high DA site, why is this link not indexed yet? http://www.inc.com/jessica-stillman/5-weird-ways-to-de-stress.html

2- why did a large number of our links get detected right after we posted this question?

thank you very much in advance,

Nima

[TVape]

BTW on our webmaster tools we have 226 linking root domains. on moz we have 68.

our consistent drop in DA seems to be a factor of Moz not crawling properly. I remember back in the day the discrepancy between WMT and moz was less.

Can you please shed some light? we use moz primarily as a performance measurement software (if it can be called that), but if the data is unreliable that will make it difficult for us to do that.

I am still looking forward to your trouble shooting advice.

Kind regards,

Nima

[DavidLee]

Hi Nima

Google will always show more links than our crawler as they are a much larger company and have multiple seeds to crawl more pages. You should use multiple tools to obtain a full link profile which is something I've seen recommended many times in our community. The 68 links we found mean those links meet the criteria we are measuring to discover them. This also means the 158 other links may not be considered as high ranking sites or your links are too deep for us to crawl or they have been crawled and are just waiting to appear in a future index update. No single index will be the same as each company crawls differently.

As I mentioned earlier:

• We grab the most recent index
• We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
• We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).

Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index. Google does not crawl this way. This and our metrics are proprietary to Moz so it isn't that our data is unreliable or is being crawled incorrectly.

The domain authority is effected by a lot of things. It is hard to pin point it without being a SEO consultant or the specific web designer on your website. Domain and Page Authority scores are both calculated using Moz's Ranking Models work. In essence, we take a lot of rankings data from the search engines (by running queries) and then try to build a predictive scoring system using our own on-page analyses and Mozscape link data to construct an algorithm that will effectively reproduce the search engines' results. Our current accuracy hovers in the 70% range, but over time, we expect to improve.

Once we have a ranking model (which we internally call "uber"), we can create scores that best approximate the combinations of all our page-specific link metrics or domain-specific link metrics (removing the keyword-specific features like anchor text, on-page keyword usage, etc). These scores represent the model's query-independent or non-keyword-based ranking inputs.

In simple terms, Domain Authority is our best prediction about how content would perform in search engine rankings on one site vs. another. Page Authority answers the same question for an individual page. Both are amalgamations of all the link metrics (number of links, linking root domains, mozRank, mozTrust, etc.) we have into a single, predictive score.

It's important to note that both Domain Authority and Page Authority are on a 100-point, logarithmic scale. Thus, it's much more difficult to grow your score from 70 to 80 than it would be to grow from 20 to 30.

Here's some places to really delve into what is going on:
http://moz.com/blog/whiteboard-friday-domain-trust-authority
http://moz.com/blog/googles-algorithm-pretty-charts-math-stuff
http://moz.com/blog/whiteboard-friday-domain-authority-page-authority-metrics

Here are some good resources to help you take a look at the factors.
http://moz.com/blog/whiteboard-friday-domain-authority-page-authority-metrics
http://apiwiki.seomoz.org/w/page/20902104/Domain Authority
http://moz.com/blog/whiteboard-friday-domain-trust-authority

I would recommend starting a new thread on the forum to seek advice from other marketers as I am only able to explain how our tools work from the technical side.

Hope this helps!