Why Open Site Explorer is Showing 0 Total Internal Links?

[DavidLee]

Hi Anirban!

OSE crawls differently from our diagnostics crawler so it doesn't quite follow the link structure within the source code unless we can reach the page through other important sites that are linking to you.

Just so you know, here's how we compile our index:

• We grab the most recent index.
• We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
• We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).

Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index.

Other times can be re-directs and javascript and server errors that would prevent access to the site. If you can tell me your domain name I would love to take a look.

Talk to you soon!

[vivekrathore]

Hi David

Thanks for the help. Domain name is www.shiksha.com. If you can look and advise.

[DavidLee]

Your robots.txt is blocking us with the wildcard directive at the very bottom.

You will want to add:

User-agent: dotbot
Disallow:

This should update when we release a new index next month.

Hope this helps!

[vivekrathore]

Thanks David.

I hope this helps.

[DavidLee]

Anytime!

Cheers

[BlooSEM]

Hi David,

I have the same problem with the internal links. site is sparta.nl

[DavidLee]

Hello!

For sparta.nl it appears we haven't fully crawled the https site but have previously crawled non-https which now have 301 re-directs to the new URL. You should see your internal linking numbers increase over the next few crawls as it can take time for the metrics to change over.

[TravG123]

Hello,

My website www.onlineslotshub.com has the same problem as above. I looked at my robots file and that does not seem to be the problem. Could you take a look and let me know if you see what would be causing this for my website?

hanks - Travis

[DavidLee]

Hi Travis

I am seeing No Data for the HTTP status for your top pages so while we can find external links to your page, we have not indexed your site yet but it is processing and you should see your site indexed after our next update on 10/27.

Hope this helps!

[bettingfans.com]

Maybe outdated but we do have the same situation for https://www.voetbalwedden.net

@travis: can you take a look on our campaign?

[DavidLee]

Hi arb3ux!

It appears your site/server is blocking our user-agent when we start from http://, we receive no response when we attempt to crawl as rogerbot or dotbot

I can reach your site directly by going to https://

So there is a configuration preventing us from reaching http to be re-directed to https

I recommend reaching out to your hosting provider to help isolate the issue. Please let me know if they make any specific changes and I can re-run the test to see if future crawls will be able to index your links.

[brandfield]

This post is deleted!

[chloeczhashemi]

Hello, I am seeing the same problem for the following domain: https://ccsadoption.org/

Could anyone provide any further info on this issue? I can't see any robots.txt issues...

Thanks,

Chloe

[DavidLee]

Hi Chloe

It's possible that your site's authority is too low for us to crawl your internal pages. The Crawl delay in robots could also be preventing us from crawling.

You will want to keep link building as currently there are no new links being discovered which means we won't revisit your site until we can find quality links from other sites: https://moz.com/researchtools/ose/just-discovered?site=ccsadoption.org%2F&filter=&source=&target=domain&page=1&sort=crawled

Hope this helps

[bijouxusa.com]

David,

Same here. (http://www.bijouxusa.com)

Our page authority shows 34

I am seeing Total Internal Links and Internal Equity-Passing Links as 0.

Is this any actionable information?

Thanks for your help

[DavidLee]

Hello!

Your site does not have enough links or authority for us to crawl.

https://moz.com/researchtools/ose/pages?site=http%3A%2F%2Fwww.bijouxusa.com&sort=page_authority&filter=all&page=1

Keep link building to all pages on your site and on sites that are frequently visited with high authority which can help increase your score.

[willscott]

Hi David,

I'm having the same issue with the following domain: http://trahanarchitects.com/

It's saying the site has 0 internal links but we know that is not true. Our external links are registering but not our internal links. Please advise.

[DavidLee]

Hi Will

Sorry to hear we have not crawled your site.

There are currently no new links going to any page on the site that will send us back for possible crawling. Also many of the pages indexed are returning 404s.

https://moz.com/researchtools/ose/pages?site=http%3A%2F%2Ftrahanarchitects.com%2Fwork%2Fholy-rosary%2F&sort=page_authority&filter=all&page=1

https://moz.com/researchtools/ose/just-discovered?site=http%3A%2F%2Ftrahanarchitects.com%2F&filter=&source=&target=domain&page=1&sort=crawled

To stay in an index and potentially have metrics calculated, you want to have continuous and fresh links going to the site to active pages. It most likely that the external links were captured over a span of time within a 160 day window but not enough close together to deem your site important to crawl. These are both the most common reasons we may index links but not crawl.

Hope this helps!

[willscott]

David,

Thank you for your prompt response.

However, I know that http://trahanarchitects.com/ has in fact obtained several external links (around 20) all pointing back to the homepage in the past 160 days. So are you saying in order for Moz to crawl our internal links (which it's incorrectly has indexed at 0) we should be obtaining external links for more than just our homepage? It sounds to me like the issue is our external links aren't diverse enough.

I also found another discrepancy when looking at a competitors website. According to the Moz tool, http://www.eskewdumezripple.com/ has 144,000 internal links, but there is no possible way this is correct. According to Google http://www.eskewdumezripple.com/ only has 610 pages indexed. That's an insane number of internal links for such few pages. And when looking at the website you can tell just by looking that there aren't 144k internal links. Please advise.

[DavidLee]

The links should be more diverse and the external pages are being link to from diverse links and so forth and so forth. There is no exact number of links I can say that would allow our crawler to crawl the links, as it can depend on many factors.

For http://www.eskewdumezripple.com, we are indexing links to multiple category/tag URLs that are contributing to the link count for example:

http://www.eskewdumezripple.com/studio-life/archives/2014/05
http://www.eskewdumezripple.com/studio-life/blog/by_tag/tag/community+engagement

You can use pagination to see what type of links are being picked up here: https://moz.com/researchtools/ose/pages?site=http%3A%2F%2Fwww.eskewdumezripple.com%2F&no_redirects=0&sort=page_authority&filter=all&page=1

HTTP status will indicate wether or not we crawled the page (200 status or No Data).

Hope this helps