Feedburner Links Showing Up In OSE

[MiguelSalcido]

I, for years now, have been seeing lots of links from http://feeds.feedburner.com/ showing up very high in backlink profiles.

Why is this?

And if these links are not passing link juice why are they being reported on in OSE?

Seems like there should be some sort of filter for these, or that OSE bot should follow robots.txt directives as googlebot does.

[randfish]

We do follow robots.txt, but Feedburner actually allows and enables crawling of many of these, so we crawl them. There's a robots.txt for http://feedburner.google.com/robots.txt but the one for feeds.feedburner.com, feeds2.feedburner.com, and feedproxy.feedburner.com only exclude items as such: http://feeds.feedburner.com/robots.txt

I'm not sure if Feedburner intended to block all crawling/indexing of content on their domain with the disallow /~a/. Robots.txt. Google doesn't appear to support the tilde ~ in their specs: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt, though other resources indicate it would only show things relative to the /a/ directory. Certainly all the apps and services that crawl Feedburner feeds in order to show the blog posts and content would be disallowed if Feedburner did block everything there. Google seems to be crawling plenty of stuff there, too: https://www.google.com/search?q=moz+site%3Afeedburner.com

For this reason, we try to show these in OSE. It's very hard to know if these URLs do or don't pass link value directly, but the original source almost certainly does... Definitely one of the tougher special-cases out there. Many site owners/webmasters/marketers want to see these links,  so we've biased to displaying them.

[MiguelSalcido]

Dear Wizard of Moz,

Outstanding answer! Thank you very much.

I wonder if anyone can figure out whether or not they pass value?