605 : Page banned by robots.txt

[bhomes]

Hello everyone,

I need experts help here, Please suggest, I am receiving crawl errors for my site that is , X-Robots-Tag: header, or tag.

my robots.txt file is:

User-agent: *

Disallow:

[VERBInteractive]

Hi Bhomes,

Try clearing you robots.txt of any content, a robots.txt with:

User-agent: *

Disallow:/

Is blocking everything from crawling your site. See: https://support.google.com/webmasters/answer/6062598?hl=en for testing and more details on robots.txt

[bhomes]

I am receiving crawl error for moz only.

There is no error at google's search console. Also, I have tested at google's robots.txt testing tool. https://www.google.com/webmasters/tools/robots-testing-too

My robots.txt file is with no slash.

User-agent: *
Disallow:

[VERBInteractive]

Are you able to provide a link to site (DM me if you don't want it posted on the forum)

[LoganRay]

Hi,

This sounds like it's more related to the meta robots tag, not the robots.txt file.

Try this:

• Run a Screaming Frog crawl on your site
• Once complete, go to the Directives tab
• Look for 'NoIndex' in the 'Meta Robots 1' column (should be the 3rd column)
• If you see anything marked with that tag, remove them - unless of course you need them there for a reason, in which case you should also block that page in your robots.txt file

[bhomes]

Hi Logan Ray

thank you for detailed guide, all tools bot are working perfectly except moz's. My robots meta is index, follow and my robots.txt is disallow for none for all user agents. Still there is confusion that why moz is showing crawl error. I have now emailed to moz. Let's see what they reply. I will share that.

thank you

[LoganRay]

I actually have come across a handful URLs that are NoIndex, I'll DM you a list once complete.

I can't be certain this is the root of the problem (I've never seen this error in the crawl report), but based on the error you said you're getting, I believe it's a great starting point.

[VERBInteractive]

I am at a lost, I can't find the issue. Let us know what Moz says.

[ChiarynMiranda]

Hey there! I just followed up on the message you sent into our help team, but I wanted to also post the answer here for reference.

It looks like the robots.txt file may have recently been changed for the site because I created a new campaign for the subdomain and I am not getting that same error. You should no longer see this error on your next campaign update or you could create a new campaign and you would no longer see the error there.

I did notice that you ran a number of crawl tests on the site since the campaign update, but the important thing to realize is that the crawl test can be cached for up to 48 hours. (I removed the crawls in this version of the screenshot for privacy.) We also cache the crawl tests from campaign crawls, so it looks like the first crawl test you ran on the 29th was cached from your campaign crawl and the two subsequent crawl tests were cached from that first crawl test.

Again, I wanted to note that it looks like there are only links to about 2 other pages (terms and privacy) that are on the specific subdomain you are tracking, so we aren't able to crawl beyond those pages. When you limit a campaign to a specific subdomain, we can only access and crawl links that are within the same subdomain.