Moz crawler is not able to crawl my website

[Harini.M]

Hello All,

I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "

We changed the robots.txt file and checked it . but still the issue is not resolved.

URL : https://www.khadination.shop/robots.txt

Do let me know what went wrong and wjhat needs to be done.

Any suggestion is appreciated.

Thank you.

[andy.bigbangthemes]

Hi there,

You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.

Allow: /

add this on line 2 of your robots.txt file.

Good luck

[SMRBathrooms]

Hi - As Andy has said, you're not allowing Moz to crawl the site.

Read up on Rogerbot here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot

[Harini.M]

Thank you Andy . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?color=
Disallow: /?manufacturer=
Disallow: /?filter_material-fabric=
Disallow: /?filter_color=
Disallow: /?query_type_color=
Disallow: /?filter_size=
Disallow: /?taxonomy=
Disallow: /?view_mode=
Disallow: /?query_type_material-fabric=
Disallow: /?orderby=
Disallow: /?source_id=
Disallow: /?source_tax=
Disallow: /?shop-2__trashed?
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.khadination.shop/sitemap.xml

this was the previous version of robots.txt that were been used ....

[jocameron]

Hi Harini,

Jo from the Moz help team here.

I've had a look at your site and it looks like there is something server side that is blocking our bot.

When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1

https://screencast.com/t/J3hhDTCM

I'm also seeing this message in this third party tool.

"The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"

All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.

I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2

You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2

You'll also want to make sure you are not blocking AWS (Amazon Web Services).

Best of luck!

Jo