Zendesk robots.txt

[zspace]

Hello!

We have a Zendesk support site at support.zspace.com - our Moz crawl report is saying that there is 85 temporary redirect issues, mostly coming from our support site.

My question is, does the Moz crawler use robots.txt?

We have currently Disallow: /search in our robots.txt, but not /search/*

Most of the temporary redirect URL's in the crawl error report look like this...

https://support.zspace.com/hc/en-us/search/click?data=BAh7CjoHaWRp

So would we need to add "Disallow: /hc/en-us/search/*" to get Moz crawler to ignore these?

[Yoav-Blustein]

Hi

Mozs crawler aka the RogerBot does obeys the robots.txt and Moz states that under the help section here.
Looks like the problem you are having is connected to your robots.txt statements/directives.
In order to block a directory **and its content,**you need to use the category name following with a forward slash (even if you have a redirect from xxx.com/search to xxx.com/search/)
(Check out Google robots.txt guidelines)

if you want to block the directory "search" and its content, use Disallow : /search/ (no need for asterisk at the end)
if you use "Disallow: /hc/en-us/search/" (again, no need for asterisk) you will block all the content under xxx.com/hc/en-us/search/
So, for example if you have content that you want to block under xxx.com/hc/fr-FR/search/ it will not be blocked because your statement/directive limits it to a very specific "search" directory which is located under "en-US".

What is it that you want to block exactly?

[zspace]

Thanks Yossi, this is kind of what I expected I think.  I guess the question should have been "has anyone had Moz crawl issues with their Zendesk support site"?

The main issue with our support site is that Zendesk does not allow access to the robots.txt file so there is no way to add regular expressions like the wildcard search/* to it.

I will re-post the question as above.