Blocking pages from Moz and Alexa robots

[Pushm]

Hello,

We want to block all pages in this directory from Moz and Alexa robots - /slabinventory/search/

Here is an example page - https://www.msisurfaces.com/slabinventory/search/granite/giallo-fiesta/los-angeles-slabs/msi/

Let me know if this is a valid disallow for what I'm trying to.

User-agent: ia_archiver
Disallow: /slabinventory/search/*

User-agent: rogerbot
Disallow: /slabinventory/search/*

Thanks.

[effectdigital]

That looks valid to me. It's possible you may not need "*" at the end of each rule but I can't see it doing any harm either

I might go more like:

User-agent: ia_archiver
Disallow: /*/search/

User-agent: rogerbot
Disallow: /*/search/

^ this would stop all search URLs being indexed, so even if you introduced new search facilities later in other directories - they would 'probably' be caught too (assuming that is your intention, assuming they were still in /search/ subdirs)

Don't think what you have done is wrong though.

Always check using Google's robots.txt tester to be safe. Just put your rules into the tester (altering them to be used for all user-agents), and try out some different URL patterns. When it works as you like, update your real robots.txt file (remembering of course, to restore your rogerbot / alexa UA targeting - if you don't want the rules to also apply to Google!)

[Xiano]

Hi,

Firstly, yes, that robots.txt is valid and would work for your purpose.

There's a great tool (https://technicalseo.com/tools/robots-txt/) that allows you to put in your proposed robots.txt file contents, the URL you want to test and even the robot you want to test against and it lets you know the result.