Why do i get multiple variations of my url with ?order=asc and ?view=list at the end of it in my crawl report?

[alexrbrg]

I just did a crawl for one my clients to validate any error in the structure. Next thing I know is that the website have multiple variation of the same url with query like ?order=asc and ?view=list at the end of it.

I am wondering why these url variations appears in the crawl I just did since bots aren't suppose to go further thant the ? normally.

Just to show you a couple of url's of my crawl test.

<colgroup><col width="484"></colgroup>
| https://test.com/exemple/?per_page=9 |
| https://test.com/exemple/?per_page=15 |
| https://test.com/exemple/?per_page=30 |
| https://test.com/exemple/?orderby=popularity |
| https://test.com/exemple/?orderby=date |
| https://test.com/exemple/?orderby=price |
| https://test.com/exemple/?orderby=price-desc |
| https://test.com/exemple/?order=asc |
| https://test.com/exemple/?order=desc |
| https://test.com/exemple/?view=list |

Thank you Guys

[samantha.chapman]

Hey there!

Sam from Moz's Help Team here! As far as I'm aware, Google and other crawlers do crawl past the '?', unless certain parameters are disallowed within the robots.txt. If the URL is: https://test.com/exemple/?per_page=9, a search engine will see something like test com exemple 'search' etc. Google recommends blocking all Internal Search Results in the Robots.txt file - for Rogerbot, it would look something like this

User-agent: Rogerbot
Disallow: ?utm

Here is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt

I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises

[alexrbrg]

Thank you Samantha your answer is very useful !