What do you add to your robots.txt on your ecommerce sites?

[ThomasHarvey]

We're looking at expanding our robots.txt, we currently don't have the ability to noindex/nofollow. We're thinking about adding the following:

• Checkout
• Basket

Then possibly:

• Price
• Theme
• Sortby
• other misc filters.

What do you include?

[LoganRay]

Hi,

It sounds like you're going down the right path. Disallow and section of the site that has personal information, as there's no value in having bots crawl that, keep them on important content longer! In addition to Checkout and Basket/Cart, you should also disallow the My Account area if your site has one.

Your next grouping, I'm assuming these are the parameters by which you pages can be sorted. If so, yes, disallow all of those, they're only going to cause duplicate content flags for you in the future. I'm not sure which CMS you are using, but some eComm platforms also have 'email to a friend' URLs that are a major source for dupes and can often be identified and disallowed by another parameter.

Hope this helps narrow it down for you!

[Deacyde]

I'm on this same path since we too cannot use noindex / nofollow due to limited backend interaction with Bigcommerce.

I like to block all cart related pages, which for ecommerce sites can be a boat load.

• /cart.php
• /checkout.php
• /finishorder.php
• /*login.php

just to name a few, then you have the sorting and compare pages, they have to be blocked or a mess unfolds.

• Disallow: /*sort=newest
• Disallow: /*sort=bestselling
• Disallow: /*?page=       ( Big duplicate page issue if you don't block this one with a wildcard, and cannot access your .htaccess file or the backend properly to noindex / nofollow )

Just to name a few, in my case, I only want the meat of the site to be indexed and rank for. Otherwise one client's site was ranking terms that more related to web development than the niche industry they lived in. Plus with a limited index budget, why would you want google or anyone else to crawl pages on your site with no SEO value towards your niche?

Unless you sold carts as in web developed carts for ecommerce sites you wouldn't want much of that indexed anyways, and even in that case, those pages aren't too useful for ranking. At least from what I've gathered in the niche industries.