Magento CE 1.9.1 - Robots.txt help please

[iamgreenminded]

This post is deleted!

[iamgreenminded]

Here is a snippit of my duplication content report issues, literally 3000+ duplicate content from MOZ report - seeking a robots.txt to help with this

thank you

<colgroup><col width="549"></colgroup>
| https://www.iamgreenminded.com/customer/account/login/ |
| http://www.iamgreenminded.com/catalogsearch/result/?q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/?q=Germ+Warfare |
| http://www.iamgreenminded.com/catalogsearch/result/?q=iamcleanminded |
| http://www.iamgreenminded.com/catalogsearch/result/?q=SKINERGY |
| http://www.iamgreenminded.com/eye-minerals-blues-greens |
| http://www.iamgreenminded.com/eye-minerals-peach-yellow |
| http://www.iamgreenminded.com/hint-lip-balm |
| http://www.iamgreenminded.com/cherry-chaos-lip-balm |
| http://www.iamgreenminded.com/so-happy-lip-balm |
| http://www.iamgreenminded.com/sendfriend/product/send/id/673/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvNjczLz9fX19TSUQ9VSNyZXZpZXctZm9ybQ,,/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/438/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvNDM4Lz9fX19TSUQ9VSNyZXZpZXctZm9ybQ,,/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/1/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvMS8_X19fU0lEPVUjcmV2aWV3LWZvcm0,/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/476/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvNDc2Lz9fX19TSUQ9VSNyZXZpZXctZm9ybQ,,/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/180/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvMTgwLz9fX19TSUQ9VSNyZXZpZXctZm9ybQ,,/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/53/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvNTMvP19fX1NJRD1VI3Jldmlldy1mb3Jt/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/52/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvNTIvP19fX1NJRD1VI3Jldmlldy1mb3Jt/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/51/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvNTEvP19fX1NJRD1VI3Jldmlldy1mb3Jt/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/32/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvMzIvP19fX1NJRD1VI3Jldmlldy1mb3Jt/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/26/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvMjYvP19fX1NJRD1VI3Jldmlldy1mb3Jt/ |
| http://www.iamgreenminded.com/sendfriend/product/send/id/7/ |
| https://www.iamgreenminded.com/customer/account/login/referer/aHR0cDovL3d3dy5pYW1ncmVlbm1pbmRlZC5jb20vY2F0YWxvZy9wcm9kdWN0L3ZpZXcvaWQvNy8_X19fU0lEPVUjcmV2aWV3LWZvcm0,/ |
| http://www.iamgreenminded.com/thebirdword/category/make-up/iamgreenminded-com/ |
| http://www.iamgreenminded.com/thebirdword/category/make-up/iamgreenminded-com/germ-warfare/ |
| http://www.iamgreenminded.com/thebirdword/category/make-up/ |
| http://www.iamgreenminded.com/thebirdword/tag/acne/ |
| http://www.iamgreenminded.com/thebirdword/tag/noncomedogenic/ |
| http://www.iamgreenminded.com/thebirdword/tag/zerocomedogenic/ |
| http://www.iamgreenminded.com/thebirdword/tag/makeup-brushes/ |
| http://www.iamgreenminded.com/thebirdword/tag/mineral-makeup/ |
| http://www.iamgreenminded.com/thebirdword/tag/germ-warfare-2/ |
| http://www.iamgreenminded.com/thebirdword/tag/midnight-magic-2/ |
| http://www.iamgreenminded.com/thebirdword/tag/natural/ |
| http://www.iamgreenminded.com/thebirdword/tag/vegan/ |
| http://www.iamgreenminded.com/thebirdword/tag/zit/ |
| http://www.iamgreenminded.com/thebirdword/category/health/sleep/ |
| http://www.iamgreenminded.com/thebirdword/category/smartphone/ |
| http://www.iamgreenminded.com/thebirdword/tag/blue-light-2/ |
| http://www.iamgreenminded.com/thebirdword/tag/melatonin/ |
| http://www.iamgreenminded.com/thebirdword/tag/orange-glasses/ |
| http://www.iamgreenminded.com/thebirdword/tag/sleep-2/ |
| http://www.iamgreenminded.com/thebirdword/tag/smart-phone/ |
| http://www.iamgreenminded.com/catalogsearch/result/index/?dir=asc&order=relevance&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?mode=list&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?p=2&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?p=3&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?cat=8&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?cat=9&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?cat=10&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?cat=11&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?cat=12&q=Ella+Rose+Minerals |
| http://www.iamgreenminded.com/catalogsearch/result/index/?dir=asc&order=relevance&q=Germ+Warfare |
| http://www.iamgreenminded.com/catalogsearch/result/index/?mode=list&q=Germ+Warfare |
| http://www.iamgreenminded.com/catalogsearch/result/index/?cat=7&q=Germ+Warfare |
| http://www.iamgreenminded.com/catalogsearch/result/index/?cat=8&q=Germ+Warfare |

[Carson-Ward]

Definitely don't add that line to your robots.txt. I think it could have unintended consequences, like your entire site dropping off Google search.

Let me also say not to worry too much about duplicate content. It's good to avoid, but unlikely to cause big problems. That said, I would probably disallow site searches, because it can look to Google like you're trying to rank for all types of keywords with the same products. Use categories instead to show up for product types.

Disallow: /catalogsearch/result/

Because these pages all appear the same, you don't necessarily want Google to index the pages each time:

Disallow: /sendfriend/

If everything in this folder is behind a login wall:

Disallow: /customer/account/

And it's just my personal preference to avoid using tags on posts unless I want to tie a recurring concept together. 99% of the time people aren't using tags to any real benefit. You could also disallow them if you wanted, but I'd rather you just not use them until your content section is larger and you've figured out topics that people would actually want to jump to.

Let me also add that it'd be better to noindex these pages, but that's a little bit more difficult to do with Magento. If you have the skill or a developer who can do it quickly, I'd noindex these pages instead, and then down the line add them to the robots.txt. If that's difficult, though, you will be fine by simply blocking the content with robots.txt

Hope that helps - let me know if there are any other concerns.

[iamgreenminded]

Thank you, it helps

[Guest]

This post is deleted!