Whats the best way to remove search indexed pages on magento?

[SeoMartin1]

A new client ( aqmp.com.br/ )call me yestarday and she told me since they moved on magento they droped down more than US$ 20.000 in sales revenue ( monthly)...

I´ve just checked the webmaster tool and I´ve just discovered the number of crawled pages went from 3.260 to 75.000 since magento started... magento is creating lots of pages with queries like search and filters. Example:

1. http://aqmp.com.br/acessorios/lencos.html
2. http://aqmp.com.br/acessorios/lencos.html?mode=grid
3. http://aqmp.com.br/acessorios/lencos.html?dir=desc&order=name

Add a instruction on robots.txt is the best way to remove unnecessary pages of the search engine?

[bjs2010]

Hi,

I am quite familiar with Magento and struggling with the SEO of this ecommerce mammoth!

As far as I am aware, you should implement the meta tag "NOINDEX, FOLLOW" on those pages that you do not want indexed - as your pages are already in the index, this is the way to go - blocking them on robots.txt does not get pages out from the index if they are already in there.

I suggest you apply some "querystring" logic to your template - you will find the page here:
app/design/frontend/default/YOURTEMPLATE/template/page/html/head.phtml

That way, you can apply the

depending on the page content.

Hope this helps you and let's stay in touch about Magento! (PM me)

[tyrrell123]

Hi,

can you expand on this and point me in the right direction if possible please BJS1976? I have the same problems too as originally asked by 'SEO Martin'.

I have obviously seen that the Yoast_MetaRobots plugin is recommended by others when searching for a solution to noindexing the non-content pages (search results, filters etc). However I am very reluctant to install this as many people who have tried said it has broken their sites.

If there is another way of implementing the noindex, follow meta tag, I would be very greatful to know how as like you I am really struggling to with this one.

Many Thanks

[bjs2010]

Hi Ian,

You are right in that Yoast Meta Robots can be cranky - I installed it and had to play around with it to get it working.

However, it does offer a very nice feature that I think is worth it - you can apply various combinations of Meta Robots directives to product pages individually - so this adds more value than just being able to do NOINDEX on reviews, wishlists, etc... pages. But install it on your dev site before trying it live.

So my solution uses both Yoast and my custom code - you check the URL for any querystrings, such as ?manufacturer etc... and apply different logic according to what you wish to be indexed or not.

Feel free to PM me.

[SeoMartin1]

What do you think about Google URL parameters? http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1235687

[bjs2010]

I have tried using them and didn´t do anything - furthermore, if you check this video out by Google themselves, you will find that using these parameters is a "hint/suggestion" as opposed to a solid directive.
http://www.youtube.com/watch?v=DiEYcBZ36po

Rel Canonical is also a hint.

But Meta Noindex,follow is a solid directive which they have to pay attention to.

Hope that helps - been there, done it got the t shirt through a lot of pain and frustration!