Do I need to disallow the dynamic pages in robots.txt?

[esiow2013]

Do I need to disallow the dynamic pages that show when people use our site's search box? Some of these pages are ranking well in SERPs.  Thanks!

[Andy.Drinkwater]

If they are ranking well, what is the reason for wanting to block them?

Andy

[esiow2013]

The reason why we want to block those pages is because they produce soft 404 errors. What should we do? Thanks Andy.

[KevinBudzynski]

I agree with Andy. Many of our search result pages rank well (and actually convert quite well). I don't think you need to disallow them unless it's for content that doesn't exist. Even at that time, you may still want them up because you may offer complementary products and etc.

[Andy.Drinkwater]

OK so when you search, you get back dynamic pages that are producing 404's, but you see the pages in the SERPs?

Just want to make sure i have this right

[esiow2013]

Yeah, some of them produce soft 404 since there's no content at all but some dynamic pages that rank well show content.

Thanks,

JC

[Andy.Drinkwater]

Well, I would try and fix why they are returning 404's as it would be a shame to block all results. Is this something you can do? Or is the a reason why just blocking is preferred?

[esiow2013]

These soft 404s produce 200 status code. We already improved our pages when someone finds a product that is not on our list. But then, these dynamic pages are still considered as soft 404s by Google webmaster tools.

[Andy.Drinkwater]

Just having a quick look what Google say about them:

Here’s a list of steps to correct soft 404s to help both Google and your users:

1. Check whether you have soft 404s listed in Webmaster Tools
2. For the soft 404s, determine whether the URL:
   1. Contains the correct content and properly returns a 200 response (not actually a soft 404)
   2. Should 301 redirect to a more accurate URL
   3. Doesn’t exist and should return a 404 or 410 response
3. Confirm that you’ve configured the proper HTTP Response by using Fetch as Googlebot in Webmaster Tools
4. If you now return 404s, you may want to customize your 404 page to aid your users. Ourcustom 404 widget can help.

Have you followed these steps?

Andy

[esiow2013]

Yes, done that.

[esiow2013]

These pages that produce soft 404 errors don't show products at all because these people search for our products that are not available.