Disallow: /search/ in robots but soft 404s are still showing in GWT and Google search?

[esiow2013]

Hi guys, I've already added the following syntax in robots.txt to prevent search engines in crawling dynamic pages produce by my website's search feature: Disallow: /search/. But soft 404s are still showing in Google Webmaster Tools. Do I need to wait(it's been almost a week since I've added the following syntax in my robots.txt)? Thanks, JC

[intSchools]

You will probably need to wait a little longer - it depends how often your site usually gets crawled and indexed.

However, robots.txt does not always stop search engines from indexing your pages. It will stop them crawling a page on your site but it tells them that they can still index that page. If they find links from external sites then the URL may still appear in the SERP.

[esiow2013]

Is the time issue related in crawl frequency of the URLs in my sitemap?

Thanks Dan, appreciate it.

[intSchools]

That is a good recommendation but ultimately search engines will make a final decision on crawl frequency. Take a look at your 'Crawl Stats' on GWTs and this will give you an idea of how often your site is crawled.

[esiow2013]

Thanks a lot Dan!

[intSchools]

Glad to help

[timhatton]

You could also look at using the meta robots = noindex tag on /search/ pages, rather than just blocking it in robots.txt, as this will remove existing URLs from the index.