Welcome to the Q&A Forum

Yes, you can set it up like this: Disallow: /user/ Allow: /user/password/ And that should do it!

Does that mean that they're likely going to be using the beta version with noindex, nofollow at a later point, or is that just a temporary thing. We're doing very well in organic listings right now, but we have Thomasnet for a client as well.  Trying to decide whether or not to keep them.  We're also getting over 10 times the traffic from PPC at only twice the cost.  Is there a major benefit from Thomasnet that I should be considering?

Hi there, For any such low quality pages, I'd suggest asking two simple questions first: Is the page important for my users, does it add any value ? Is the page beneficial for the site's SEO ? which I think you have asked already from reading your question, so well done there. Pages can usually be removed or redirected, perhaps not yet though at some point, they can be. From what you've said, it sounds like those such pages are adding no value to anybody whatsoever, so Mike is right in suggesting to block them. I have some further thoughts for you; If the page adds no value to your visitors though could be useful for search, then think about Consolidating that page into another (re-using some of the content) or simply 301 redirect it to an appropriate & more valuable page. This idea came from an SEOmoz Mozinar (web conference) from yesterday, is a great concept. There is no point in keeping pages of no value there but hidden away from search engines or users, simple get rid of any such pages by Consolidation or 301 redirects. If the page holds no user value though has some SEO value, then consider 301 redirecting it to another appropriate page. Consolidation and Redirects help to keep a site clean, rather than blocking or noindexing pages that no longer need to be there. If in doubt, then by all means make use of Robots.txt or the NoIndex attribute until you are sure of what to do with them long term. Hope that helps, Regards Simon

UPDATE - Rankings and Traffic restored beginning this very morning for us.  If something like this has happened to you... **I got a very authoritative link yesterday afternoon.  Not 24 hours passed before rankings and org. traffic was restored.  ** This was after 2 weeks of "penalty". What a relief!!

I'd add a comment on the blog post itself asking about it, though it may be something that we don't share because it is proprietary.

Hello Clinthend, Panda makes it difficult for user-generated content sites to control what and how much of their site gets indexed. It used to be that if a page didn't rank well it wasn't a big deal since that didn't affect the rest of the site. I certainly sympathize with your dilemma. I think your idea sounds like a good way to approach it, though I'd probably go ahead and let users access the other geotargeted pages if you think they're useful (and if not, why have them at all, ever?) but block bots from indexing them. You can then allow indexing of particular ones as you get more high quality content onto them. Good luck! Everett @balibones

Probably not the easiest way: I'd just run an advanced report with external links to any page on this root domain. You get up to 100.000 backlinks which equal 1/3 in your case. Then you order them by URL, so you can see which URL provides more than let's say 100 backlinks and with text filter you can extract the exact amount per "mass-linking" domain. After you identified those sitewide linking sites, you can filter them out, so you have a number of links that's more meaningful.

Thats a good question! I can't honestly say but if that is showing correctly then you don't need to worry about it effecting the rankings or your CTR.

SEO Spider from Screaming Frog is a great crawler and a notable alternative to Xenu

Yea, you don't have to login in this case.  I think it could provide benefit to someone if the View Cart page were to come up in the Sitelinks when someone searches for the particular brand. I think I'll go ahead and remove from robots.txt. Thanks

seomoz may script that nofollow because people always post url's with Q's ? True?

I explained the answer to this in the second part of my original post.

I agree, harder to get seo value.  Better to keep things as clean as possible, seo is hard enough as it is...

The simple solution is to use a sitemap generator. There are many solutions of them. Just google it and find which one works best for you. Personally, I don't care for companies I use to tell me "no". I would take that as an indicator I need to look for another ecommerce platform. With the above noted, a sitemap is really not necessary for a well-designed site. Yes, I use one and submit it mainly because I use an easily automated process. As long as your content is well-linked, then Google will see all of it.

I've put up a new site recently that I had been link building to during development. It got indexed in a strange way - just the main keyword, hyphen, domain name... I had it set to noindex but Google obviously followed a few links to it to see where they went. I'm hoping this will clear up for me once the site is cached. I assume you're experiencing another quirky Google ranking script, so it should be all cleared up on the next index of your site, regardless of the 302's actually supposed to be 301s Aaron