Welcome to the Q&A Forum

While you may not be able to compete directly with the big dogs immediately, you aren't doing yourself any favors by sticking pages you wish to rank deeper inside your site. Even though the content may be more specifically relevant, the reality is that once you are targeting traffic on a national scale, you are probably looking at your link profile as the defining factor of success rather than your content. Content still retains power in terms of generating relevancy, but link building is what you need to compete with those long-standing enterprises. In my opinion, content location is secondary in this situation. A better way to go about this might be to run some competitive analysis on a wide range of keywords to determine: a) Where are your customers coming from? b) What are they looking for? c) Where does your competition rank? d) Why are they ranking where they do? Once you can answer these questions, you have the ability to make a strategy that will work regardless of where your content is located. Additionally, adding more pages will just make work for your web dev team without any discernible increase in your potential rankings, and will take time. Given the size of your site and the variety of products you carry, this may be an ineffective use of time when you might just be building a significant link profile to a couple of major landing pages. Whatever you decide, learning why your competition is succeeding needs to be your priority. Then you can decide on a content-driven strategy or a link building strategy.

Thanks, Hurf. Thanks for the excellent reply. Unfortunately, with Volusion as our web-store we don't have the option of displaying anything but the full product name on listings by category unless we engage a Volusion partner to modify our store or we migrate 80K products to a different and more complex product/option structuring model. For the sake of the clients, 12-products-per-page would be maddening. Text search is very ineffective when it comes to parsing product dimensions, so we produce long listings, sorted by size so the user can quickly locate their product. In the short term, I think it makes most sense to pay for custom programming to reduce the product description to just the dimension and quantity when displayed on a category page.

Wow! thank you, many of the robots.txt testers still show them as disallow, good to know! thank you!

On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time. While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two. As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.

I'm afraid there is no way to delete them. You get to ask a question of SEOMoz staff each month however, so rather than post this, I would ask them the question of if they can add a delete button or delete a question for you.

Hi! Yes, our crawler does read wildcards in a robots.txt file. This change was made in the past 3 months or so. Hope that helps! Jen