Welcome to the Q&A Forum

You need to get a developer who understands a lot about http requests.  You will need to have one that knows how to basically run a spidering program to ping the website and look for changes and scrape data off of those sites.  You will also need to have the program check and see if the coding on the page changes, as if it does, then the scraping program will need to be re-written to account for this.

Ideally, those sites would have some sort of data API or XML feed etc to pull off of, but odds are they do not.  It would be worth asking, as then the programming/programmer would have a much easier time.   It looks like the site is using CMS software from http://www.cts-america.com/  - they may be the better group to talk to about this as you would potentially be interfacing with the software they develop vs some minion at the help desk for the dept of motor vehicles.

Good luck and please do produce a post here or a YouMoz post to show the finished product - it should be pretty cool!

That is a completely valid question.   This is why setting up the separate GWT account for the dev.domain.ext vs www.domain.ext.   When you submit the removal request it will only be in the dev.domain.ext account.

The only thing you want to watch is that if you setup robots.txt in your dev environment you want to make sure that it does not get pushed out to your production server. That is the only gotcha as I see it.

Yes.  Google will use formatted data even when there are not Rich Snippets.  They will also pull from Lists.  See this articles here: http://moz.com/blog/how-do-i-get-googles-bulleted-snippets

If you read in the on page optimization tool, it is inconsistent with the crawl tool.

"Avoid Excessive Internal Links

Employing an excessive quantity of internal-pointing links may not directly harm the value of a page, but it can influence the quantity of link juice sent through those links and dilute it's ability to help get link targets crawled, indexed and ranked.

Recommendation: Scale down the number of internal links to fewer than 100 (preferrably), and, at a minimum, fewer than 300"

That said the 100 links rule is a "Warning" (Yellow) and not a Error (Red).  It is still confusing.

Here is also a Matt Cutts video that refutes the 100 links

http://www.youtube.com/watch?v=l6g5hoBYlf0

Seems like Moz needs to update its messaging around this item.

2. use robots.txt vs the meta tags - robots.txt is preferred.

Technically - yes you can do it.  You can use 301 redirects and make it all work without too much of a hit.  You will get a bit of a hit short term and if you setup all the redirects properly, recover within 3-6 months.

I like Rands recent post to move to moz.com as a nice case study

http://moz.com/rand/10-traffic-graphs-seomoz-moz-domain-migration/

Now - throw out everything I just typed.  You have to consider that your "brand" online is the shorter URL.  People are used to it, they already have habits and bookmarks and emails, etc etc in place.  You could "screw up" your online brand and not need to.  Plus, longer URLs are a pain in the butt.  Why frustrate your users by making them type in a  longer URL?  Why frustrate all the people who want to email you when they have to type out a longer URL?  Why, why why?

Southwest Airlines uses southwest.com, American Airlines uses AA.com, New York Times uses nytimes.com, I could go on, there are big brands that use shorter URLs.   People are lazy, why increase friction when trying to get people to your site.  I know for a fact that people search more often for keywords "NYC $services" vs typing our "New York City $services" (service could be a locksmith or restaurant etc) - why?  Who wants to type all that out when you can just put in NYC?

You could argue about the technical reasons, but what will convince them is if you give them the reasons why customers will not like the new domain,  I think that is your best bet.

You rock Dan!  Thumbs up!

301 redirects is the way to go.  If you have a user on the wrong page, it forwards them to the correct one.  Link equity is passed as well.

The noindex nofollow will help get whatever version you want out of the SERPs, but it does not fix it from happening again.

First and foremost when you finish all this - password protect your dev instances.  A url will leak out eventually and then this happens.  I know it is a PIA, but it is worth it.

To remove subdomains.  Go into GWT and register the subdomains as separate websites in GWT.  Create a robots.txt for each subdomain (not the one you mention, you need a robots that is specific to that subdomain that disallows all files.  If you cant do that, have your subdomains include a noindex meta tag on all pages.   You have to be careful with this as you do not want to push out your dev. robots.txt or the noindex meta tags to your production server, but it can be done.  Talk to your devs.    Then go into GWT and use the URL removal tool.  Just leave it blank and it will remove the whole site.

Poof.  Gone.  You can then watch the GWT accounts.  They will show errors for the dev site like "Severe health issues are found on your site - Some important page has been removed by request."  This is a good error as it confirms that that subdomain is removed.

We actually used this not on a dev site but on our www1 server that was indexed.  We use a load balancer with multiple copies of the site.  www1 was completing with www.   Using this above did the trick.

whoop there it is!

You can also noindex those pages to simply take them out of the index and then later nofollow links to them.

I forgot to add.  I use a canonical on all my printer friendly versions pointing to the original page and then I have a self canonical on that page - again on thousands of pages.  AOK.

It is not "bad", although typical style would be that you can drop it as the extra characters are not needed and nobody likes extra typing - just ask Mr. Twitter.  He used brevity to become a billionaire!  Hmm .. I wish we could get Moz points for alliteration.

What is key is that you are consistent in your use of it.  If you want to use /index.php then go for it.  Just make sure every time you link to that URL in your menus or in articles when you Twiddle it of Farcebook it, you include the /index.php at the end as you do not want to have duplicate URLs for the same page.  I would also setup 301 redirects so that the / only version redirects to the index.php version.

All of that said, you are going to find that after the 104th time of Twiddling that URL, you will say, "Gee, it sure is a pain to type all those extra characters."  You will also find that when people are going to share your URLs they may have a tendency to drop the index.php as again, it is extra work.  If you have the redirect in place, you will be ok, but I say, why have you and everyone else do all that extra work to start with.  Just start with the URL ending in the slash and stay with that.  Have all other versions of the index page (index.php, index.htm or even a non slashed version, etc)  301 redirect to the URL that ends in a /.

I would answer the same as Kurt for the install.   You put the noindex tag in the header of the core page and so when all the other pages are generated with the parameters it will be added to those pages automatically.    Once you get the pages out of the index, then I would nofollow links or use robots.txt to those pages to keep the bots out to start with.

Hard to say without seeing the products and so you may not get much response here more than "it depends".   Sit down and be creative, you may be able see if the differences can be highlighted enough that they are different from the others and so you can use them.

Here is what I would do.  If you have 4 versions of a widget, create 1 page with information on all four of them.  Then you can talk about all of them at once and optimize around the 4 versions.  Lets say you have a red, blue and green widget - those are the only differences and everything else is the same.  The page should talk about how great the widget is, talk about the 4 colors, have pictures of the widget in all 4 colors, let people vote on what color they like the best, have links to PDFs on each of the 4 widgets etc.  This may be the better approach.

I am just not sure about using of the canonical as people probably need to be able to find the different versions of the widget and so you need to make sure that the information is out there.  If you setup a page for the 4 versions and then canonical 3 of the versions to the 1, you are still "de-emphasizing" the other 3 versions that people may be searching for.

I really try to only use canonicals if there is an exact duplicate (say a printer friendly version of the original) or I use it for business purpose.  Example: I have one site that has city and zip search options for locations - I canonical all my zip result pages to the parent city page - I do this as I know 1) for this area people do not search in the SERPs by zip code so no use optimizing around zips, and 2) the zip pages are pretty much dups of the city page anyway.  I only leave the zip pages on my site as an option from a usability for people once they hit my site if they want to search that way.

Good luck!

Yes bad for both.  You now have the name of a file acting as the name of a folder.

As mentioned above - kill the use of index.php as your index "file" - just end in a slash.

I know php treats these as routes/queries that then produce a page, but it can get things all messed up real quick.

See the URL examples above.   Basically dont nest product pages under sub categories or category folders.

Thanks for the example.  It seems that for a server there would be more information about it that that paragraph.  Processor information, RAM, disk space, ports etc.    I just did a search and found this info

https://www.avantidataproducts.com/product/ibm/7379c2u

I think the bigger problem is not differentiating these pages from each other on your own site, but differentiating from other sites.

This is just good old content development.  You need to see if there is a way to tap into people rating and leaving comments.  Do you have any case studies on this product etc.  Can you find any other media on it, can you create it.  Go to Zappos.com they produce a ton of videos that just describe a pair of shoes.  Do you have someone who could be your video representative and you shoot a bunch of videos on your top products and embed them.

I think otherwise, you are just going to be similar to everyone else with the same product.

You have to reduce the number of pages that Google "sees" when you are showing search result pages.  If you have too many sort versions on a given query, you are giving Google a bunch of duplicate (or closely duplicate) pages to spider and it prevents them from getting down into your item detail page.

I would find a way to setup a main / generic pagination result so that Google can spider that and then as they go through each page, they find links into the products.   All the other versions of your search results, I would no follow links to them or put them in a folder that you can robots.txt.

This way consumers can search and sort to their hearts desire, but you provide a clear path for Google.

Also, be sure to setup a XML sitemap and include links to all of your item detail pages.  You can also split out into multiple sitemaps if you have large enough number of item detail pages.   A good article on this is here

http://www.blindfiveyearold.com/optimize-your-sitemap-index

Hi Remus,

He was only talking about 1 domain as I read it so you may be confused.  The 301 is a stronger signal than a canonical, also, you do not want other versions of the same URL functioning as then they could be shared out and so you have links coming into different URLs for the same page.   The 301 redirect eliminates that possibility.