Welcome to the Q&A Forum

We had the same issue on one of our sites.   Here is how I understand it after looking into it and talking to some other SEOs.

The duplicate content Title and Meta description seem to lag any 301 redirects or canonicals that you might implement.  We went through a massive site update and had 301s in place for over a year with still "duplicates" showing up in GWT for old and new URLs.  Just to be clear, we had the old URLs 301ing to the new ones for over a year.

What we found too, was that if you look into GWT under the top landing pages, we would have old URLs listed there too.

The solution was to put self canonicalizing links on all pages that were not canonicaled to another one.  This cleaned thing up over the next month or so.  I had checked my 301 redirects.  I removed all links to old content on my site, etc.

What is still find are a few more "duplicates" in GWT.  This happens on two types of URLs

1. We have to change a URL for some reason - we put in the 301.  It takes a while for Google to pick that up and apply it to the duplicate content report.  This is even when we see it update in the index pretty quick.  As, I said, the duplicate report seems to lag other reports.

2. We still have some very old URLs that it has taken Google a while to "circle back" and check them, see the 301 and the self canonical and fix.

I am honestly flabbergasted at how Google is so slow about this and surprised.  I have talked with a bunch of people just to make sure we are not doing anything wrong with our 301s etc.  So, while I understand what is happening, and see it improving, I still dont have a good "why" this happens when technically, I have everything straight (as far as I know).  The self canonical was the solution, but it seems that a 301 should be enough.    I know there are still old links to old content out there, that is the one thing I cannot update, but not sure why.

It is almost like Google has an old sitemap it keeps crawling, but again, I have that cleared out in Google as well

If you double check all your stuff and if you find anything new, I would love to know!

Cheers!

This is not a Google issue. You have an issue with the way your server is configured. It is showing a different link for when a bot is visiting your site versus when someone is browsing the site.  This is why you can't see it.    Google can only see what you show it.

Here is how you find this.   You can run a tool like Screaming Frog or there are plugins for FF or Chrome that lets you change the user agent and how the page is shown

https://chrome.google.com/webstore/detail/user-agent-switcher/ffhkkpnppgnfaobgihpdblnhmmbodake

https://addons.mozilla.org/en-us/firefox/addon/user-agent-switcher/

If you do not want to mess with plugins you can use the good ole rex swain tool

http://www.rexswain.com/httpview.html

Start with one of your pages that list jobs - I found this one

http://www.office-angels.com/en-GB/Pages/all-job-results.aspx?kws=Catering%2c+Restaurant+%26+Bars

Take that URL and put it into the RexSwain Tool in the URL section.  Dont change anything else and press submit. You will get a page with a bunch of code (shows you the source code that is returned).  Use the search page function in your browser to look for the offending "job-id?/"   you should not find it.

Click back and go back to the form.   Leave the URL in place and now add "Googlebot/2.1 (+http://www.google.com/bot.html)" as the user agent and submit.  Do the same search with "job-id?/" you will see it is added in front of all the links to your jobs.  See image Rex Swain Crawl as Google Bot attachment below.

If you do this same exercise with one of the plugins, just change the agent to Google and then hover over the link.  See image Page in Chrome with Google User Agent attachment below.  Bottom left you see the "bad" url and top right you see my User agent set to On to act as google.

Get with your IT guys as you are technically cloaking right now, showing 1 thing to Google and another to users.  You are also potentially showing duplicate content to Google as well for 2 URLs with the same job.

Also, I would take out that parameter handing in GWT, that has nothing to do with all of this.

Good Luck!

fllIsG4 DvSK6Dp

Go Dan!

It would help to use the canonicals as you describe and it is proper the way you mention, but because the products are up for only 4 weeks, the new pages probably don't even have time to rank.  It would be kind of a moo point

http://www.youtube.com/watch?v=fLwYpSCrlHU

You need to probably focus on category pages that might point to the items as those will be more static and can build rank.  They can point to the "sold" archive item page and then when a new item comes up will already have rank on the related category to make the sale.   We have used this on job boards as it works the same.   Any given job will be posted for 1 month at the most, but the category page of $type job in $city will rank for that key phrase and then point people towards the ever changing job listings.   Not sure what you are selling, but think about how you might be able to setup an evergreen type of topic page that people would search for related to your products and focus on that.

No follow links to all login pages and noindex meta tag all login pages.  Just keep those login pages etc out of the index.

I have a URL in the report being called up from lord only knows where!:

Well, the good Lord does know where those pages are linked to due to the whole omnipotent thing He has going on, but if the Moz bot found those pages then that means that the bot knows and you can too!  If Moz bot found the links, then somewhere, someplace on your site, there are links to those pages.  Somewhere in your CMS or due to a typo, those links are being generated on your website.

If you download the CSV within the Moz crawl report section, you can look in the column that shows what pages link to those pages that do not exist anymore and have the temp 302 redirect url.  You can then go to those pages that link to the 302s and update them to correct the 302 redirect links.   I would do this ASAP on those 302 redirects.   You can also use other spider tools such as Screaming Frog, or Botify or Site Condor (there are a bunch out there) to find links like this.

One other suggestion, I assume if you have a temporary redirect, you are seeing a 302 (temporary) vs a 301 (permanent) redirect.  If you can, setup your server to use a 404 (page not found) vs the 302 (temporary redirect) when a page does not exist.  IMHO, 302s are from Satan himself and I have seen them wreak havoc on how Google indexed one of my sites when I used them.  At least with the 404 you get the right signal to Google and then when the crawler finds them, you do not have to be as worried about them.

Good luck and keep the faith!

One other reason I like ending in slashes. GA and other software that crawls your site and then produces reports will look at the slash and then include that URL as the home page with everything else under it in the same reports.

Some developers like to leave off the slash and just have the index page as  .com/resume-tips

You then get people who will naturally add the slash at the end in links, or you have a footer where you add the slash when you did not mean to and then you have a duplicate content issue.

I like to end with the slash and just be consistent.  Seems like most reports "expect" that convention and so it just will help down the road.

You have a few options here.  Option A is if you are going to build a site that will have similar topic based content as the old one and you want to use a larger portion of that domain authority from the old site to the new.

1. Pull those 404 errors from GWT in a spreadsheet.   This gives you a corpus of links to work with.

2. Go into Bing WT and they have a way to browse what they have and had indexed.  What is nice here is that Bing will tell you what URLs (even old 404s) have links to them.

3. Run your links through Open Site Explorer. You can then also get linking data, FB and Twitter data in addition to OSE data on the old URLs

4. If need be, run the more important dead URLs through the Wayback Machine http://archive.org/web/web.php  you can now even see what the actual content was on the old URLs.

5. After doing all of this, pretty quick you should be able to see if there were any authority pages on the site that have now expired and you also know what those pages were about via the wayback machine.

6. On the authority pages, create new pages on the new site that have to do with the same topic, i.e. semantically related to the old page.

7. 301 the old authority pages to the new authority pages.

8. The rest of the URLs you can just let them 404.  They will continue to 404 several time until Google drops them.  I would leave them in GWT as over time they should drop out as Google starts to ignore those pages, this may take a few months.  You can then just check GWT for any new 404s that might show up from the new site and you need to deal with.

One thing to note on all of this.  You may have to let the old sitemap 404 vs redirecting the sitemap.

http://moz.com/blog/how-to-fix-crawl-errors-in-google-webmaster-tools

"One frustrating thing that Google does is it will continually crawl old sitemaps that you have since deleted to check that the sitemap and URLs are in fact dead. If you have an old sitemap that you have removed from Webmaster Tools, and you don’t want being crawled, make sure you let that sitemap 404 and that you are not redirecting the sitemap to your current sitemap."

If you delete the 404s from GWT the next time Google spiders the old pages they will just show up again, up to you then.

Option B - if you dont care about the old pages, just let them 404 as mentioned above, but be aware of the issue with old sitemaps.  You can check the Google index for old URLs in the SERPs or also if you look into GWT and look for data on your Search Traffic.  Make sure that the old URLs are not showing up under your Search Queries.

Look at your analytics data.  If you have text around each of your photos (descriptions etc) on those pages then you are probably ranking for a bunch of long tail terms on these pages and would see traffic to a bunch of different pages.  If you find that most of your traffic only goes to your main page, then you are probably ok.

The potential impact of removing the pages would be that you would remove all of the referral traffic coming to those individual pages and so that traffic would go away.

You can probably still setup a thumbnail gallery to all of your image pages and then link out.   Take a look at the way USA today has you browse articles on the site (click the arrows on the sides, but watch the URL bar).  It feels like you are just flipping through images/page, but each article has a unique url etc.  Clever way of combining interface and site structure.

Good luck!

I am with Erwan.  You have essentially two authority sites, two authority domains that link to each other and support each other.  That is golden and you want to keep taking advantage of that

Sure, you can move the blog and use 301s, and it would "work" but I bet you would lose some ranking.

I think it depends on what pages you are removing.

If you have old blog posts that still get traffic and have links to them, you may want to reconsider keeping them or how you redirect them.  You should be able to look in your Analytics and see the amount of traffic they get, and also look in OSE to see what kinds of links they have pointing to them.

We are doing something similar on one of the sites I manage and I was able to look at these factors objectively vs subjectively and make some good decisions.  When we looked at old content (content prior to 2005) and looked at the factors above it was pretty interesting.  There was one page that accounted for 65% of all pageviews out of several hundred articles in that time period.  We obviously treated that page with greater care than the rest of the pages.

You will probably find a similar pattern, the 80/20 rule.  There will be about 20% of your pages that you really should keep or be very careful where you redirect and the rest you can just let 404 or 301 to a subdirectory higher.

When you look at it this way, you can account for any SEO "equity" that a page might have and then know what the impact would be.  Back to my example above, even if I deleted all of those old pages (including the one page that accounted for the majority of traffic to that group of pages) all of those pages combined only accounted for 2% of all my total pageviews.   So, even if I screwed something up when I moved thi group, I know the impact was minimal and I can sleep a little better at night.  

Generally speaking, you don't want thin content.  Don't do it.  Take each of the 8 topics and write to the fullest.  You may be surprised how much you can write on a given topic (vs just 150 words). Hire out a freelance writer on oDesk to help if you need.  You may then want to have an overview page that talks about the 8 section and links to them so that people (and bots) can see how things are organized.

If you create the GWT account for the dev site and you submit for removal, GWT requires that you either a) have the site blocked in robots.tx or have a noindex meta tag on the pages. Otherwise they will just crawl you again later and you are back in the index.  See my post from earlier.

You want to have as many pages in the index as possible, as long as they are high quality pages with original content - if you publish quality original articles on a regular basis, you want to have all those pages indexed.  Yes, from a practical perspective you may only be able to focus on tweaking the SEO on a portion of them, but if you have good SEO processes in place as you produce those pages, they will rank long term for a broad range of terms and bring traffic..

If you have 20,000 pages as you have an online catalog and you have 345 different ways to sort the same set of page results, or if you have keyword search URLs, or printer friendly version pages or your shopping cart pages, you do not want those indexed.  These pages are typically, low quality/thin content pages and/or are duplicates and those do you no favor.  You would want to use the noindex meta tag or canonical where appropriate.  The reality is that out of the 20,000 pages, there are probably only a subset that are the "originals" and so you dont want to waste Googles time in crawling those pages.

A good concept here to look up is Crawl Budget or Crawl Optimization

http://searchengineland.com/how-i-think-crawl-budget-works-sort-of-59768

http://www.blindfiveyearold.com/crawl-optimization

I help run a directory site and we have City pages with multiple listings.  We don't mark up that page but we have a landing page for each location.  The landing page for a location is what we markup that way.

If you look on sites like Yelp they do the same thing.

http://www.yelp.com/dallas

On the Dallas page there is no schema markup but

http://www.yelp.com/biz/eddie-vs-prime-seafood-dallas

If you visit a local restaurant then the schema markup shows up.

I was looking through the schema.org documentation

http://schema.org/docs/gs.html

Using the url property. Some web pages are about a specific item. For example, you may have a web page about a single person, which you could mark up using the Person item type. Other pages have a collection of items described on them. For example, your company site could have a page listing employees, with a link to a profile page for each person. For pages like this with a collection of items, you should mark up each item separately (in this case as a series of Persons) and add the url property to the link to the corresponding page for each item, like this:

 _[itemprop="url">Alice Jones](alice.html)

  [itemprop="url">Bob Smith](bob.html)_ 

So in the example on Schema, you can tag the location with the type and then use the URL parameter to point to the actual page that has the information.

I then looked at CitySearch I did see an example of this

http://dallas.citysearch.com/find/section/dallas/restaurants.html

and

http://dallas.citysearch.com/profile/34327220/lewisville_tx/mama_s_daughters_diner.html

If you look at the code on the Dallas Restaurant pages they use Item List markup

itemscopeitemtype="http://schema.org/ItemList">

That is a list of Breakfast Restaurants in Dallas and then for each place on that page they will mark up

itemtype="http://schema.org/LocalBusiness" itemprop="itemListElement">

and

itemscope itemtype="http://schema.org/AggregateRating" itemprop="aggregateRating">

and then they reference the URL to the location page (as suggested above in the schema.org documentation)

itemprop="url" class="slyLink fn url org" name="popular.breadwinnerscafebakery.1.businessName" id="graphicBrowse.1.1.name">

[Check with your developer, but it looks like if you define the list of locations first, the spiders can see that all of the locations are a part of that list (vs the page being dedicated to a single location) then when you have the link to the landing page for the location you can do the full markup.

Good luck!](http://dallas.citysearch.com/profile/41040743/dallas_tx/breadwinners_cafe_bakery.html)

Usually, this would be true that you would need to use the noindex tag to get things out of the SERPs and need to leave the robots.txt "open" to the crawlers.  But when you are working with the remove URL tool in GWT,they rx that you then put the site in robots.txt to keep them out of it

The removal tool in GWT takes care of Google taking the URLs out and then the robots.txt keeps the bots from coming back.  Just a different sequence than if you were to use the noindex meta.

Even though there are less pages indexed compared to those that are blocked, you still have a significant increase in indexed pages as well.  That is a good thing!  You technically have more pages that are indexed than before.   It looks like you possibly relaunched the site or something?  More pages blocked could be an indexing problem, or it might be a good thing - it all depends on what pages are being blocked.

If you relaunched the site and used this great new whiz-bang CMS that created an online catalog that gave your users 54 ways to sort your product catalog, then the number of "pages" could increase with each sort.  Just imagine, sort your widgets by color, or by size or by price, or by price and size, or by size and color, or by color and price - you get the idea.  Very quickly you have a bunch of duplicate pages of a single page.  If your SEO was on his or her toes, they would account for this using a canonical approach or possibly a meta noindex or changing the robots.txt etc.  That would be good as you are not going to confuse Google with all the different versions of the same page.

Ultimately, Shailendra has the approach that you need to take.  Look in robots.txt, look at the code on your pages.  What happened around 5/26/2013?  All those things need to be looked at to try and answer your question.

No.

Here is a Google documentation on how to use the GWT to remove a page/directory/site and then the interaction with robots.txt

http://googlewebmastercentral.blogspot.com/2010/03/url-removal-explained-part-i-urls.html

"In order for a directory or site-wide removal to be successful, the directory or site must be disallowed in the site's robots.txt file."

Side story.  I once had a subdomain that I needed to take out, but I could not modify the robots.txt file properly (long story).   So, we used the GWT tool and the meta noindex tag.  It still worked, but I think that would only be a backup approach to the one suggested by the documentation.

It will keep on crawling.  I have seen 6 months to 1 year later they come back.  Especially if there are links from other websites pointing to your old URLs.

Why they do repeat crawls on 301s and 404s for that matter is that they know how much the web changes and assume that there may have been a mistake and so want to double check.    Also, you have to consider, they will not recrawl 1000s of pages all at once.  They will probably pick up the most popular URLs first as they crawl them more often and then the less popular pages later.

IMHO. You will need to find a long term solution in place.

Here is the why you dont want to use htaccess

http://stackoverflow.com/questions/209209/htaccess-redirect-performance

One suggestion for Apache - RewriteMap

http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html#rewritemap

That or all the developers who are telling you not to use .htaccess should have at least 2 solutions to use instead that should work.  Go back to them to solve the issue.