Google still listing old domain

[Ham1979]

Hi

We moved to a new domain back in March 2014 and redirected most pages with a 301 and submitted change of domain request through Google Webmaster tools. A couple of pages were left as 302 redirect as they had rubbish links pointing to them and we had previously had a penalty.

Google was still indexing the old domain and our rankings hadn't recovered. Last month we took away the 302 redirects and just did a blanket 301 approach from old domain to new in the the thinking that as the penalty had been lifted from the old domain there was no harm in sending everything to new domain.

Again, we submitted the change of domain in webmaster tools as the option was available to us but its been a couple of weeks now and the old domain is still indexed

Am I missing something? I realise that the rankings may not have recovered partly due to the disavowing / disregarding of several links but am concerned this may be contributing

[RobertJakobson]

If your goal is to delist the old domain I am going to copy the answer I just gave at http://moz.com/community/q/how-to-exclude-all-pages-on-a-subdomain-for-search, simply because it's clear and works quickly (48h) in my experience.

This is the authoritative way that Google recommends at https://support.google.com/webmasters/answer/1663419?hl=en&rd=1:

1. Add an robots.txt file for your domain. Usually via FTP. Add the "noindex" meta-tags to every page as well.
2. Add your subdomain as a separate site in Google Webmaster Tools
3. On the Webmaster Tools home page, click the site you want.
4. On the Dashboard, click Google Index on the left-hand menu.
5. Click Remove URLs.
6. Click New removal request.
7. Type the URL of the page you want removed from search results (not the Google search results URL or cached page URL), and then click Continue. How to find the right URL. The URL is case-sensitive—use exactly the same characters and capitalization that the site uses.
8. Click Yes, remove this page.
9. Click Submit Request.

To exclude the entire domain, simply enter the domain URL  (e.g. http://domain.com) at the 7th step.

[Ham1979]

Thanks for your reply,

It's not that I want to de-list the old domain as I would rather people get to the site using that domain than not at all but, my concern is that for whatever reason the transfer hasn't completed as it's been such a  long time and we're for instance not getting the full benefit of sites linking to the old domain passed to the new one

[RobertJakobson]

Got it, you must have tried adding the canonical URL meta tags already, right? If not, check out: http://moz.com/blog/rel-confused-answers-to-your-rel-canonical-questions

"...in late 2009, Google announced support for cross-domain use of rel=canonical. This is typically for syndicated content, when you’re concerned about duplication and only want one version of the content to be eligible for ranking...

..First off, Google may choose to ignore cross-domain use of rel=canonical if the pages seem too different or it appears manipulative. The ideal use of cross-domain rel=canonical would be a situation where multiple sites owned by the same entity share content, and that content is useful to the users of each individual site. In that case, you probably wouldn’t want to use 301-redirects (it could confuse users and harm the individual brands), but you may want to avoid duplicate content issues and control which property Google displays in search results. I would not typically use rel=canonical cross-domain just to consolidate PageRank..."

[Ham1979]

Hi there

The old pages don't exist any more to add the canonical they're 301's from old domain to new but over 1000 pages show up for site:www.fhr-net.co.uk

[RobertJakobson]

Hmm.. certainly someone with more experience than myself would have a more elegant solution, but I would still try to do this by establishing the canonical URL because you don't want to delist: https://support.google.com/webmasters/answer/139066#6

If you can configure your server, you can use rel="canonical" HTTP headers to indicate the canonical URL for HTML documents and other files such as PDFs. Say your site makes the same PDF available via different URLs (for example, for tracking purposes), like this:

_http://www.example.com/downloads/white-paper.pdf
http://www.example.com/downloads/partner-1/white-paper.pdf
http://www.example.com/downloads/partner-2/white-paper.pdf
http://www.example.com/downloads/partner-3/white-paper.pdf_ 

In this case, you can use a rel="canonical" HTTP header to specify to Google the canonical URL for the PDF file, as follows:

Link: <http: www.example.com="" downloads="" white-paper.pdf="">; rel="canonical"</http:>

[RobertJakobson]

Oh, it may be that it's the other way around with canonical URL-s. At least according to Google (here: https://support.google.com/webmasters/answer/6033086?hl=en

1. _Each destination URL should have a self-referencing rel="canonical" meta tag.  _

[MarieHaynes]

This doesn't answer your question, but I just wanted to point out that the 301 or 302 redirects are not a good idea.  Even if you got the penalty lifted, there still can be unnatural links there that can harm you in the eyes of the Penguin algorithm.  A 301 will redirect those bad links to the new site.  A 302, if left in place long enough will do the same.

Here's an article I wrote today that goes into greater detail:

http://searchenginewatch.com/sew/how-to/2384644/can-you-safely-redirect-users-from-a-penguin-hit-site-to-a-new-domain

[Ham1979]

Thanks for responses,

One week on and since submitting the second change of domain in GWT we've seen the number of pages indexed for the old domain drop from over 1300 to around 700 this week which is something

Regarding the redirect debate, it's an interesting read thanks for sending that. Isn't the situation the same as a site that didn't have a penalty in that you should be monitoring your backlink profile and reconfiguring or disavowing links outside the guidelines whilst carrying out activities that will naturally build decent links and therefore redress the balance?

[MarieHaynes]

I think that I'm so used to answering questions about penalized sites that I assumed that you had moved domains because of a penalty.  My apologies!

Sounds like you've got the right idea.  

[Ham1979]

No worries,

I appreciate you taking the time to answer my question

[Ham1979]

Even after all this time, there are still over 700 pages indexed on our old domain even though we have submitted the change of address twice in Webmaster tools, the second one being about 6 months ago if not longer

old domain is www.fhr-net.co.uk

Any advice would be appreciated

[MarieHaynes]

This is odd. The pages all seem to redirect from the old site to the new, so why is Google still indexing those old pages?

I can't see the robots.txt on the old site as it redirects, but is it possible that the robots.txt on fhr-net.co.uk is blocking Google? If this is the case, then Google probably wouldn't be able to see the old site and recognize the redirects.

It may also help to add a sitemap for the old site and also to ask Google to fetch and render the old site's pages and then submit them to the index. This should cause the 301's to be seen and processed by Google.

[Ham1979]

Hi Marie

Many thanks for your response,

I've just looked in Webmater tools at the old domain and the option to change domains is there again but I also noticed when looking at the crawl errors there was a message along the lines of crawl postponed as robots.txt was inaccessible.

At the moment it's just a blanket redirect at IIS level so following your advice I'll re-establish the old site's robots.txt and a sitemap and see if Google crawls the 301's to the new domain.

In some ways I'm glad I haven't missed anything but would be nice if just the new domain indexed after all this time !

Thanks again

[Ham1979]

Hi

I now have a robots.txt for the old site and I created a sitemap by replacing the current domain with the old one and uploaded.

Weirdly when I search for the non-www version of the old domain the pages indexed has increased!

According to WMT the Crawl postponed because robots.txt was inaccessible however I've checked it returns status 200 and the Robots.txt Tester says it's successful even though it never updates the timestamp.