Welcome to the Q&A Forum

1. If you add just noindex, Google will crawl the page, drop it from the index but it will also crawl the links on that page and potentially index them too. It basically passes equity to links on the page.

2. If you add nofollow, noindex, Google will crawl the page, drop it from the index but it will not crawl the links on that page. So no equity will be passed to them. As already established, Google may still put these links in the index, but it will display the standard "blocked" message for the page description.

If the links are internal, there's no harm in them being followed unless you're opening up the crawl to expose tons of duplicate content that isn't canonicalised.

noindex is often used with nofollow, but sometimes this is simply due to a misunderstanding of what impact they each have.

George

Hi,

I took a quick look at your site, sitemap and index status and only 25 urls in Google, but very many more in the sitemap.

What I couldn't work out is where your /item-details/ urls in the sitemap are linked to from your website? I can't get to them through buying -> catalogue. It won't help indexing status if they aren't being linked to from anywhere.

The biggest issue you have however is the way canonicals are set up on the problem pages. If you go to this page:

https://www.wilkinsons-auctioneers.co.uk/item-details/?ID=2710

It has the following canonical (without the id):

rel='canonical' href='https://www.wilkinsons-auctioneers.co.uk/item-details/' />

If you search on Google, that canonical URL is indexed, so if you fix this by adding the id to the canonical they should start to appear in SERPS.

You have exactly the same problem on your auctions pages. e.g. https://www.wilkinsons-auctioneers.co.uk/auction-items/?id=13&pagenum=51

Another point that will help you rank is to use friendlier / more descriptive URLs for the items.

Hope that helps

George

Hi Lee,

The foundation site idea sounds like a real roundabout way of achieving organic traffic and hence sales - which from a high level I'm assuming is what you're trying to achieve. It would perhaps make more sense if you were going to use the Foundation site to drive referrals, or to use for PR, rather than solely for link equity purposes.

It wouldn't take much for Google to work out that the foundation site is a bit of a cynical attempt to gain rankings.

If I was you I'd focus on improving the content and linkability of your client's existing site and address some of the branding issues head on rather than side-stepping them with a sister website. You can incorporate the "foundation" idea into the existing website (perhaps on a subdomain or directory), which if done properly - with valuable content - will earn natural links and therefore gain far more organic value than having a sister website.

George

I've not come across any reason ever that would give cause to be concerned about losing Page Authority by having a page canonical to itself.

No need to be concerned. Aside from all the really well documented best practices on canonicals, in your original question you've spotted at least one big site that does this. They pay the SEO big bucks and rank well.

Yes this is a good idea as it's a catch all for URLs that might include tracking URL parameters, or other parameters that don't affect the page content. When there are no tracking parameters, it's going to be more development and testing work to hide the canonical, when having it there doesn't cause any issues. It's also quite a brutal but effective catch all if your page was accidentally accessible via other URLs - e.g. non-www or https.

George

Hi,

I migrated a load of product category pages on one of my websites recently to cleaner URLs and to force the crawl I submitted the new URLs (and children) to index via WMT. This was to pick them up quickly - and it worked (within seconds). The old URLs appearing were never a problem. However there are limits to the number of times you can do this so that might be a sticking point for your solution as I'm guessing you have lots of products. Try it with one page (a low traffic and selling product!) and see what happens - and let us know.

It's possible Google is holding onto your old URLs because they have a number of inbound links and the crawl will eventually catch up to only display the new URLs if you give it time.

Aside from agreeing with the sitemap submission suggestion, I'd also triple check that your 301s / canonicals are set up properly on your website's old URLs by firing Screaming Frog or another crawler at it.

George

Hi Graeme,

For old product pages - your solution is good regarding showing users alternatives to the out of stock products. No need for an "out of stock page" as there's no value in that for crawlers or users. Regarding point 2 - if you redirect discontinued product pages to category pages that should be fine although Google may regard that as a soft 404. If there are loads of products like this and you 301 them in one go then the chances are it will flag up in Google WMT. If there are a small number and you introduce them gradually then you'll probably be fine.

For the crawl errors question, adding value to the pages in terms of related products is a good solution if that's viable and the pages will be different enough from each other (i.e. no duplicate content). One thing that isn't clear at the moment is if you're redirecting empty category pages all to the homepage - or if it's possible to redirect or canonical them to their parent category.

e.g. For home -> clothing -> men's clothing -> shoes

If all the men's shoes are discontinued, then redirect that page to men's clothing rather than to the homepage. This reduces your chances of getting a soft 404, and is also arguably a better user experience.

Hope that helps,

George

Yes it's the worst possible scenario that they basically get trapped in SERPs. Google won't then crawl them until you allow the crawling, then set noindex (to remove from SERPS) and then add nofollow,noindex back on to keep them out of SERPs and to stop Google following any links on them.

Configuring URL parameters again is just a directive regarding the crawl and doesn't affect indexing status to the best of my knowledge.

In my experience, noindex is bulletproof but nofollow / robots.txt is very often misunderstood and can lead to a lot of problems as a result. Some SEOs think they can be clever in crafting the flow of PageRank through a site. The unsurprising reality is that Google just does what it wants.

George

Hi Rafal,

The key part of that statement is "we might still find and index information about disallowed URLs...". If you read the next sentence it says: "As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the site can still appear in Google search results".

If you look at moz.com/robots.txt you'll see an entry for:

Disallow: /pages/search_results*

But if you search this on Google:

site:moz.com/pages/search_results

You'll find there are 20 results in the index.

I used to agree with you, until I found out the hard way that if Google finds a link, regardless of whether it's in robots.txt or not it can put it in the index and it will remain there until you remove the nofollow restriction and noindex it, or remove it from the index using webmaster tools.

George

SearchMetrics would be a good place to start - you won't get individual keyword historic performance but it will show your website's overall SEO visibility over time. Particularly useful for tying in with Google algorithmic updates.

George

Hi Finnmoto,

You're in luck - it does use a 301 for that homepage redirect. The results of this test were brought to you by the mighty Fiddler (http://www.telerik.com/fiddler).

I've migrated pages like this before and it can take a bit of time for the dust to settle. Remember you've migrated an entire website to a new subdomain in one go and that takes time for Google and other services to process (depending on how authoritative your site is).

It's worth crawling your entire site's old URL structure with ScreamingFrog to check the redirects were implemented correctly.

Regards,

George

+1 what Martijn says.

Even though the pages have no traffic, Google may still be crawling them if they have links so there may still be some equity in keeping them. If this is a concern you can redirect them to the most relevant (quality) page on your site. If it's only a few pages this is fine but any more than a few they might come up as "soft 404s" in Google WMT.

You can check if Google is crawling the pages by checking your web logs. That said, if the content is of no value and they don't have any decent links then I would just delete them and move on.

Regards,

George

Hi Tanveer,

It's hard to answer your questions without seeing the raw data. I presume these are external rather than internal links, and that they are genuinely new as opposed to only just having been discovered. I would start with going to Webmaster Tools, downloading your latest links and having a look at where they are coming from.

There could be a number of reasons for this, and so there's no point me speculating and you're right to investigate further. Using a link profile checker such as cognitiveseo.com will give you a clearer idea on the quality of any new links you acquire.

Feel free to post more information if you need,

Regards

George

The problem you're describing is almost exactly the reason why canonical URL functionality exists. Just pick your canonical (with or without slash - it doesn't matter) and make sure you roll it out consistently across your website and sitemap.

Regards,

George

Hi Nicole,

Personally I've had lots of issues getting images indexed on large websites - and I've come across other webmasters with the same problems. If you really want to get diagnostic then you need to start splitting out content into different sitemaps as SEO-Buzz suggests so you can see a clearer breakdown in Webmaster Tools.

Another approach you might want to try is doing some image link building - your image content is ripe for being active on Pinterest and other photo sharing platforms. Getting the content placed like this should help with indexing.

Regards,

George

Just adding this to robots.txt will not stop the pages being indexed:

Disallow: /*login?

It just means Google won't crawl the links on that page.

I would do one of the following:

1. Add noindex to the page. PR will still be passed to the page but they will no longer appear in SERPs.

2. Add a canonical on the page to: "www.exemple.com/user/login"

You're never going to try and get these pages to rank, so although it's worth fixing I wouldn't lose too much sleep on the impact of having duplicate content on registration pages (unless there are hundreds of them!).

Regards,

George

I bumped SEO-Buzz's answer

In practice, sometimes it's impossible to write unique product descriptions. I've worked with websites that have over 15K products, with around 5K changing every quarter. To keep on top of that you'd need an army of copywriters.

In that situation I would recommend doing the following:

1. Make sure your category / hub page content is awesome / unique / relevant. These will then be your main landing pages.

2. Pick key product sections - based on high margin / good stock availability / competitive pricing and update the product descriptions for them. If you client sees improved sales as a result, they will probably roll out this strategy to the rest of the site.

Hope that helps

George

Hi there,

Any reputable affiliate should be adding nofollow to outbound links, and I would be concerned if they weren't. Even if PR isn't passed on 302s, what's stopping the affiliate from making them 301s or simply direct links in future?

Blocking GoogleBot using robots.txt on landing pages isn't going to stop Google indexing URLs it finds - you would have to use canonicals or noindex on the page for that. Even that wouldn't negate the impact of inbound links from potentially "toxic" websites.

I would say that you can't have your cake and eat it. If you're getting traffic and sales from these affiliate links then they're good for business. However there's a good chance if the sites are poorly regarded by Google, and they aren't using nofollow on the links that you will be penalised in organic listings.

I have used affiliates in the past, and learnt to keep them on a tight leash.

If you're concerned about your link profile I can recommend using cognitiveseo.com to analyse it.

Regards

George

Hi Sika,

What you're seeing isn't anything to be concerned about, and Moosa has already answered the cannibalisation part. I'd advise against tracking positions for individual keywords using a single tool from one week to the next. If you imagine there's a natural fluctuation of rankings and Moz is taking individual snapshots that might be up one day and down the next so they're only really meaningful when tracked over a longer period of time. You're also not taking into account the long tail - bunches of keywords that are similar which may not be fluctuating nearly as much as the one you're tracking.

The real indicator of where you rank should be the organic traffic to the page. I doubt that traffic will be fluctuating even nearly as much as the positions for the few keywords you are looking at.

As for algorithmic negative impact - you would probably see significant drops across multiple tracked keywords if this was the case - and those drops would be sustained until you diagnosed and fixed the problem.

Regards,

George