Welcome to the Q&A Forum

If you use 301 (permanent) redirects, there's no need for canonicals on the old pages because as far as the search engines are concerned, you're telling them the old pages no longer even exist. In fact, done properly, you can actually remove the old site altogether after a month or 6 weeks of the redirects being in place. (You point the old domain to the new site and place the redirects in the new site's .htaccess file)

This kind of move can be a big deal for a site that's been around for 10 years. Hopefully there's a compelling reason for wanting to change the domain name? Business rebranding, for example?

Paul

I think the root of your problem comes from a common misconception about the robots.txt file, Lewis.

A robots.txt no-index directive is NOT designed to get pages removed from the search index. It simply tells the crawler: "when you encounter this directive, don't crawl any further". So the crawler never even gets a chance to discover whether there are any further pages, never mind whether they might be in the index already

THEREFORE! Any pages that are already in the index will simply stay there. (And if any outside sources have links to internal pages behind a robots.txt no-index directive, those linked pages' URLS will often be added to the search index anyway!) Any pages which are in the index this way will have their meta-descriptions blocked from displaying by the robots.txt directive, as you are seeing in your case.

Since a robots.txt no-index directive stops the crawler from looking any deeper, the engines are blocked from actually discovering the 301 redirects on your dev pages, and so aren't getting the cue to drop them in favour of the new pages! Hence the dev site stays in the index and shows up in SERPs. The human user does get the redirect so ends up on the new page, but you still have the duplicate content/competition problem.

NOTE: to actually tell the search engines not only to not index the page, but to remove it if it already exists, you must add a meta-no-index tag in the header of individual pages. The robots.txt no-index MUST NOT be in place in order for this tag to be discovered and obeyed. There is an automatic setting in WordPress Settings -> Reading page to disallow crawling which automatically adds the meta-no-index tag to each page's header

Unfortunately, the problem is bigger than you stated, as I'm finding almost 70 pages from the dev site indexed in the.co.uk SERPs

Here are what I see as your two main options, along with their ramifications:

1.  Remove the robots.txt no-index directive and allow the 301 redirects to be crawled, eventually causing the dev pages to drop out of the SERPS

• this would be the preferred option if the existing dev site pages have actually started to acquire incoming links and ranking value, but you'd have no control over how long it would take for the competing dev pages to drop out of the index, meaning they will continue to interfere with your SEO until that process completes
• you'll need to check whether any of the other 70 pages in the results have incoming links and if so 301 redirect them as well
• you'll need to add meta-robots no-index tags to the header of each of the remaining non-redirected pages on the dev site to get them removed from the index.

**2. ** Use the URL Removal Tool in Google and Bing Webmaster Tools to have the dev site removed from the index

• likely the fastest way to get the competing URLs out of the indexes, but would mean that any acquired link authority from the dev pages would be lost, not transferred to the live site.
• would still require either the robots.txt no-index directive to stay in place, or better yet, remove it and replace it with meta-no index tags in the header of every page on the dev site.
• you'd need to remove the 301 redirects
• since the search engines consider subdomains completely separate sites, you'd need to set up and verify the dev subdomain as a separate site in both Google and Bing webmaster tools in order for the URL Removal Tool to work.

I've never actually used the URL Removal tool on a full subdomain before, but see no reason why it wouldn't work as expected. You could actually test it out first on your dev.birdybanners.co.uk/ site as it has the same problem of the dev site being indexed in the SERPs.

Hope that helps give you a strategy to resolve the problem? Be sure to holler if you need me to better clarify anything.

Paul

Welcome, Eblan. Also, don't forget to use the Change of Address tool in Google Webmaster Tools as an additional signal to Google for the domain change.

Paul

This situation is quite easy to resolve if you are using the Yoast WordPress SEO plugin.

Under the plugin's settings, go to the Permalink Settings page. The second section down offers the option to "Redirect attachment URLs to parent post URL." Put a checkmark in the box, click Save, and from now on, those individual image-only pages will automatically redirect back to the post on which they appear.

No idea if the same option is available in other SEO plugins - I've so long standardised on Yoast's plugin I'm not up to date on the others any more.

Brett's point about changing where images link will also work, but the plugin approach above means you don't have to go back an retroactively edit all your images.

Hope that helps?

Paul

I think the solution to your server resource issue is to create multiple sitemaps, Aaron. Given that the sitemap protocol only allows 50,000 URLs max. per sitemap and Google News sitemaps can't be over 1000 URLs, this was going to be a necessity anyway, so may as well use these limitations to your advantage.

There's a functionality available for sitemaps called a sitemap index. It basically lists all the sitemap.xmls you've created, so the search engines can find and index them. You put it at the root of the site and then link to it in robots.txt just like a regular sitemap. (Can also submit it in GWT). In fact, Yoast's SEO plugin sitemaps and others use just this functionality already for their News add-on.

In your case, you could build the News sitemap dynamically to meet its special requirements (up to 1000 URLs and will crawl only last 2 days of posts) and to ensure it's up-to-the-minute accurate, as is critical for news sites.

Then separately  you would build additional, segmented sitemaps for the existing 200,000 pages. Since these are historical pages, you could easily serve them from static files, since they wouldn't need to update once created. By having them static, there's be no server load to serve them each time - only the load to generate the current news sitemap. (I'd actually recommend you keep each static sitemap to around 25,000 pages each to ensure search engines can crawl them easily)

This approach would involve a bit of fiddling to initially set up, as you'd need to generate the "archive" sitemaps then convert them to static versions, but once set up, the News sitemap would take care of itself and once a month (or whatever you decide) you'd need to add the "expiring" pages from the News sitemap to the most recent "archive" segment. A smart programmer might even be able to automate that process.

Does this approach sound like it might solve your problem?

Paul

P.S. Since you'd already have the sitemap index capability, you could also add video and image sitemaps to your site if appropriate.

The plugin is working on your site and is handling all of the meta-data except the page title, Arthur.

Many themes aren't coded to handle meta-titles in the standard way so plugins can control them. There's a setting in Yoast's plugin that can be used to force the rewrite of the titles.

If you go to the plugin's settings and click on the Title and Meta page, there's a check-box at the very top of the General tab called Force Rewrite Titles. If that's not selected, try putting a check-mark in the box. Then clear the WordPress cache if you;re using one, and clear your browser cache before reloading the test page.

Let us know if that works?

Paul

PS this assumes you've also turned off the SEO settings in the theme, which it looks like you have as Yoast's plugin is already setting the other metadata.

AHA! Figured it out. You're using a responsive slider, Rahul. That means the slider (and its images) get resized depending on the size of the browser window.

Those of us seeing blurry images are using a larger browser window, causing the images to be up-sized and going blurry.

To get around this, you're going to have to use larger images for the slider to keep the bigger browser windows sharp. (This is one of the classic tradeoffs of responsive design - the files sizes actually have to be bigger, even though they may get displayed on a small phone.

You'll need to arrive at a happy medium, and your analytics can help here. Ideally, you want to check the average browser window size for desktop users, but that takes some extra customization  Instead, at leas look at the average screen size of your desktop users and make an estimate of the typical browser window they use. Then check what size the slider images are for those screen sizes. Then redo the images to look crisp at that size. Compress them as much as possible while still maintaining quality, so you don't kill the file size.

That way, small screens will get the image as compressed as possible, but large screens will still see a large enough version of the image to be crisp.

Hope that makes sense?

Paul

Agree w/ what Jesse's saying here, but need a little tidy on his description of how to look at how Google sees the addresses differently. You'll need to Google www.whoopsee.com and then whoopsie.com (note the .com needs to be included both times). Otherwise you're doing a keyword search, not a domain name search.

In fact, the best way to do these is to do a search for site:www.whoopsie.com and then site:whoopsie.com. That will limit the search results to actual URLs from only your own site.

Paul

In addition to the speed issue, 301-redirects function just like a regular link - that is to say they lose a little bit of "link juice" in the transfer. So definitely fix the incorrect links to point directly to the correct URL. No sense wasting any "juice" unnecessarily.

My $0.02

Paul

That tool that Matt mentioned looked interesting, but it would have been painful to have to go through your site one page at a time.

As usual for crawling tasks like this, the paid version of Screaming Frog will do what you want. You can tell it to crawl your site looking for **href="yoursite.com **to find all occurrences of absolute internal links. You'd have to do a bit of regex magic to get it to find the relative links, but since by their nature a relative link will work even with the domain change, not sure why you'd be looking for those.

Or you could just do a find and replace of the URL string using something like phpMyAdmin directly in your database. That would be fastest as it would find & replace in one go, instead of having to manually edit each page.

Is this a WordPress site, there's a plugin specifically for finding and automatically updating these links. (It basically automates and puts a UI on the phpMyAdmin process mentioned above.)

Any of those ideas help?

Paul

The answer to this is really going to be dependent on the page content, Micelleh. A simple page with a clear call to action could result in a user getting exactly what they want from a page within a few seconds and then leaving. A 350 word page might mean 45 seconds, but a 1500 word page might need 2 minutes to prove a user actually got value.

At best, if you insist on a value, get several users to use a good number of your pages, record their on-page time, then create a site-specific average from that.

However, you might be even better off using events for this process, instead of something nebulous like dwell time.

You could add event tracking to the amount of the page a user scrolls, and if they scroll more than half a page (for example), an "interactive" event triggers. "Interactive" events have an effect the same as another pageview (without screwing up your pageview metrics) so a single page visit that scrolled at least half way down the page would no longer be recorded as a bounce.

You could also create interactive events for things like pdf downloads, form submissions, sending emails, viewing a video etc that you consider appropriate for your site to negate what would be considered a bounce.

The biggest benefit to this events-based approach is that it would be vastly more accurate. It would  track visitors' actual actions, as opposed to just assuming a dwell time meant a valuable interaction. (For example, we all know that the habit of opening multiple tabs at once for sequential reading significantly over-inflates time on page for many users.)

Perhaps that idea would work better for what you're trying to accomplish?

Paul

Your site is not handling "page not found" errors correctly, Madlena. Currently, when a user requests a URL that doesn't exist, your site is sending a 302 (temporary) redirect to an error page, but that error page's header is responding with a 200 status that means "page was found correctly".

You need to have your site's developer change that behaviour so that when a non-existent URL is requested, the server immediately returns a page that has a 404 not found header response (and no 302 redirect). This is necessary even if the page itself looks like a 404 error page.

ONLY by having the page return a 404 response code IN THE HEADER will that tell Google that the page doesn't actually exist and to drop the URL from the index. As Google themselves state - it’s not enough to just create a page that displays a 404 message. You also need to return the correct 404 or 410 HTTP response code.

Once this is configured properly, it's easier to keep track of these 404 errors so that you can fix them by manually 301-redirecting to a suitable page if there is one. (although this may not be the case with just images, it can be very useful in other cases.)

Hope that helps? If anything's not clear, just holler!

Paul

In pretty well all cases, the crawlers crawl code only, Stephanie. It would be too computationally expensive (both for your server and theirs) for them to render every page before crawling it.

The Fetch as Googlebot tool in your Google Webmaster Tools account will show you exactly what the Google crawler sees. When you us the Fetch tool, the result will show a line indicating the success of the fetch. The word Success (in green) is actually a link to a view of the page exactly as Google see it.

That make sense?

Paul

You shouldn't need to change the IP address twice. I'm pretty certain that if you tell the SiteGround folks you want to upgrade your plan and purchase a dedicated IP address at the same time, they'll apply the new IP address when they set up your new hosting. So you can move directly from your current IP address to the dedicated one.

Changing IP addresses will have no effect on your rankings, as long as both addresses are listed as hosted within the same country (as will be the case if yo're buying the new one from the same host as you currently use).

The one thing a dedicated IP can do for you is help insure you're not sharing an IP address with other potentially spammy sites. This can help protect you from "bad neighbours" and may also help protect your email from getting blacklisted due to sharing the IP address with other possible spammy senders.

Contrary to what many may wish to believe, having a faster site will have essentially no impact on your rankings, unless your old hosting was terribly slow. A lot was made of the announcement that page speed has become a ranking factor, but careful reading shows that it's only a minor factor that only applies to a small subset of queries and even then only when it's significantly slower than it's competitors.

The big reason to improve your site's speed is to improve your users' experience. This can lead to more engagement on your site, and higher conversion rates.

One last note - your English is terrific! My second language is French, but I sure couldn't do SEO research in that language. I'm always impressed by people who can do technical discussions in a second language.

Hope that helps. If anything's not clear, just ask and I'll try to go over it better.

Paul

The problem here,. as I see it, NY60, is that there's no way of knowing for sure. The engines are "supposed" to be able to parse the text strings, but there's no way they're infallible and in fact there's no way of knowing if they're even acceptably good at it under the conditions that are important to you.

For that reason, I always opt to make it as hard as possible for the engines to goof. In this case, that means dividing at word boundaries with a hyphen in strings like URLs and other code. Spaces are problematic because they often get encoded into html entity %20 which can cause yet more havoc - though in straight content like meta-titles, meta-descriptions, alt text and page content they are fine.

There's my $0.02. Whattaya think?

Paul

Glad that helped Dario. I'm checking in with a connection I have at SiteGround to see if he can find a way to make your upgrade directly to a dedicated IP on the new account. Will let you know what I find out.

Paul

As Lynn mentions, these entries form a blacklist for "bad bots". These are bots that are identified as being harmful (or at least non-helpful) to the real use of a website. Bots are essentially spiders that crawl and record the pages of your site the same way the GoogleBot does.There are 2 main reasons for blocking them

1. Too many unnecessary bots can put a real strain on server resources, causing the site to slow down for real users. This can be especially problematic with bad bots as they do not respect the entries in your robots.txt file and so will crawl even blocked pages. This can mean huge numbers of extra pages get crawled, leading to even more load.

2. Many (most?) of these bots are collecting data for nefarious purposes. Some are scrapers to collect your site content in order to re-use it illegally on another site, some are scanning for certain files/plugins on your site known to be insecure so they can target them for attack, etc.

Best case scenario, these bots waste your bandwidth and can cause site slowdowns on low-powered (e.g. shared) servers. Worst case, they can actually cause harm to your site.

There are literally many thousands of these types of bots out there, and their creators often change their identifying user agents just to get around these types of blacklists. But many have been around for some time and still use the same identifier. So having a blacklist to block the most common of them is actually very good security practice. To be totally proactive however, you'd need to update the list every couple of months.

Bottom line - those entries are providing some security and overload protection for your site, and there's essentially no downside to having them in place even if they're not catching everything.

Hope that helps - if any of my explanation isn't clear, just holler

Paul

Don't think that link you posted is the one you meant, Miriam. That product is the actual Google server appliance that you would install in-house to drive extremely sophisticated search functions for your company's internal documents (and can also be configured to index your web site.) and would be overkill for your purpose.

Weren't you actually looking at Google Site Search? It's the one that starts at $100/yr to remove Google ads & branding from their default internal website search, and allows you to apply your own styling, to bias the search for recency, and even to insert your own promotional results into autocomplete on your site's search engine. It starts at $100/yr for 20,000 search queries.

And yea, that page's example search functionality is borked for me too - not sure what's going on there. The product itself works really well though. It does only need html/css to add your own look & feel. The addition of labels, customizations, promo results etc is all done through check boxes and text fields on the admin dashboard.

The one challenge can be that you're depending on Google to do the indexing. You can trigger a request for indexing yourself, but if the site gets a lot of new content added really frequently, there can be a delay (up to 24 hrs) before that content is available to internal search. Note that this process in no way affects, for better or worse, your Google SERPS indexing. The two processes are completely separate.

For your specific situation of returning both product and info pages, you'd need to work with the label functionality of Site Search. The built-in synonym capability is also a huge improvement over most default site search.

Overall, I've found this a great solution for websites that are either built on a framework that doesn't have a quality site search built in or plugin, or for sites that are a combination of several technologies whose built-in search isn't capable of indexing each others' technologies. (An indexer that can directly access your own site's database is usually faster & more configurable, but takes more coding, and puts additional load on the site's own server.)

That the kind of info you're looking for?

Paul

I call this a "belt & suspenders" situation, Taysir. If you don't want your pants to fall down, you put on a belt. If you want to be really, really sure your pants can't fall down, you put on a belt and then wear suspenders as well

The critical item is to have the correct 301 canonical redirection to a single version of your URL as you have done. Doing it this way communicates that information to all the search engines. (Setting a preferred domain in Google Webmaster tools only communicates to Google, not Bing or others.)

That said, since it only takes a few minutes to add the preferred domain settings in both Bing and Google Webmaster Tools, there's really no reason not to do it as well, as you're just giving yet another indication to back up what you redirect is telling them.

The "preferred domain" setting in GWT is primarily there to help out webmasters who are blocked by their hosting from creating the correct redirect rules. But as I say, no harm in being really, really sure

Does that make sense?

Paul

She clearly states "Content, filenames, titles, urls etc. will all remain identical just the change of a template." so redirects don't enter into the equation here.