Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
That's the staging URL for GoDaddy-hosted sites, Ryan. It likely means that when the site was being developed under the hosting account's temporary URL, it got indexed by the search engines.
It's also possible there's a domain resolution issue on the hosting server. Are there any other pages showing this problem?
The reason its display is broken is because the CSS and JavaScript on the page will be getting called from the wrong URL, so they can't resolve and format the page content.
Paul
I'm not seeing the video there anymore either, Nick.
P.
Knowing that with a large body of documentation like this, the chances of being able to rewrite it all to combine into a single page are pretty slim (and knowing that might be a very negative user experience) you're really only left with the canonical tag option - assuming the older docs need to be maintained.
You're right to be concerned, as Google has been clear that canonical only applies to pages that have substantially identical content. Unfortunately, they do a really poor job of explaining just how much variation would be allowed.
Is it okay if the canonical is not an exact duplicate of the content?
We allow slight differences, e.g., in the sort order of a table of products. We also recognize that we may crawl the canonical and the duplicate pages at different points in time, so we may occasionally see different versions of your content. All of that is okay with us.
~ https://webmasters.googleblog.com/2009/02/specify-your-canonical.html
My impression is that they would honour canonical in your use case.
Really, the only way to know is to select a couple of products' documentation pages and conduct a test. Canonicalise all old version to the current version and request re-indexing for each page. Then monitor the results (The new index monitoring tools in the new GSC are useful for this). You'll want to choose at least one test case that involves featured snippets - it would be incredibly useful to know if the FS transfers across to the new canonical page!
Do note that you'll need an ongoing process for managing the canonicals s each new iteration of documentation is added - all related pages will need to have their canonicals updated to point to the newest each time new docs are published.
Interesting conundrum. Please let us know the results if you decide to try a test!
Is that useful?
Paul
Please don't hijack someone else's question, zecase. Much better if you start your own question as it's completely unrelated to the current one.
Paul
Absolutely do NOT change from the www just for the sake of saving four characters in the URL. It would mean that every single page on your site would now have to go through a 301-redirect.
You should only change URLs when absolutely necessary, and the reasoning for this change certainly doesn't qualify! 
Also worth noting - a www URL has more technical flexibility. Those reasons may not apply to you now but could in the future.
Paul
Your understanding of canonical tags is correct, GhillC.
If Tools are showing errors for those canonical tags you've listed, then the tools are wrong.
As long as the protocol and subdomain prefix (or not) exactly match and the only difference is the exclusion of the parameters (the "?" and the stuff after it) then the canonicals are correct.
Any tool's reports have to be filtered through your own understanding and knowledge. They often get things wrong. That's on eof the key differences between experienced SEOs and less-experienced. They kow when to question what an automated tool is telling them. So good on ya for questioning the results!
Paul
Another +1 for Screaming Frog's Log File Analyzer. (It's a separate tool from their superb Crawler tool.)
Paul
Direct traffic isn't a ranking factor, VT. There's no way it can be, as it would far to easy a signal to fake and abuse.
Paul
A proper canonical tag does a lot more than "just be telling Google not to rank it" When used properly (i.e. pages that truly do contain the same content), the canonicalised page passes its ranking signals back to the canonical source.
I agree with Kristina - while a 301 would be preferable (it's a directive, while canonical tags are taken as suggestions), a canonical tag would be vastly better than not doing anything about the issue. At least until the dev can get the problem with the 301-redirect properly resolved.
Paul
My suspicion would be that those URLs are accessible somewhere without the second variable. (I.e. That second variable isn't always present when the first variable is.)
There is no individual tool capable of providing the info you're looking for, Seth. At least as far as I've ever come across.
HOWEVER! It is possible to do it if you are willing to do some of the work on your own to collect and manipulate data using several tools. Essentially this method automates the approach Takeshi has mentioned.
The short answer
First you'll create a list of all the pages on your website. Then you'll create a list of all the URLs that Google says are indexed. From there, you will use Excel to subtract the indexed URLs from the known URLs, leaving a list of non-indexed URLS, which is what you asked for.
Ready? Here's how.
Collect a list of all your site's pages You can do this in several ways. If you have a reliable and complete sitemap, you can get this data there. If your CMS is capable of outputting such a list, great. If neither of these is an option, you can use the Screaming Frog spider to get the data (remember the free version will only collect up to 500 pages). Xenu Linksleuth is also an alternative. Put all these URLs into a spreadsheet.
Collect a list of all pages Google has indexed.
You'll do this using a scraper tool that will "scrape" all the URLs off a Google SERP page. There are many tools to do this; which one is best will depend largely on how big your site is. Assuming your site is only 7 or 800 pages, I recommend the brilliantly simple SERPS Redux bookmarklet from Liam Delahunty.
Clicking on the bookmarklet while on a SERP page will automatically scrape all the URLs into an easily copyable format. The trick is, you want the SERP page to display as many results as possible, otherwise you'll have to iterate through many, many pages to catch everything.
So - pro tip - if you go to the setting icon while on any Google search page, and select Search Settings you will see the option to have your searches return up to 100 results instead of the usual 10. You have to select Never Show Instant Results in order for the Results per Page slider to become active.
Now, in Google's search box, you'll enter site:mysite.com as Takeshi explained. (NOTE: use the canonical version of your domain, so include the www if that's the primary version of your site) You should now have a page listing 100 URLs of your site that are indexed.
- Click the SERPRedux bookmarklet to collect them all, then copy and paste the URLs into a spreadsheet.
- Go back to the site:mydomain results page, click for page 2, and repeat, adding the additional URLs to the same spreadsheet.
- Repeat this process until you have collected all the URLs Google lists
Remove duplicates to leave just un-indexed URLs
Now you have a spreadsheet with all known URLs and all indexed URLs. Use Excel to remove all the duplicates, and what you will be left with is all the URLs that Google doesn't list as being indexed.
Voila !
A few notes:
Whew! I know that's a lot to throw at you as an answer to what probably seemed like a simple question, but I wanted to work through the steps for you, rather than just hint at how it could be done.
Be sure to ask about any of the areas where my explanation isn't clear enough.
Paul
Sorry, I have to categorically disagree with Sangeetac here. Kimberley, your understanding is entirely correct.
There's so much misinformation and misunderstanding about canonicalization, and there doesn't really need to be.
The rel canonical tag is ONLY to be used when there is a justifiable reason why the multiple duplicate pages in question should actually be reachable via different URLs. Unless such a reason exists, you should ALWAYS use a 301 redirect.
Put another way, duplicate page issues should ALWAYS be resolved using 301 redirects unless doing so would harm the user experience.
In the case of multiple versions of a home page, there is absolutely NO BENEFIT to the user to be able to reach that page via multiple different URLs. (In fact the multiple URLs could be confusing.) Therefore, the 301 redirect should be used to FORCE the user (and search engine) to the single version the site owner has decided should be the primary URL. Again, the other URLs are pointless, so the user should be blocked from ever being able to reach them. Using the 301 in this case has the added benefit of explicitly and automatically telling the search engines that all the redirected URLs for that page should be dropped from the index as they are non-functional. (And that's exactly what happened, Kimberley, as you saw default.asp and index.htm disappear from the index.)
The ONLY time you'd use a rel canonical tag in this case, instead of a 301 redirect, is if for some reason the site owner doesn't have sufficient access to the sever files to be able to implement a 301 redirect properly. In that case, a rel canonical tag can be used as a "better than nothing" solution.
The classic example of when to use a rel canonical tag is in the case of a page that lists a large number of products. That default (and therefore canonical version) version of the page may very well list the products in the order they were added to inventory. There may well be additional versions of the page, each at a different URL (often using query parameters, for example) which lists exactly the same content, but one page sorted by size, another version sorted by colour, yet another sorted by price.
Obviously in this case, 301-redirecting those additional dupe pages to the original page would be harmful to the user experience, because the user would never be able to access the content sorted the way they want to see it. So each duplicate version of the page has value to the user and therefore MUST remain available to the user, and therefore to the search engine.
In this case, the site owner adds a rel canonical tag to each of the dupe pages pointing to the primary (canonical) version, to make it clear to the search engine that the dupe pages are intentional secondary variations that should pass all their ranking value back to the primary page. But NOTE! Google themselves say they consider the rel canonical tag as a suggestion only, and will ignore it if they deem it incorrect or manipulative. So it is NOT NEARLY as strong an indicator as a 301 redirect which forces both user and search engine into the desired behaviour.
Sorry if that got a little long-winded, but once we get clear on the basic purpose of 301 redirects vs rel canonical tags, it becomes much more straightforward which to use when. The confusion usually stems from folks trying to justify using canonical tags for purposes for which they were never designed. They are not a panacea or a catch-all for fixing site architecture mistakes.
Hope that clears, rather than muddies the waters?
Paul
I'm going to have to offer a different opinion than Chad, njam. There's nothing about tags that is intrinsically beneficial for SEO, and in fact misusing them can cause considerable SEO problems.
Bottom line, the primary purpose of tags is as an additional navigation tool for your visitors, and must be carefully implemented so not to damage your site's SEO. There's nothing magic about a tag - it's a link to your content organized in different ways
Tags are part of what's called your site's taxonomy - i.e. how it organizes its information. Tags allow the visitor to get listings of your existing content grouped by different topics - much like categories do, but on a more granular scale. And there's the kicker - tag listings are just your regular content, but grouped differently.
Because they're the same content as exists elsewhere on the site, they can easily be seen as duplicate content by the search engines, leading to a dilution of the ranking power of the actual posts. (This issue can also occur with category listings if they're not handled correctly, which is why you're already no-indexing them).
Most WordPress SEO specialists actually recommend that tag pages be no-indexed - a command that tells the search engines to not even include them in search results as they can so easily confuse the search engines about which version of your site's content is most important.
So... use tags on a website if you think they will significantly add to the visitors' ability to understand and navigate your site. Then, make sure you've taken steps to ensure they don't confuse the search engines.
Do NOT use tags just because they're available. If you don't have a clear strategy for using them, don't clutter your site with them.
If you want an idea of how tags fit into a taxonomy, here's a real-work analogy:
Think of your blog as a book on raising kids, for example. The book would have a bunch of separate chapters on things like childbirth, schooling and education, care and feeding, discipline, family finances - you get the drift.
These are the equivalent to your blog's categories - broad topics that would allow someone to pick up your book and leaf through it, immediately understanding what the book's about, and enabling them to read just the sections that apply to them.
In the back of the book is the index. It's full of words and phrases that you have used a couple times in the book, but aren't topics that deserve a whole chapter (category). These are the equivalent of your tags. For example, perhaps you mention peanut butter in the care & feeding chapter. Later, you also mention in the schooling chapter that parents shouldn't send peanut butter sandwiches in kids' lunches because of the allergy risk to others. In addition, you mention PB in passing in the finance chapter as a cheap source of protein for kids meals.
Obviously there'd be no point in having a whole chapter (category) on peanut butter, but a reader might find it useful to have the index (tags) at the back of the book show them all the places in your book you've mentioned peanut butter.
That's a bit of a facetious example, but hopefully it gives you an idea of the different uses? If not, lemme know and I'll try to clarify further. Just remember, there's nothing magic about tags for SEO and they in fact carry some risk. They are for human visitors. If the visitors don't find them useful, they shouldn't be there.
Paul
It's not unusual at all to experience this kind of minor, temporary drop, Gordian. The search engines will need to re-index the changes in content and URLs created by your redesign, and this can easily take a week or two.
The critical thing is to be certain you have effective monitoring processes in place to make certain you catch any issues that the redesign might have created just a soon as they occur, so you can fix them before any long-term damage is done.
The idea is that yes - it's natural to see a small, temporary drop after a redesign. But you want to be certain that the drop isn't being caused by a correctable technical issue. Hence the need for close monitoring, even if assuming it's just temporarily due to new site design.
Hope that all makes sense?
Paul
P.S. Getting some new, quality, authoritative links to the newly designed pages can really help too. Social Media, especially Google+ can be really effective for this.
There is only one way the use of Hootsuite could negatively affect SEO - if you choose the ow.li URL shortener as your default instead of the default ow.ly version (in the setup section).
The ow.li version is using 302 redirects instead of 301s which is a no-no for SEO.
Otherwise, since the dashboard is only posting contents to Twitter/Facebook and not your website, there's no possibility for harm.
HOWEVER!!! Using Hootsuite (or any 3rd-party tool) to post to Facebook can definitely cause user-engagement problems. Any update posted to Facebook using a 3rd-party tool (like a dashboard) automatically has a harder time earning views by your followers. Facebook's Edgerank algorithm is what determines how many of your followers might see your post, and posts by 3rd party tools always have a lower Edgerank.
So for all important Facebook posts, it is best to submit them using Facebook's own site or their mobile app rather than Hootsuite. (You can now pre-schedule posts from within Facebook itself, so this helps alleviate one of the reasons why most people used 3rd-party tools) In fact, I still believe it's better to make all Facebook updates directly.
Paul
Thomas, as Takeshi has tried to point out, you have misread the original question. The original poster is asking for a way to find the actual URLS of pages from his site that are NOT indexed in the search engines.
He is not looking for the number of URLS that are indexed.
None of the tools you have repeatedly mentioned are capable of providing this information, which is likely why you're response was downvoted.
Best to carefully read the original question to ensure you are answering what is actually being asked, rather than what you assume is being asked. Otherwise you add significant confusion to the attempt to provide an answer to the original poster.
Paul
You could always just check the source of the page and count the hrefs to know how many links you actually have, Erin
Easier way would be to use the SEOMoz extension for Chrome or Firefox. It has a button for Analyze Page (looks like a magnifying glass) which has a tab for Page Attributes. That will show you the total count of internal and external links on a page. (It just does the manual counting for you, essentially)
You could also use a standalone page like http://linkcounter.submitexpress.com/ but the SEOMoz plugin/extension is so useful for many things that it's worth installing anyway.
Sounds to me like your first crawl didn't get through the pages thoroughly, but you're right to be careful. Especially in view of possible spam links, as you mentioned.
Hope that helps?
Paul
You're in a delicate spot, Bob. I'd say your plan should be to "hope for the best, but plan for the worst".
Obviously, as you indicated, you're going to need to do something about those links as that link profile is just begging to get hammered.
You could clean them all at once, take the traffic hit, and then try to build back as quick as possible, but if the site is doing well now, it seems a shame to take such a hit.
I'd suggest putting a clear, well-prioritized, well-funded plan in place to start building link-worthy content and promoting it in ways that earn those backlinks as quickly as possible. (This work is going to have to be done regardless, so not like it's a temporary expense).
Then, for every 6 or 8 new quality incoming links, clean up one of the 10 problematic links. This will look natural to the SEs (as it is natural) and hopefully won't attract the attention of the slappers while you're working through the process.
Best case scenario, you'll get through offsetting all the problem links without getting hurt by a penalty or algo update.
In order to be ready in case of the worst-case scenario, (Google slaps the site with a penalty a week from now), you should also immediately build a confirmed contact list of the webmasters in control of the problematic links. (I mean an email or phone number that you've confirmed actually gets a response from a human). That way if you get hit before you can clean up naturally, you can get those problem links dealt with immediately and can show Google what you've done in a quick reconsideration request.
Also, document the process as you work through attracting the new links, so you can be specific about what you've been doing in that direction, should a reconsideration request become necessary
As far as how to deal with the problem links - do not submit a disavow!! That is a last-ditch process if there's no other way to get links removed, which is not your case. (Plus the disavow process could attract unwanted attention. Yea, I'm cynical like that
I'd actually suggest a mix of tactics for those 10 sites, depending on different circumstances:
Does that approach sound like it might work?
Paul
The main reason PageSpeed and YSlow recommend including width and height for images is as much the perception of page speed as the actual load time, Ben.
When you include the image dimensions, the browser can draw out the "containers" that will hold the images, reserving the space for them while they download. The browser can then go on the paint the rest of the pages CSS and objects around those "containers" without having to go back and redraw the whole page once the images have downloaded and their sizes are then known.
This gives the user the illusion of a much faster, cleaner page load, and hence the impression of a faster website.
Does that make sense?
Paul
[Edited to add: You should still keep doing what you're doing to produce "size-as" images for your pages. You don't want to be resizing images with the html dimensions, just reporting in html the actual size of the image]
The big benefit to having the redirects written into htaccess is that htaccess will run them much more quickly (and with lower server overhead) than from inside WordPress. If you only have a few redirects at a time to correct for moved pages or creating short URLs for marketing campaigns, doing it within the Redirection plugin is fine.
But if you're writing a large number of redirects (to handle a site move for example) you're far better off writing them into htaccess.
Paul