Welcome to the Q&A Forum

In general, for link value to transfer either through 301s or canonicals, the content of the page needs to be nearly identical. See Cyrus' post for more. And canonicals are not always followed by Google, they are just a "hint", so it's unlikely you'll pass much value that way.

Hi Joshua

The settings look OK - what keywords were you ranking for that have now dropped? Also, when was the site launched?

-Dan

Hi Michael

I don't think this is a concern in terms of a penalty or anything severe. But I would say, if you'd prefer the other breadcrumbs to show up, perhaps try some breadcrumb structured data (unless you have already?) - to get the preferred ones to show up.

https://developers.google.com/structured-data/breadcrumbs

Hello

Ahh ok, missed that detail.

I created a quick video for you ---> http://screencast.com/t/IKkEikyr

I think this is a bit of a complicated situation which will be tough to diagnose and fix in a Q&A thread. I would suggest catalog the different settings of your site in a spreadsheet like I show in the video.

Essentially, the canonical settings are just "suggestions" for Google and not "directives" so they will ignore them if they think they have been set in error.

I would start by clearly defining the end result you want (what pages should be crawled, and what should be indexed) and work backwards from there to apply the right settings.

I would probably try to use noindex, robots.txt etc before resorting to a canonical.

FYI - to check if a page is indexed try typing site:http://perfectlinens.com/collections/all into the Google search bar, or cache:http://perfectlinens.com/collections/all into your browser.

Hi There!

That page is in fact indexed and cached for me! Can you check again? And let me know?

-Dan

Hey Andy

To answer your questions:

1. So if you're 301'ing the page, it's not really a 404 page, it's a 301 So yes, you can remove the 301 redirect, making it a true 404 page (check that it returns a 404 code using fetch as google or a tool like urivalet.com).

2. If they are in the sitemap, this won't prevent Google from removing them from the index, but it will throw an error. And not that many people care about Bing, but Bing is apparently super picky about having XML sitemaps perfect.

So yes I would just 404 them without the redirects.

Hi - right, then if the URL changes for the user, you'll want to probably use the PushState method (linked above) to convey this to Google. They likely can't see the URL change by default.

You can check by trying to crawl the site with Screaming Frog SEO Spider with the user agent set to Googlebot. Then go to "outlinks" for the page with the facet links, and see if they are listed.

Hope that helps some more! Let me know if you need further direction.

-Dan

Hi - right, I should have answered your specific situation too

When the user selects a facet - does this change the URL too? Meaning, it's supposed to be a totally different page?

Google recently updated how they prefer AJAX is handled:

https://webmasters.googleblog.com/2015/10/deprecating-our-ajax-crawling-scheme.html

They now recommend using the PushState Method - I won't pretend to know all the intricacies of how it works for implementation, but that's the best method to go with. If you need any more help let us know, and I'll have another associate jump in and take a look.

-Dan

I think you're having trouble because there's really three criteria to evaluate this on. Link authority, quality and relevance. In your situation you need to find a keyword where the current results are low in at least two of those areas. In other words - right now - you'e not going to win against high link authority and high quality. Let me break down each:

Link Authority - this you already know about. Domain Authority helps assess this. It's basically the ranking power of a site/page based upon backlinks.

Quality - this is how good the current results are at fulfilling the query with quality information. This can only be judged by a human eye - meaning - you have to look at each result as a topic expert and ask yourself - can I do better? Open times I'll see pages with high link authority but the content is low quality. This is still a hole in opportunity. Lack of quality can also be due to the fact it's not written by a true expert. Old content ranking can also be "low quality" as well.

Relevance - something can be high quality and have high domain authority, but just not be a relevance match. In other words - maybe the search is "Dominican Republic Weddings" but the result is about "Dominican Republic Events". This is a slight mismatch and also an opportunity.

Link/keyword tools can give you "link authority" but not many really measure quality or relevance that well - this is where your human eye comes in.

So... you need to find a SERP with a gap in at least two of those. Low DA and low relevance. Low relevance and low quality. You get the idea

I would suggest making a quick spreadsheet and mark each keyword with the volume etc (the usual metrics) but assign a quality and relevance score based upon your judgement.

Now - this isn't magic of course - your content has to be the best and you have to do some promotion etc - but it's definitely the approach I would take for choosing keywords.

Laurie

It should be clarified that Moz's Domain Authority, while a really solid metric, is not the metric Google has or uses. And domain authority can have a few artificial quirks. So I would not be alarmed at all.

That said - can you explain where you are seeing the two different number? I see a Page Authority of 39 for both http and https - and I see a domain authority of 27 for both http and https.

Now, even IF Moz has two different numbers for http and https, again, this is not what Google is doing, it's just an approximation.

Setting a canonical from https to http is just a band-aid and I would not recommend that approach. I would recommend having a site-wide 301 redirect so if a user lands on the https version of a URL it redirects them to the same version of that page on http. Or vice vera, whichever version you are prioritizing.

I have to respectfully disagree with Dmytro and Robert - as mentioned, Moz's metrics are not Google metrics - and the best action here is always to prioritize http or https with redirects.

Hey there

Dirk pretty much hit upon the issue, which I'll reiterate with a visual. If you enter any gibberish /blog URL (like this: http://primepay.com/blog/jglkjglkjg) in the browser it returns a 200 OK which, but it should return a 404 code --> http://screencast.com/t/cStpPB5zE

Otherwise pages that are really broken will look to crawlers like they are supposed to exist.

Escaped fragments are not recommended by Google anymore. Google now recommends Progressive Enhancement such as the History API pushState. See this article for details:

https://googlewebmastercentral.blogspot.com/2015/10/deprecating-our-ajax-crawling-scheme.html

Here's another post that you may want to check out:

https://builtvisible.com/javascript-framework-seo/

-Dan

Hi Joshua

I don't think this will cause a search engine conflict, but I'll pass this to a Moz product person to see if they can answer re: Moz On-Page Grader

-Dan (Moz Associate)

Hi Luke

Matt has the right idea. If the pages are going to "exist", you should block search engines from crawling them with the robots.txt file.

I would get your dev to help, but basically you'd find the folder or path in which you want to crawler to stop at. Maybe it's /month/ or something and you'd block that in robots.txt.

Ian covers this in his recent article about "Spider Traps". And you can also read about robots.txt on Moz or on Google.

No problem. Screaming Frog (or any crawler) won't pick it up, because it's not being linked to within the website (it's an "orphaned" page).

Google could still index them because they are in the sitemap, but it took so long because they are no actually linked to from the website.

So... if it's not supposed to be indexed at all in the first place, you can add a meta "noindex" tag to the page and remove it from the sitemap. Then you'll be all set

It was probably indexed so late because Google couldn't find it

I just crawled the whole site with Screaming Frog and that URL wasn't picked up in the crawl --> http://screencast.com/t/xzunkNR3K

But it's in your sitemap --> https://www.policygenius.com/blog/post-sitemap.xml - so this makes total sense why it took Google so long to find it

Great. It really sounds like they should keep the sub-categories

Hi There

This depends on a few things I think:

• How many products on average are there in each sub-category? Maybe there's only a few, in which case this makes these subcategory pages less useful, and the products themselves better to rank.
• Is there a lot of search volume around the subcategories? They may be ranking, but is there enough traffic to warrant keeping them?
• Will the amount of products in each subcategory grow in the future? In other words, if there's only a few now, but it will be increasing, you'd probably want to leave them so there's room to expand.

Chris has a good point too - if the sub-category pages are useful for searches to land on after a search query, this is good reason to keep them too. But it of course does depend a bit on the variables mentioned above.