Welcome to the Q&A Forum

For the sake of balance, probably worth mentioning that I'm with David in that I've seen a robots.txt noindex work. It has been relatively recently used by a large publisher when they had an article they had to take down but which Google was holding on to. That's irrelevant nuance in this situation but I think David deserves more credit than he got here.

In terms of this specific fix I agree with Nigel - remove the Disallow and add a noindex (prompt Google to crawl the pages, with a sitemap if they don't seem to be shifting). You can re-add the Disallow if you think it's necessary but once all of the appropriate pages have a noindex tag they should stay out of the index and if they are heavily linked to on the site disallowing them could result in a loss of link equity (it'll stop with the link to the disallowed pages). So if you think you can achieve this with just a noindex you might want to leave it at that.

Forgive me if I'm misinterpreting but if your concern is that your top navigation is pushing things like your H1 further down the page, I wouldn't worry about that. Google understands that web pages have a number of different elements to them and it's very common for a page to have the top navigation above the H1.

I definitely wouldn't remove the navigation or move it further down the page - it's a key part of the user experience and if the site is difficult to navigate that's likely to have a negative impact on the bottom line, and search ranking as a secondary effect.

Hope that helps!

Hi there, interesting question!

So in terms of whether you are allowed to try to rank for brand name/trademarked keywords the answer is yes, absolutely. Google makes decisions about which sites it thinks are most relevant for a search and you don't have any responsibility to shy away from that attempt.

In terms of whether it's possible for you to rank for those keywords, that's actually kind of related to the point above. Google decides what should rank based on best user experience. If Google has really strong evidence that whenever someone searches a particular term they are looking for a specific brand it'll be very hard for you to break into that. However, as you've mentioned, there comes a time when a term becomes generic enough that users aren't necessarily searching for the brand, that's when you'll have more and more chance with pages using the term as a generic term. You can fairly quickly check by just Googling the terms and seeing what comes up. For example, when I search "spinning" the fourth text result is "Boom Cycle" - sounds like it doesn't just have to be a brand called "Spinning" for that term. If on the other hand you Google Apple - it's pretty clear Google thinks there's only one topic that's relevant as a result.

If it's a term you think your users will be searching for, create some content for it. If it's a stretch to think you'll rank, create something good but not terribly time consuming and go from there. If it looks like the only content showing up is about this brand, consider creating a post about the differences between that and what you offer, as a way to seem a bit more relevant for Google.

Hope that helps

To echo MichaelAMG - it is possible to install wordpress on a subdirectory rather than subdomain so you'd be able to host your blog at mysite.com/blog  instead of blog.mysite.com. For SEO reasons, subdirectory is the recommended way to go.

On https://moz.com/learn/seo/domain, Moz explain that search engines view subdomains differently than they do subfolders, which means that it can be harder for successful content in a subdomain to benefit the site as a whole. Companies who host their blogs on subdomains have started moving them to subdirectories and have quite quickly seen positive results, for instance Salesforce moved their blog from a subdomain to a subdirectory and their traffic doubled pretty immediately.

If you are installing wordpress instances on one domain it might be a good idea to keep an eye on your permalinks because if you have url rewriting turned on then the different installs can sometimes confuse each other. Just something to watch out for.

Hi there! There seems to be a bit of confusion in this thread between URL structure and Information Architecture. Having more folders in a URL doesn't reduce the authority but pages with more folders in the URL tend to be deeper in the sites linking architecture, which means they tend to have less authority because they aren't as close to the surface. The difference between internal links and url format is an important one. There's a blog post here which explains in more depth.

From my perspective, here are the benefits of having pages within folders;

• There is an opportunity to put more relevant keywords in the URL without stuffing
• Easier folder-level reporting in Google Analytics, Search Console etc.
• Some increased understanding for Google of how pages hang together - there is some evidence that Google uses folder structure for ranking before it knows much about the page for example.

In terms of managing authority for pages and signals of relevance I'd be looking much more towards the internal linking to those pages. I wouldn't rely on Google intuitively understanding the topical connection between two pages unless both of those pages target that topic or have relevant links between them. So for example, say you have two pages;

1. site.com/widgets
2. site.com/doodads

If those pages are both subcategories of trinkets you could reformat them to be;

1. site.com/trinkets/widgets
2. site.com/trinkets/doodads

Having "trinkets" in the url might help both pages rank for "trinkets" type keywords, like "doodad trinkets" for example. However, I wouldn't rely on this change to help Google understand that widgets are related to doodads - you can handle that much more effectively with relevant internal links between /widgets and /doodads that make the relation clear.

In terms of whether there is a risk to making this change - this is essentially a migration and definitely comes with risks associated, even if all of your redirects are 1:1 and direct. It'll take time for Google to find the redirects and new pages, and as a rule of thumb, link equity isn't passed perfectly along a 301 redirect so I wouldn't expect these new pages to just inherit the strength of the old ones.

I think it comes down to weighing up whether the benefits I listed above outweigh the risk of an in-site migration. If you think the keyword targeting opportunities will make enough of a difference then great but I wouldn't rely on url structure as a way to get Google to understand your site differently - the impact of internal links is going to be a far greater factor.

I wouldn't do that just to push keywords closer to the top of the page. What I'd aim for is what's most useful for users - are there a lot of elements in the nav that don't add to user experience? Navigation can help with link equity flow around the site because it's a way to have links from every page to certain other pages, but putting everything in the nav dilutes link equity and can have a negative impact on user experience.

Essentially, as I say, it's about prioritising the important stuff and being ready to cut the elements that aren't helping.

I'm not sure I agree here - I have seen a few cases where pages which aren't exact duplicates compete with each other for specific keywords and end up dragging each other down. Removing that competition helped push us from page 3 to around position 5.

That's not to say that this will definitely be a problem for you, I just couldn't say that it definitely won't. If I were you I'd use a rank tracking tool to monitor which pages are ranking for the keywords you care about, and how well. If you keep an eye out for cannibalisation then you can take quick action. If you end up just ranking better then you don't need to do anything.

Appreciate that's a bit of a noncommittal answer but hopefully it helps!

Hi David, I'm assuming there are no common structural differences between the urls you want to individually redirect and the ones you want to bulk redirect? And what is your set up? It may be that the simplest solution is updating your htaccess file with a list but htaccess needs to be treated with care.

Hi there, as effectdigital says above, Google has a tendency to append brand to the end of a title tag where it thinks it's confident of it.  I think the question here is - how much do you mind it being added to the end? I think unless it's causing active harm (which I think it probably isn't) probably most effective to just leave it and focus on other things.

Wish we had the answer you were looking for!

I think seoelevated has your answer - if you can overlay the text on the image as actual text that should tick all the boxes for you

Yea that's definitely tricky. I'm assuming you haven't taken out any load balancing that was previously in place between desktop and m. meaning your server is struggling a lot more? The Page Speed Insights tool can be good info but if possible I'd have a look at that user experience index to get an idea of how other users are experiencing the site.

A next port of call could be your server logs? Do you have any other subdomains which are performing differently in search console?

In terms of getting Google to crawl more, unfortunately at this point my instinct would be to keep trying to optimise the site to make it as crawl-friendly as possible and wait for Google to start crawling more. It does look like the original spike in time spent downloading has subsided a bit but it's still higher than it was. Without doing the maths, given that pages crawled and kilobytes downloaded have dropped, the level of slowdown may have persisted and the drop in that graph could have been caused by Google easing back. I'd keep working on making the site as efficient and consistent as possible and try to get that line tracking lower as an immediate tactic.

Hiya, my instinct here would be to update both if possible - canonicals are a hint rather than a command, Google will still crawl canonicalised pages and could decide to ignore the canonical if it doesn't think the pages are closely related enough.

Updating the titles etc. on the canonicalised pages can help confirm that they _are _the same content as the pages you're canonicalising them _to. _Plus, worst case, if Google decides to ignore the canonical, the page is at least optimised until you can take stronger action.

If those additional pages really don't have any additional value for visitors I would keep pushing the client to consolidate if you can. Even if they can't do it now, they could be thanking you in a year.

Good luck!

Hi there, in a similar vein to Kevin's comment - Google has said that subdomain/subdirectory isn't really a distinction they draw and that it's more to do with how much linking there is between the different parts whether they consider it to be the same website. That said, we don't have a good idea of what the threshold is or what historic data Google is using to make that call.

I would say this comes down to how people are using this subdomain. Do you get researchers landing on it from Google? Or is it something that they only use once they are already on your site? If you find enough researchers are coming to this subdomain from organic search then it's probably worth fixing it up because, entirely aside from the question of whether this subdomain is dragging your site down this could be a way to really strengthen your site and take advantage of a very popular resource to boost Google's respect for your site and help you rank for other things (not to mention giving your users what they want). If, on the other hand, you don't get users landing on this subdomain from organic search (you can check by going into Google Analytics for the subdomain and checking the source and medium of sessions landing on those pages) then it could be worth noindexing the subdomain. If, as you say, it's riddled with errors, you don't want Google to see it/waste time on it and users don't get there from search then by noindexing it you're saying "we don't want you to pay attention to this".

We can't know for sure how Google will treat the main domain based on this subdomain but we can plan a strategy based on use, which is often what Google tries to push webmasters towards.

When you say 1300 landing pages are coming online every night that doesn't mean 1300 new pages are being created does it? Based on the rest of your comment I'm taking it to mean that 1300 pages, which were already live and accessible to Google, are being updated and the content is changing if appropriate.

In terms of the specific situation I describe above, that should be fine - there shouldn't be a problem with having a system for keeping your site up-to-date. However, each of the below things, if true, would be a problem;

• You are adding 1300 new pages to your site every night

• This would be a huge increase for most sites, particularly if it was happening every night, but as I say above I don't think this is the case

• You are actually scraping key information to include on your site

• You mention an API so it may be that users are submitting this content to your site for you to use but if you are scraping the descriptions from some sites, and reviews from others that is what would be viewed as spammy and it seems like the biggest point of risk I've seen in this thread.

Hiya, there's an SEO Round Table article from 2009 where Matt Cutts and John Mueller said that Google is looking for an exact match to "nofollow" but that they would consider broadening it to also include the space if they felt the intent is clear enough. I haven't seen anything since and, while it's reasonable to assume Google have broadened the match, I wouldn't count on it.

That said, more importantly, in March this year Google announced a change to how the search engine treats nofollow links. Rather than being treated as a command (as they previously were) they'll now be treated as a strong hint. We know that Googlebot can ignore other hints (like canonicals) so it's quite possible that it is doing the same here. In this case, where you don't want something indexed, I'd rely on noindexing the page itself rather than nofollowing links. Noindexing the page should also help the situation you're in now where the cat is out of the bag, so to speak, and you need to get the page removed.

Hope that helps!

Hi jfishe1988, could I ask - do you have analytics on these pages? In your initial question you suggest that the pages don't have any organic value but in responding to Alick you say that lots of them get organic rankings.

As a couple of broader points:

• Google has given conflicting advice about how low-quality links are treated, Gary Ilyes has said that low-quality links are just ignored (which makes sense as a way to negate negative SEO tactics) so this may not be something you need to worry about if you aren't relying on these 100 odd domains to channel link equity to the site you want to rank. However, John Mueller has said that the disavow file still has value, as I say - conflicting advice.
• If the only purpose of these landing pages is for hyper-targeted PPC and you don't want any link equity that is coming from them in case it poisons the main site, you could consider adding a nofollow tag to those pages. That should mean that Google ignores all links from those pages to your main site. It would insulate your main site from any link-network based penalties but it would also remove any benefits the main site might currently be getting due to any backlinks those domains have.

Hi there, good question! I've run into a similar situation and had all the same concerns as you. I think my recommendations would be (in order from most preferred to least preferred)

• Convert pdf to full HTML page with internal links and ability to convert, redirect PDF
• Replace PDF with HTML page which has the PDF as the content but also contains things like the internal nav (sometimes this is the only possibility with limited resource)
• Create summary page and link to pdf with canonical on pdf page (just as you say)

Making the content of the PDF more easily readable by search engines (by turning it into a HTML page as in solution 1) would hopefully boost ranking potential as well as helping give users a better experience.

The only reason I put your solution as lowest on the list is because if we're creating a page then we may as well create the full page, rather than creating a page which kind of  gives the same information as the PDF but still relies on the PDF existing. You could find that the two pages compete and neither ends up ranking.

Hopefully that helps!

I agree with you both that "blog" in the urls is valuable in terms of segmenting Analytics data. The only thing I'd bear in mind is that changing the URL isn't actually just changing the URL. To Google - you are deleting one page and creating another, and it takes time to understand that the new page should replace the old page, even if there's a redirect in place.

Given that you've presumably already done this once and you're reversing it, the risk should be lower, but I'd still approach this change as though it's a migration. I'm not saying "don't do it" just "take precautions".

In terms of tracking in the interim - you could consider using a custom dimension to identify blog posts. Use Google Tag Manager to determine the page is a blog post then send the dimension along with the page hit. Not ideal but it's something if you find you don't have other options.

Good luck!

I agree with the other answers on the following points;

• Moz DA is not a Google metric
• In territories like Thailand and spaces related to drugs, Google tends to have less of an idea of what is spammy or not
• On-page optimisation across your site is a great way to improve your rankings
• Giving your users the best possible experience is a great way to get long-term success.

However, I wouldn't go as far as to say that Google pays no attention whatsoever to links. I think you can treat these findings as frustrating but informative - from what you are seeing, that site having a worse backlink profile is not preventing them from ranking. So for those searches, backlinks don't seem to be as big a deciding factor. From there I'd start identifying what factors are correlated with rankings and, provided those things are above-board and won't come back to bite you (here I mean - don't buy links etc.) then try emulating that stuff while you continue to build and improve your own site.

My new favourite phrase is "You are the Igniterman"

Wow, this is pretty thorough, good effort!

Chantelle, you mentioned that you did a http -> https migration, looking at some of the pages in http they seem to redirect to https using a 302 redirect. I would use a 301 redirect instead - 302 means "this move is temporary" so Google treats it as less strong. Essentially it doesn't give a clear enough signal to Google that the new pages are replacing the old ones, Google seems to have some http pages still indexed but I don't know whether it's just not picked up all the redirects. Once you have all the 301s sorted, I would submit a http sitemap to Google to prompt it to crawl the old http pages and realise they are redirected.

Chantelle, when did the drop happen? And how defined was it? It sounds like, from your initial message, you went from a lot of traffic to not much at all in the space of days but was it more gradual than that? Was it before the https migration/ When did the agency make the changes?

The rogue canonicals effectdigital have found sound like they might be confusing things, that could line up with the coverage issues you saw Chantelle.

effectdigital - those links you found. Does it look like they have been around and like that for a while? Or were they recently added?