Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
What keyword are you searching to try and find your site? I am assuming your site is this one. If we can recreate the search and see the issue, we might be able to scan for stuff that the competitors are doing which you aren't doing (e.g: more local links / citations, schema, GMB usage)
In this case you would do well to read this documentation:
https://moz.com/help/moz-procedures/crawlers/rogerbot
This forum sees a lot of general SEO queries and is a bit of a broader community, not just for Moz products (though we do see lots of Moz product questions as well!)
Can you just deploy Meta no-index through the HTTP header instead of through the HTML?
The advice is no longer current. If you want to see what Google used to say about rel=next/prev, you can read that on this archived URL: https://web.archive.org/web/20190217083902/https://support.google.com/webmasters/answer/1663744?hl=en
As you say Google are no longer using rel=prev/next as an indexation signal. Don't take that to mean that, Google are now suddenly blind to paginated content. It probably just means that their base-crawler is now advanced enough, not to require in-code prompting
I still don't think that de-indexing all your paginated content with canonical tags is a good idea. What if, for some reason, the paginated version of a parent URL is more useful to end-users? Should you disallow Google from ranking that content appropriately, by using canonical tags (remember: a page that uses a canonical tag cites itself as non-canonical, making it unlikely that it could be indexed)
Google may not find the parent URL as useful as the paginated variant which they might otherwise rank, so using canonical tags in this way could potentially reduce your number of rankings or ranking URLs. The effect is likely to be very slight, but personally I would not recommend de-indexation of paginated content via canonical tags (unless you are using some really weird architecture that you don't believe Google would recognise as pagination). The parameter based syntax of "?p=" or "&p=" is widely adopted, Google should be smart enough to think around this
If Search Console starts warning you of content duplication, maybe consider canonical deployment. Until such a time, it's not really worth it
And if people don't have Search Console access, they can always try queries like:
https://www.google.com/search?q=site%3Abbc.co.uk
... however, due to the difficulty in exporting all the listed URLs, and due to some anti-scraping measures by Google, this data tends to be inferior to that supplied by Search Console. Always use Google's official tools first
For SEO it's always better to link directly, rather than linking to a page which then redirects. Redirects only pass large amounts of SEO authority to 'similar' pages (there are checks and balances on this). You could experiment with something like this, but I doubt it will result in much uplift really
So in July (2019) John Mu from Google stated that URLs are generally ok up to 1,000 characters:
Google 'can' crawl and index URLs over 1,000 characters long (up to about 2k characters) but best practice seems to be up to 1,000 characters
Due to this, I personally don't agree with Moz's evaluation of, when a URL is getting too long. Your example URL, is nowhere close to 1,000 characters long. Where Moz and Google disagree I tend to side with Google info
That being said, your URL has redundant layers. Why even have "/category/" in the URL? Just go like this:
mysite.com/downloads/premium-downloads/sub-category/
People aren't so stupid that they need a fake URL layer called "/category/", to know that the following URL layer 'is' a category. IMO that's redundant architecture which is getting in your way for no reason
If you don't perform your redirects properly and you change the architecture of the site, you absolutely could see a negative impact on your rankings. Unless you're confident in terms of crawling your whole site, performing very granular redirects with high accuracy and missing nothing - I'd just leave it as it is
Ignore the user declared canonical thing, that's not an error and you are misreading it. The full line says this: "Google-selected canonical: Same as user-declared canonical". This means that Google is choosing the SAME canonical as the user is, and in this case the user is you (not your site's users. They mean the search console user). That mans that Google is agreeing with you
Also you are checking the wrong URL:
https://d.pr/i/341c5u.png (edit of your screenshot)
In the red box you can see that you are checking the URL (in Search Console) which does not end with "/". But in the green box, you can see that the canonical URL which you have set, does end with "/". So you are checking the non-canonical version of your page and hoping it is indexed
If that still doesn't change anything, there is no error, Google just doesn't like your page enough to rank it properly (sorry)
I'm not a massive display network person but AFAIK (as far as I know) Google's is one of the cheapest
That attachment shows that non HTTPS and non WWW URLs are being 301 redirected to the HTTPS-WWW version(s). That's what you want right? From your screenshot it seems like it is working how you want
Just so you know, when you put one architecture into Screaming Frog (e.g: you put in HTTP with no WWW), it doesn't limit the crawl to that specific architecture. If the crawler is redirected from non-WWW non HTTPS to HTTPS with WWW, then the crawler will carry on crawling THAT version of the site
If you wanted to crawl all of the old HTTP-non-WWW URLs, you would need to list all of them for SF in list mode and alter the crawlers settings to 'contain' it to just the list of URLs which you entered. I'm pretty sure then, you would see that most of the HTTP-non-WWW URLs are properly redirecting as they should be
As for the XML thing it's very common especially for people using Yoast. I think Yoast is really good by the way, but for some reason, on some hosting environments the XML sitemap starts blank-rendering. Most of the time hosting companies say they can't fix it and it's Yoast's fault but I don't really believe that. If a file (e.g: sitemap.xml) cannot be created, it's more likely they went in via FTP and changed some file read/write permissions and due to it being more locked down, the XML cannot be created anymore. If you were hacked by malware, they were likely over-zealous when locking your site back down and it's causing problems for your XML feed(s)
I can't seem to replicate your problem
https://d.pr/i/l9sPYn.png (SERPs screenshot)
For me, your site is loading ok in Google's SERPs and showing all the Meta data you have set
The URL which you have shared says that it is indexable with DeepCrawl's indexability checker. Can you give more details and screenshots, or let us know if things have changed?
Quite an old style of architecture, it's a shame it cannot be changed. Just so others understand a bit more, what you refer to as a 'suffix' is actually called parameters. In a URL, anything following "?" is parameters
If the language on the root is dynamic (it changes) then it's very difficult for you to hreflang to it effectively as it will conflict the the parameter URLs (which are language based) AND additionally, you won't know for certain what language to hreflang to. That also makes canonicals to the root quite tricky IMO
I think what you are already doing, is the best of a bad situation. At least the parameter-based URLs set a designated language which you can rely on to be the same
If you look at this official URL from Google: https://support.google.com/webmasters/answer/182192?hl=en
Scroll down until you find the heading "Using locale-specific URLs" and look at the table underneath of that heading
Parameter-based geo-targeting, is actually the only one of of multiple architectures, which Google put in red text and explicitly warn people away from. Since the site you are looking at has crossed that red line, you may need to manage expectations about results. If they're going to pick the worst possible format and stick with it, without asking you as a consultant what is best, they've kind of shot themselves in the foot there
P.S: Regarding 'actual' redirects, not canonicals. For sites that have proper sub-folder structure, usually you redirect users based on their location, but allow them to flag select to 'escape' the redirections (which can sometimes go wrong). You also usually exempt Google's user-agent ('googlebot') from regional redirects, as they can only crawl from one location at once and otherwise they think areas of your site keep going up and down due to all your redirects. But with your structure, I'm not sure I would even touch redirects. It's in enough of a state as it is without rolling those dice
These invasive URL hacks can be very tricky to get under control. Sometimes if you're lucky, it's just a matter of erasing the pages and stuff. In other circumstances scripts are injected which recreate the bad URLs (and links to them) when you take them down. You certainly want to be on the lookout for unknown scripts on your site, or links to external scripts which didn't exist before
It could happen, but it's unlikely to happen. If Google hasn't discovered that URL at all and then you make it easier for Google to discover, they may then apply internal authority to the page through your internal link structure. That could cause the page to rank higher. But the SEO authority wouldn't come 'from' the XML sitemap, only the 'discovery' of the URL would be impacted by your XML feed. If Google has already known of the page for a while (as is probably the case), then XML tweaks likely won't make any difference
Nope, Sitemap.XML is just for indexation. It doesn't ever boost any page's authority in any way
As far as most of us know in SEO, canonical tags don't consolidate SEO authority (like 301 redirects do). They merely alleviate duplicate content risk. Usually having a divided structure is the issue, not the usage of canonical tags (or not)
Personally I usually do this as well as robots.txt blocking them to save on crawl allowance, but you should no-index first as if Google is blocked from crawling (robots.txt) then how will they find the no-index tags? So it needs to be staggered
I find that the tag URLs result in quite messy SERPs so I prefer to de-index those and then really focus on adding value to 'actual' category URLs. Because categories have a defined structure they're better for SEO (IMO)
Categories are usually good for SEO if you tune and tweak them up (and if their architecture is linear) but tags are very messy
When you blend different architectures you WILL get contextual tag collisions (e.g: canonical tags don't work the same across both architectures, or Meta robots tags, or hreflangs). What you are suggesting is probably the best thing you can do, I stand behind WordPress quite firmly, even when it's used to augment another CMS (instead of replacing it). That being said, if you believe you are going to be able to predict all the issues which arise in advance - think again!
Do your best on that front (obviously) but you need to manage your client's expectations. There is going to be some fallout. There is going to be analysis needed. There is going to be dev work needed to create a seamless integration. It's good that you're being diligent and trying to predict everything, but with the complexities of the modern web (the same thing can be coded in thousands of different ways) you really need to accept there will be some problems that will need fixing. You'll need to hold some budget back for that stuff, or you'll be up a gum tree
Are you sure that this:
http://mechfieldfm.com.au/JCB-BACKHOE-IGNITION-SWITCH-WITH-2-KEYS-PART-NO-70180184-70145500-372796/
... is an example of a bogus or hacked page? Because when I check Google's cache for the URL:
https://d.pr/i/6sioLy.png (screenshot)
It looks like it was once a 'real' page. Google's cached version only looks so messed up as they failed to capture the CSS properly. It just seems weird that someone would hack your client's site and just leave info for construction-related products on your client's domain. Usually with hacks it's a phishing / political agenda or something like that (or some kind of scam)
Still, having said that; although your client is in construction they don't seem to list any products of any kind (your client appears to be service-based). You have to admit though, to an outsider, it looks more like 'dev-gone-wrong' than a hack. It's just really weird, at the very least
Can't you just write a redirect rule that 301s or 302s all of those bogus URLs somewhere else? Those URLs shouldn't be accessible anyway. Your client has a valid SSL certificate and the site loads just fine on HTTPS: https://mechfieldfm.com.au - so another question is, why was HTTP left exposed? The bogus URLs are on HTTP, maybe they found an entry-point because that architecture was left exposed instead of being properly redirected
Even Google mostly link to the site on HTTPS:
https://www.google.co.uk/search?q=site%3Amechfieldfm.com.au
https://d.pr/i/XfbNF7.png (screenshot)
Where they're not doing this, it's because they are linking to more examples of 'bogus' URLs. Personally I'd just slap no-index tags and status code 410 (gone) on all of those problematic URLs. If you're worried you can't use no-index as the HTML of those pages is gone, you can still fire Meta no-index directives through the HTTP header using X-Robots, so be sure to look that up
It will probably take a while (some weeks) for Google to digest all the disruption and fixes. In the future, ensure that disused architectures (like HTTP) are made physically inaccessible via redirects. Leaving old architectures active is a recipe for disaster. Once this is all cleaned up, be sure to properly implement HTTP -> HTTPS redirects
Here's something odd. On your client's homepage, the HTTP->HTTPS redirect is properly implemented. Somehow these 'hacked' URLs are evading that redirect. You need to at least lock that down or it could happen again
