URL Parameters

[bbop33]

Hi Moz Community,

I'm working on a website that has URL parameters. After crawling the site, I've implemented canonical tags to all these URLs to prevent them from getting indexed by Google. However, today I've found out that Google has indexed plenty of URL parameters..

1-Some of these URLs has canonical tags yet they are still indexed and live.

2- Some can't be discovered through site crawling and they are result in 5xx server error.

Is there anything else that I can do (other than adding canonical tags) + how can I discover URL parameters indexed but not visible through site crawling?

Thanks in advance!

[GastonRiera]

Hi there,

Its important to note that canonicals are a signal. Google can obey them if its algorithm considers that those pages are actually canonicals between each other. 
In my experience, this does not happen immediately, it usually takes Google some time to figure out if the canonicalization is correct. Keep in mind that pages being canonicalized HAVE TO be nearly identical and refer to the same topic.
And on the indexation part, pages can be indexed and be shown only when you search for that specific URL or using any advanced search parameter (such as site:).
More information about canonicals
- Consolidate duplicate URLs - Google Search support

• Canonicals tags - Moz resources Center

Regarding the second issue, if you refer to "site crawling" as what you do with an external tool, such as Screaming Frog or Moz, you are getting 5xx errors because that tool is making to many requests, try lowering its crawl frequency. I know for a fact that Screaming Frog allows you to do that.
But, unfortunately, I don't know any other way of discovering URL parameters in bulk but using an external tool.

Hope it helps,
Best luck.

[Rajesh.Prajapati]

I think you need to show some examples.

[effectdigital]

Just so you know, if a URL results in a 5XX server error then it usually won't render your canonical tag to begin with! You might want to check your sitemap XML, to check that it's not 'undoing' your canonical tags by feeding these URLs to Google. Indexation tags must be perfectly aligned with your sitemap XML, or you are sending Google mixed messages (e.g: a URL is in sitemap XML so Google should index it, but when it is crawled it contains a canonical tag citing itself as non-canonical, which is the opposite signal)

Everything which Gaston said is right on the money

[bbop33]

Hi Gaston,

Thanks very much for your time. The canonicals have implemented around a month ago and the pages are almost identical. I discovered all URL parameters without performing an advanced search.

Also, I come across the 5xx errors when I clicked indexed URL parameters on Google SERP and I cannot discover them when I crawl the site with Screaming Frog.

I'd appreciate if you have any other suggestions based on your experience!

Many thanks

[bbop33]

Hi Rajesh,

Thank you for your response. I cannot share the website due to client's confidentiality but basically when I search to find a stockist {brand name}, Google lists similar URLs below on the first page. The pages are showing a list of stockists depending on the product availability:

1-website.com/find-stockist?model=10 (5xx status code)
2-website.com/find-stockist?model=11 (200 status code)
3-website.com/find-stockist?model=10 (5xx status code)
4-website.com/find-stockist?model=11 (200 status code)

Thank you!

[bbop33]

Hi there,

Thanks very much for your response. I checked the sitemap and there are no URL parameters listed - only the canonical URL listed on the sitemap.

If you have any other suggestions it'll be much appreciated.

Thank you!

[hamzauuuu]

I'm also facing the same problem with my website pages. My Blackpods pro website pages don't show the exact permalink urls.