Welcome to the Q&A Forum

Very hard to prove these things before they're done - good luck with getting buy-in for what you need to do and in undoing the worst of the damage.

I also want to know how to increase the Authority of my website. I have taken the Moz course to learn the SEO to give their services. Can you guide more?

Hi effectdigital Yep I won't be doing that! I don't think I'll be using the robots.txt file at all for this. Thanks Frankie

It could happen, but it's unlikely to happen. If Google hasn't discovered that URL at all and then you make it easier for Google to discover, they may then apply internal authority to the page through your internal link structure. That could cause the page to rank higher. But the SEO authority wouldn't come 'from' the XML sitemap, only the 'discovery' of the URL would be impacted by your XML feed. If Google has already known of the page for a while (as is probably the case), then XML tweaks likely won't make any difference

If you have a site with, at least 30k URLs, looking at only 300 keywords won't reflect the general status of the whole site. If you are looking for a 10% loss in traffic, I'd start by chasing the pages that lost more traffic, then analyzing whether they lost rankings or if there are some other issues. Another way to find where there is traffic loss is in search Console, looking at keywords that aren't in the top300. There might be a lot to analyze. It's not a big deal having a lot of pages blocked in robots.txt when what's blocked is correctly blocked. Keep in mind that GSC will flag those pages with warnings as they were previously indexed and now are blocked. That's just how they've set up flags. Hope it helps. Best luck. Gaston

There are no cons (that I can see) to losing the "/shop-by-department/" folder, the new URL structure you illustrated is better as it offers an opportunity to add pertinent keywords. Just be sure you have your 301 redirects set up correctly.

Thanks Rajesh! Very clear, will pass that on to developers!

Thanks again AL123al! I would be concerned about my internal linking because of this problem. I've always wanted to keep important pages within 3 clicks of the Homepage. My worry here is that while these products can get clicked by a user within 3 clicks of the Homepage, they're blocked to Googlebot. So the product URLS are only getting crawled in the sitemap, which would be hugely ineffcient? So I think I have to decide whether opening up these pages will improve my linking structure for Google to crawl the product pages, but is that important than increasing the amount of pages it's able to crawl and wasting crawl budget?