Welcome to the Q&A Forum

We're all students of the game. I hope I never stop learning about marketing. I had read about the sitemap implementation but never tried it. Then I ran into an enterprise client who has over 40,000 indexed pages & 20 separate languages. One of the languages didn't have a subfolder but all the rest did. Instead of coding it in, I figured my way through sitemaps & thought it was SO much easier in the end. I've used it a few times since and it's great.

Hi Mike, I did see that there was an update last week, but as far as I can see we weren't impacted until Sunday/yesterday. In terms of pages indexed there has been a slight drop, but only 700 out of 19,900. So in line with the variance we get on a week to week basis. The drop in ranking seems to have only effected non branded keywords. Would a penalty effect both?

Not really an expert in robots.txt - but you could always try to simulate Rogerbot using a tool like Screaming Frog (the tool is free if you crawl less than 500 pages). You can enter a custom user agent. Rogerbot seems to be rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) and try crawling the pages on your site. In default mode, the tool will only crawl pages that are allowed by the robots.txt. If the tool crawls your pages, then something is wrong with the robots.txt - if not, than you should contact Moz staff to check with them.

Looks like Page Rank for all our key product listing pages has taken a hammering. They're all getting a PR of 1, even though our domain authority remains ok (it has dropped too, but not as dramatic).

Thanks.. it took me a while but I did realise that one eventually.

Do you have implemented the hreflang mark-up? Remember, Googlebot comes from an US IP, hence it will probably will see always the .com general version of the site. If you implement the hreflang, you are telling it what URLs to show depending on the country the visitors performs a search from.

Which leads me stumped as to whats stopping these two subfolders being ranked.. we're no.1 for branded keywords in the US for some pages on example.com/us and not even ranked in the UK for example.com/uk for the same keywords.

I actually had a look at the robots.txt last night to block the effected areas and discovered the dev's who built the site hadn't set it up. We've added a default one which we're going to test over the next week to see how it fares. If that doesn't work then we'll start looking at the parameters. Thanks.

Sorry about that, I'm sure we can get this figured out. Send the web address you are trying to use and paste the above response in an email to help@Moz.com. This way we can take a deeper look at your account and figure out why this is taking place. Hope to hear fro,m you soon!