Welcome to the Q&A Forum

I second Siteliner, it's a good tool to help reduce duplicate and common content

Hi there! Ok, it is difficult to know all the ins and outs without looking at the site, but the immediate issue is that your robots.txt setup is incorrect. robots.txt files should be one per subdomain, and cannot exist inside sub-folders: A **robots.txt **file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers From Google's page here: https://support.google.com/webmasters/answer/6062608?hl=en You shouldn't be blocking Google from either site, and attempting to do so may be the problem with why your hreflang directives are not being detected. You should move to having a single robots.txt file located at https://www.clientname.com/robots.txt, with a link to a single sitemap index file. That sitemap index file should then link to each of your two UK & US sitemap files. You should ensure you have hreflang directives for every page. Hopefully after these changes you will see things start to get better. Good luck!

Τhere is a way to understand if it is ok and in Greek language? On this link -> Can my URLs use non-English words? - Google Webmaster Youtube i think that refers is general. How can i cross that its ok and with Greek language? I'm more close to put url names on Greek-lish (= English characters) with Greek meaning, and not in my language. Thank you very much

Hi James, I always prefer shorter over longer URLs. They look a lot cleaner which I believe Google likes. However saying that, I don't think there is a preference over one or the other as long as your website structure is SEO friendly. What are the pros to the longer URL for you? Cheers, Casey

Hi Kibin Firstly it's unlikely that their scraped content will affect your rankings - Google generally knows who originated it. However: Do you have the hreflang tag on your website? specifying your language and location? If theirs has this as well then technically you are targetting a different country, so there should be no duplicate content if you added it. https://support.google.com/webmasters/answer/189077?hl=en I would tell Google about the URL and add a sample 10 URLs first: https://www.google.com/webmasters/tools/dmca-dashboard. Telling them is an absolute must even if it's only a few URLs. Also email the hosting company informing them that they are hosting copied content and that the penalties are severe. Finally, write to the company themselves and tell them/warn them that you are going legal and send them a cease and desist legal letter. I am sure you can knock one up for a few dollars from a friendly solicitor. Watch this: https://www.youtube.com/watch?v=gGc_jc3Oznk It's a bit long but worth it. Do all of these things. Regards Nigel

What I can count and have contrasted in my web, is that the Javascript in Google is processed differently. Google crawls javascript and also the url inside, while Bing does not. I think I'm a bit short to answer your question, but that's what I've checked.

Hi Barret. I think you can try to change the video thumbnail on YouTube. It would be necessary to access as administrator the account where the video is. Then you would have to do the following steps: Sign in to YouTube. Go to Creator Studio > Video Manager. Next to the video, click Edit. On the left of the preview screen, choose a thumbnail. Click Save changes. You can try doing another search and see if that forces Google to use the thumbnail.

Hi Daniel, That does seem very odd! There can be various different things at play here in my experience: Publishing and un-publishing in the CMS - Sometimes clients and CMS users will switch the page publish status around which can make it difficult to keep track of whether the page exists or not Removal of redirects - Much the same as above, there's the possibility that someone is adding/removing redirects. This could also be something to do with the removal of the server level redirects file, be that htaccess or IIS. Crawling tool issue - If you're using a crawling tool (especially if doing so with JS rendering active), sometimes this can put a lot of pressure on the server and it can spit out the occasional error where there actually isn't one It could also be a problem with how you're handing hard 404 errors vs soft 404s - i.e. actual not founds vs pages that don't function but the server is under the impression that they're fine. Best of luck! Sean

I recently answered a similar question here.... https://moz.com/community/q/how-to-handle-annual-content-2018-2019

Hi Zack, I think your concern here is valid (your render with Screaming Frog or any other client is unlikely to be precisely representative of what Googlebot will see/index). That said, I'm not sure there's much you can do to eliminate this knowledge gap for your QA process. For instance, while we have seen Googlebot timing out JS rendering around the ~5s mark using the "Fetch & Render as Googlebot" functionality in Search Console (see slide 25 of Max Prin's slide deck here), there's no confirmation this time limit represents Googlebot's behavior in the wild. Additionally, we know that Googlebot crawls with limited JS support - for instance, when a script uses JS to generate a random number, my colleague Tom Anthony found that Googlebot's random() JS function is deterministic (returns a predictable set) - so it's clear they have modified the headless version of Chrome they use to conserve computational expenses in this way. We can only assume they've taken other steps to save computing costs. This isn't baked-into Screaming Frog or any other crawling tool. We have seen that with a 5s timeout set in Screaming Frog, the rendered result is pretty close to what "Fetch & Render as Googlebot" functionality demonstrates. And with the ubiquity of JS-driven content on the web today, provided links and content are rendered into the DOM fairly quickly (well ahead of that 5s mark), we've seen Google rendering and indexing JS content fairly reliable. The ideal would be for your dev team to code these pages to degrade gracefully - so that even with JS support totally disabled, navigation and content elements are still rendered (they should be delivered in the page source, then enhanced with JS, if possible). Failing that, the best you're likely to achieve here is reasonable confident that Googlebot can crawl, render and index these pages - there'll be some risk when you publish them to production. Hope this helps somewhat - best of luck! Thanks, Mike

If the Disallowed URLs are linked to externally, or internally from a page that isn't Disallowed, they'll appear in the index with the snippet text you've quoted. If you want to ... Typically, that page will already be populated with parameters the crawler has discovered, though you can specify them manually too.

Yes, finally I will add more content to the pages. Thanks.

Thanks for the response. Looks like we have 55 in search console that I am working on fixing now (mostly things I drafted in CMS) Would I need to add anything I don't want indexed anymore to robots.txt? I assume that some of the incomplete 301's might stem from this?

Have a page for the pillar topic and one for the sub topic if you have the word count to justify it. Otherwise sub topics can be H2s if you have two separate pages I would avoid keyword cannibalization were possible. Keep your keywords on the page that target them. However it may be necessary to talk in the context of pillar content. If you do mention the Pillar keyword, make it a text link to the pillar page. That way google will attribute the use of that word to the pillar page when ranking - and it will in fact strengthen the pillar page. Does that make sense?

Thats great thanks for the help

First of all - don't stress out! You won't be penalized for broken links. Google penalizes for shady behavior, not honest mistakes. Like Andrew said, make sure that you aren't linking to any of the pages you removed from your site, and that you've removed them from your XML sitemap. Then you should be good! Good luck, Kristina

Yeap, URLs that are now 301 redirected will eventually be removed from google and stop being showed. Keep in mind that what Moz shows in their reports does not exactly reflects what is in Google. Hope it helps. Best Luck. GR.

Yes, and make sure to do a self-referencing canonical to the generic page as well.