Welcome to the Q&A Forum

Hi Christian,

The current robots.txt shouldn't be blocking those URLs.

Did you or someone else recently change the robots.txt file? If so, give Google a few days to re-crawl your site.

Also, can you check what happens when you do a fetch and render on one of the blocked posts in Search Console? Do you have issues there?

Cheers,

David

Hi ricoplaza,

There isn't going to be any negative impact but it's also not going to be much benefit to you (if any benefit at all).

Just because you resubmit your sitemap, doesn't mean Google is going to crawl the URLs in your sitemap instantly.

For your e-commerce site, if you are wanting to speed up the indexing of new products or product categories, you can use the 'Fetch as Google' tool in Search Console to request faster indexing.

Also, news sites don't need to resubmit sitemaps to have their content indexed faster. Sites that are in Google news will have a news XML sitemap that is basically a direct feed for Google to index their news content instantly - no need to resubmit sitemaps.

Unless you've made some major structural changes to your site, you don't really need to worry about resubmitting your sitemap.

Cheers,

David

Hi Becky,

Along the lines of what James said, it's because this is not the canonical URL for the product.

The URL is: http://www.key.co.uk/en/key/0-4-tonne-wll-6mm-single-leg-wire-rope-sling-580a078n

But your canonical and hreflang tags don't match this URL.

You will get errors for any URLs that don't have a self-referring hreflang tags, and any URLs that don't have reciprocal hreflang tags from the alternate versions specified.

I would work with your devs to only display hreflang tags on canonical URLs - not on canonicalized URLs.

Cheers,

David

Hi Masonman,

I generally "noindex" tags in Yoast and for really big sites I will also block them in robots.txt (after I know they have been removed from index) - but it depends how you are using them on your site.

Before you "noindex", check GA to make sure your tags pages aren't getting any organic traffic. If they aren't, I would go ahead and set them to "noindex".

If you want to then block them from being crawled, you must wait until the pages have been removed from index. Otherwise, they will remain indexed because the pages can't be crawled and the "noindex" tags won't be seen.

Tags only exist to help users find similar content. You could check GA pageviews for your tag pages to see if people are actually using them - if they aren't, you might decide to stop using them.

You have a few options here...I could give more specific advice on what I would do if I knew more about your site, but I hope this helps!

Cheers,

David

Hi tiagoarruda,

For a while now, it seems to be Google's strong preference to have the brand name (rather than keywords) at the front of homepage titles.

It is very common for Google to re-write your homepage titles if your brand name isn't at the front and I think it's unlikely that you'll be successful in having it changed.

Cheers,

David

Hi Eddy,

Edit: this was already answered before I could post my reply. But I've left the example.

The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>

This will fix your issue.

Cheers,

David

Hi jacleaves,

I'm interested to see how this works out for you!

Please post an update after you've finished your work

Cheers,

David

Hi Sonar,

Sorry to hear about your migration troubles!

I don't think you will find generic advice too helpful, but I'd be happy to take a quick look to see what's going on and possibly give you some specific recommendations.

Are you able to share the domains with me in this thread or through PM?

Cheers,

David

Hi Fubra,

You can disavow at a domain level, so no regex is required (and I don't think it will work).

Just add "domain:" before the domain, eg. domain:spammysite.com

Marie Haynes wrote a good guide to using the disavow tool here if you need any further information: https://moz.com/blog/guide-to-googles-disavow-tool

Cheers,

David