Not sure how we're blocking homepage in robots.txt; meta description not shown

[SearchDeploy]

Hi folks!

We had a question come in from a client who needs assistance with their robots.txt file.

Metadata for their homepage and select other pages isn't appearing in SERPs. Instead they get the usual message "A description for this result is not available because of this site's robots.txt – learn more".

At first glance, we're not seeing the homepage or these other pages as being blocked by their robots.txt file: http://www.t2tea.com/robots.txt.

Does anyone see what we can't? Any thoughts are massively appreciated!

P.S. They used wildcards to ensure the rules were applied for all locale subdirectories, e.g. /en/au/, /en/us/, etc.

[Andy.Drinkwater]

The best way to test this is to head into Search Console and use the Robots.txt tester. If a URL is being blocked, or suspect it is, just add that URL to be tested and it will show you.

https://support.google.com/webmasters/answer/6062598?hl=en

-Andy

[LynnPatchett]

Hi,

It looks like they are 302 redirecting the homepage to internal language/region specific storefronts but are doing that based on an internal url structure that includes /on/demandware.store/ which is indeed being blocked in the robots.txt. It looks like those urls are then being 301 redirected to the user friendly url you see in the browser so there is a potentially odd redirect chain going on there. The original blocked urls are probably the immediate issue (although the 302 redirects and region/language redirect logic might be putting more complication on top of that).

[ThomasHarvey]

According to screamingfrog the current line:

Line:40 http://www.t2tea.com/on/demandware.store/

Is the line on robots.txt is causing you an issue.

[Saijo.George]

I can see the meta descriptions in SERPs. do you have any sample pages where it does not show up?