Search engine blocked by robots-crawl error by moz & GWT

[TimHolmes]

Hello Falguni,

I believe the error is saying pretty much everything you need to know. Your Robots file or robots meta would appear to be blocking your site from being crawled.

Have you checked your robots.txt file in your root - or type in http://www.yourdomain.com/robots.txt

To ensure your site is being crawled and for robots to have complete access the following should be in place

**User-agent: ***
Disallow: To exclude all robots from the entire server

**User-agent: ***
Disallow: **/**If it is the a meta tag causing the issue you will require, or have it removed to default to the below.

As opposed to the following combinations which could result in some areas not being indexed, crawled etc

_Hope that helps

Tim_

[falguniinnovative]

Thanx,  Tim Holmes For Your Quick reply

But My robots.txt File is

User-agent: *
allow: /

Also in All pages i have Add Meta-tag

Then After Page is Not Getting Fetched or Crawl by GWT.

Thnx

[TimHolmes]

Have you followed the following and in Google Search Console tried testing your robots file.

test your robots file guide.

If you are allowing all, I would maybe suggest simply removing your robots.txt all together so it defaults to just crawling everything.

[falguniinnovative]

Yes, It is Also okay there see Attached screenshot

88872m

[TimHolmes]

Just to check it that the live one, or just a test in GSC. Can you send a link to your site maybe in a PM.

[falguniinnovative]

www.tajsigma.com This the Domain For Query robots 605- code

[TimHolmes]

Headers look fine and as you correctly said your robots and meta robots are also ok.

I have also noted that doing a site:www etc in google search is also returning pages for your site so again showing it is being crawled and indexed.

To all intents and purposes it looks ok to me. Someone else may be able to shed more light on the issue if they have experienced this error to this degree.

[MattRoney]

That's very strange. The robots.txt looks fine, but here's what I see when I search for your site on Google.

[falguniinnovative]

Yes , Exactly

I am also worried For that only, Can you please help to identify my site problem

Thnx

[RuthBurrReedy]

When was your last crawl date in Google Webmaster Tools/Search Console? It may be that your site was crawled with some kind of problem with the robots.txt and hasn't been re-crawled since.