Why isn't my site being crawled ? only crawled one page ..

[TIUConsulting]

I have added my website JTCValvesales.com at Moz dashboard . But in crawl diagnosis section their is no error found and only one page is indexed.. i have sent query to Moz support and generate a ticket but still i didnt get any response from them. Plz help

[MattAntonino]

You have a nothing in it disallow but I don't know if you either a) recently changed or b) are accidentally blocking with this code.

Why not instead of disallowing nothing just allow everything?

User-agent: rogerbot
Disallow:

becomes

User-agent: rogerbot
Allow: /

Try that + give it a bit of time. Let us know how you go.

[Kokolo]

It's the robots.txt file I believe.

I've spent a lot of time on this, and me myself I am newbie when It comes to coding.

I was forced to re-work this issue on 4/21 when Mobile Update came out.

Here is my current robots.txt file that provides me good crawl results:

**User-agent:  *

Allow: /

Disallow: /wp-admin/

Disallow: /wp-login/

Disallow: /?p=*

User-agent: Mediapartners-Google*

Allow: /

Disallow: /?p=***

I hope it helps you. And if someone believes I have some mistakes, please let me know as well,

[DavidLee]

Hi Ashish

Sorry you didn't receive my reply last week. Here was my response:

The crawlers are working as intended but it appears the issue is both sites have a re-direct setup our crawlers are not able to follow. JTV takes long time to load which will also prevent crawling.

http://tools.pingdom.com/fpt/#!/bYcyeD/JTCVALVESALES.com

The site is re-directing our crawler to the same URL so we are stuck in a loop

http://www.evergreenmidwest.com/?xid_12868=eb4522ee2cea11a1137f51dc830bb32b sends us back to http://www.evergreenmidwest.com/?xid_12868=eb4522ee2cea11a1137f51dc830bb32b

and https://www.jtcvalvesales.com/?xid_157b9=0b27dce510fce9a3e20a01802dcde99d back to https://www.jtcvalvesales.com/?xid_157b9=0b27dce510fce9a3e20a01802dcde99d

You will need to work with your web dev/hosting provider to investigate the re-directs for search crawlers.

Hope this helps!