My Website is not being crawled by Google

[Ganesh34]

This post is deleted!

[MoosaHemani]

Ops but i see this website (http://www.websitedevelopersuk.com) in Google search results... try this URL and you will see the website in the search results http://goo.gl/h8irr

[SanketPatel]

Hi,

I think you have seen your cache date it is 23 aug, it is normal don't worry about it. My site also last cache on 24th aug. You can see your site in SERP it is indexed. If you have done any changes in URLs then you can resubmit your sitemap.

[Ganesh34]

This post is deleted!

[Ganesh34]

This post is deleted!

[MoosaHemani]

Ops my bad :s sorry! Well honestly when ever this happens with us what we usually do is to share the page on social media (Google+ and twitter preferred) and also link to external websites if possible...

Usually with social sharing and links give Google a hint to crawl the latest version of the page... currently the page’s social value is zero (the age that you mentioned in the 2^nd comment)

I would advise you to share the website socially (why not add share button on pages) and do some quick basic link building and you will see Google will crawl the new pages of the website within few days of time...

[blueSummit]

Try to find other quality sites with the same topical relevance and post backlinks to your clients start- and major category pages: strong link signals often make Google "revaluate" the page and the first step is to re-crawl the linked site.

[ThompsonPaul]

Ganesh, have you tried submitting some of the specific pages directly to Google for indexing?

To help speed up the process of getting re-indexed, use the "Fetch as Googlebot" and "Fetch as Bingbot" tools in Webmaster Tools for some of the pages that aren't getting re-indexed. Once you see a successful Fetch status, click Submit to Index, and then specify  to submit URL and all linked pages. This significantly helps jumpstart indexing of pages.

You can only do this with 10 pages per month, so choose wisely.

Paul

P.S. Longer term, you're still going to need to figure out if any changes you've made have interfered with Google's ability to crawl and index the pages.

[Ganesh34]

Thanks Moosa, for this Great Idea...I will specifically work on this & Let see what happen next

[MarieHaynes]

Had you gone a while without any on page changes?  Sometimes if Google sees that a page has remained the same for a while they'll decrease the crawl frequency.

Have you made any changes to your robots.txt?  You've got a lot of things disallowed...this could certainly inhibit a crawl.

Have your rankings changed?

My guess is that this is just normal and nothing to worry about.

[Ganesh34]

Hi Marie, I had gone through on page changes and also made changes to robots.txt file and Yes, my ranking fall down...that's why I am worried about. What can be the reason Google is not crawling the web page

Thanks

[MarieHaynes]

I'm no expert in robots.txt but you've disallowed Google from crawling the following directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /website-designers-uk.html
Disallow: /sem-smo.html
Disallow: /bespoke-web-applications.html
Disallow: /web-application-framework.html
Disallow: /Articles/seo-design-development.html
Disallow: /alquilandote
Disallow: /amptpp
Disallow: /bait
Disallow: /bin
Disallow: /brudedixox
Disallow: /bt
Disallow: /bug
Disallow: /cgi-bin
Disallow: /debamet
Disallow: /divinsms
Disallow: /fb 	
Disallow: /ft
Disallow: /ft-music
Disallow: /micasalla
Disallow: /new
Disallow: /orders
Disallow: /projects
Disallow: /Scripts
Disallow: /site
Disallow: /Templates
Disallow: /test
Disallow: /tiwido
Disallow: /alquilandote.websitedevelopersuk.com 
Disallow: /bait.websitedevelopersuk.com 
Disallow: /bt.websitedevelopersuk.com 
Disallow: /debamet.websitedevelopersuk.com 
Disallow: /ft-music.websitedevelopersuk.com 
Disallow: /micasalla.websitedevelopersuk.com 
Disallow: /orders.websitedevelopersuk.com 
Disallow: /tiwido.websitedevelopersuk.com

If any of those contain content, then Google won't be able to crawl it.

[Ganesh34]

Thanks Marie, Now I could understand why my site is not crawling

[MarieHaynes]

Is that what the issue was?  That you had blocked an important directory from being crawled?  If so, I'd love to write this up in my blog.

[Ganesh34]

Yes, that was the issue I had blocked some important directory which contain content. Go ahead, you can write about it, this would be great topic

Thanks

[MarieHaynes]

Great!  Would you prefer that I didn't mention your actual website name?  If I can mention it, it would be in the form of a link which would be good...but I can understand if you'd like to remain private.

Also, did you notice any drop in traffic yet associated with this issue?