My website pages are not crawled, what to do?

[lucidsoftech]

Hi all.

I have made some changes on the website so i like to crawled them by the search engines Google especially.

I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites.

Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml.

Still my pages are not crawled your opinion please.

Thanks

[SeoMartin1]

make sure you have a robots.txt on public folder. You must allow bots and disallow some folders.

Example of robots.txt of wordpress site

---

User-agent: *
Allow: /

Disallow: /*?
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /tmp

Allow: /wp-content/uploads/

---

[ShaMenz]

Hi lucidsoftech,

The site is definitely indexed, but without knowing the specific URLs you are waiting on, it is a little difficult.

I ran a quick crawl with Screaming Frog and there are actually quite a few pages on the site which have the noindex meta tag in place - have you checked that the pages you are wanting indexed are not excluded with a tag?

Just a thought,

Sha

[lucidsoftech]

HI Martin,

Before making these changes they crawled regular basis and I just changed a meta title keywords and description  but don't know the reason why after making these changes on few pages my web pages are not crawled in past 2 weeks and and still they are not crawled.

Martin i have my robots.txt in public.html and i allow the page that should be crawled by the bots.
waiting fro the page crawling.

Thanks

[lucidsoftech]

HI Sha,

yes site is indexed and crawled on regular basis before making these changes. don't know why not this time and not crawled around 2 weeks. yes i checked for thart all have including tag.
Normally when u submitted your pages on few good social bookmarking, workout in  webmasters, after generate few good back links all these thing are very helpful for page crawling.
Still waiting for the page crawling.

Thanks Sha

[SeoMartin1]

Lucid,

I´ve just made a search on google and I found 93 results for your site,   http://www.lucidsoftech.com/ , is that url right?

How many results do you actually own?

[lucidsoftech]

HI Martin,

Exactly, all url are indexed in search engines but they are not crawled after making a changes that is the main concern for me now.

For example:

Take a look on Magento Development page that showing last crawled 17 Mar 2013. I have made some changes and that should be crawled on a regular basis same as previous crawling.

Url :http://www.lucidsoftech.com/magento-development.html

your thoughts?

Thanks

[lucidsoftech]

Hi Sha,

I just add this meta tag (as below) to help to crawl the page but not sure how it is.

I am just worried that if we make changes in our website pages and submit the site map in webmaster (that's mean that we are put request to search engine crawler to crawl the page ) but why they do not.

what is the actual reason behind this.

also can you please explain me that what the other methods to force the search engine crawler to crawl our page.

Regards

Lucid

[lucidsoftech]

SEO Martin,

all these 93 results are index successfully also shows in our Google webmaster account.

But i am worried that why crawler is not visit our website pages on regular basis as we submit sitemap.xml in webmaster account  after the changes have made.

what is the reason behind this.

also can you please explain me that what the other methods to force the search engine crawler to crawl our page.

Regards

Lucid

[SeoMartin1]

this is somenting I would love to know more!-)

I think news and updates, maybe a blog in your site can make google "feel" it has to come back more times to refresh your site again ( "hum dude, they are doing something there)...  Links to your site and to new articles in social media could help a lot.

If you find anything let us know!-)