Very well established blog, new posts now being indexed very late

[rookie123]

I have an established blog.We update it on daily basis. In the past, when I would publish a new post, it would get indexed within a minute or so.

But since a month or so, its taking hours. Sometimes like 10-12 hours for new posts to get indexed. Only thing I have changed is robots.txt.

This is the current robots file.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-login.php
Disallow: /*wp-login.php*
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /author
Disallow: /category
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /login/
Disallow: /wget/
Disallow: /httpd/
Disallow: /*.php$
Disallow: /*?*
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

User-agent: TechnoratiBot/8.1
Disallow:

# ia_archiver
User-agent: ia_archiver
Disallow: /

# disable duggmirror
User-agent: duggmirror
Disallow: /

# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow: /wp-includes/
Allow: /*

# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

Sitemap: http://www.domainname.com/sitemap.xml.gz

Site has tons of backlinks. Just wondering if something is wrong with the robots file or if it could be something else.

[dignan99]

Could you possibly reverse the changes of the robots.txt to a previous "working" version where your site was getting indexed quicker?

[MBayes]

Do you use a word press platform? If so do you use a SEO plug in.  Different plug ins can effect the index time.

[MBayes]

Do you use a word press platform? If so do you use a SEO plug in.  Different plug ins can effect the index time.

[rookie123]

Yes, its a wordpress site and I always had the all in one SEO plugin enabled.

[WilliamBay]

Are you using Feedburner? Has the feed publishing service gotten out of sync? You can re-sync it under the Troubleshootize section.

[RyanKent]

The robots.txt file is designed to completely block content. Normally, if your robots.txt file was a factor then your content would not appear in SERPs at all.

It is possible for content to appear in SERPs even though it is blocked by robots.txt if it is linked from other sources. Since this is new content, it is less likely that is the case unless you are immediately sharing links and Google is seeing those links within the time frame you shared.

The first place I would look is your sitemap or whatever tool is used to inform Google that you have new content. When you publish a new blog article, your software should ping Google and inform them there is new content. That is where any investigation should begin. Next step is to check server logs to see how long it takes Google to respond to the alert. If it takes them 12 hours, then there is nothing further you can do about it.

I would be interested in a lot more detail. How many articles how you confirmed as being affected by this issue. Exactly how did you confirm the issue?

As a side note, your robots.txt file is bloated and doesn't adhere to any standards I have seen. How exactly was it created? Did someone go in and make manual modifications to the file?