My site is not being regularly crawled?
-
My site used to be crawled regularly, but not anymore. My pages aren't showing up in the index months after they've been up. I've added them to the sitemap and everything. I now have to submit them through webmaster tools to get them to index. And then they don't really rank?
Before you go spouting off the standard SEO resolutions...
- Yes, I checked for crawl errors on Google Webmaster and no, there aren't any issues
- No, the pages are not noindex. These pages are index,follow
- No, the pages are not canonical
- No, the robots.txt does not block any of these pages
- No, there is nothing funky going on in my .htaccess. The pages load fine
- No, I don't have any URL parameters set
What else would be interfereing?
Here is one of the URLs that wasn't crawled for over a month: http://www.howlatthemoon.com/locations/location-st-louis
-
You forgot one...what is the URL of your site

-
oh sorry! Here is one of the pages that wasn't crawled since it was posted in early November.
-
That page is indexed. Can you give me an example of a page that hasn't been indexed yet? Your onsite SEO seems fine except you should get rid of your meta keywords tag and your Meta Description tag is a bit long.
-
Google may have reduced their crawl budget for your site if they found enough pages to be low quality, thin, duplicate, etc... Here are some examples that you can probably apply a noindex tag to in order to reduce crawl budget waste on them.
Google these:
site:howlatthemoon.com/ inurl:tag
1,360 indexed pages that don't need to be in the SERPssite:howlatthemoon.com/ inurl:tag page
Even if you want the tag pages indexed, you don't need their paginated pages indexed toosite:howlatthemoon.com/ inurl:page inurl:category
A few dozen category pagination pages in the SERPs, many more on the site that have been booted from the index
Example:
http://www.howlatthemoon.com/dueling_piano_bar/category/nightlife/page/2/I would install the Yoast Wordpress SEO plugin, which should fix these in the following ways:
Use rel canonical tags
Use rel Next/Prev tagsNoindex tag pages
etc...