The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Technical SEO Issues
    4. My site is not being regularly crawled?

    My site is not being regularly crawled?

    Technical SEO Issues
    5 3 138
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • howlusa
      howlusa last edited by

      My site used to be crawled regularly, but not anymore. My pages aren't showing up in the index months after they've been up. I've added them to the sitemap and everything. I now have to submit them through webmaster tools to get them to index. And then they don't really rank?

      Before you go spouting off the standard SEO resolutions...

      • Yes, I checked for crawl errors on Google Webmaster and no, there aren't any issues
      • No, the pages are not noindex. These pages are index,follow
      • No, the pages are not canonical
      • No, the robots.txt does not block any of these pages
      • No, there is nothing funky going on in my .htaccess. The pages load fine
      • No, I don't have any URL parameters set

      What else would be interfereing?

      Here is one of the URLs that wasn't crawled for over a month: http://www.howlatthemoon.com/locations/location-st-louis

      1 Reply Last reply Reply Quote 0
      • Mike_Davis
        Mike_Davis last edited by

        You forgot one...what is the URL of your site 🙂

        howlusa 1 Reply Last reply Reply Quote 0
        • howlusa
          howlusa @Mike_Davis last edited by

          oh sorry! Here is one of the pages that wasn't crawled since it was posted in early November.

          http://www.howlatthemoon.com/locations/location-st-louis

          Mike_Davis 1 Reply Last reply Reply Quote 0
          • Mike_Davis
            Mike_Davis @howlusa last edited by

            That page is indexed.  Can you give me an example of a page that hasn't been indexed yet?  Your onsite SEO seems fine except you should get rid of your meta keywords tag and your Meta Description tag is a bit long.

            1 Reply Last reply Reply Quote 0
            • Everett
              Everett last edited by

              Google may have reduced their crawl budget for your site if they found enough pages to be low quality, thin, duplicate, etc... Here are some examples that you can probably apply a noindex tag to in order to reduce crawl budget waste on them.

              Google these:

              site:howlatthemoon.com/ inurl:tag
              1,360 indexed pages that don't need to be in the SERPs

              site:howlatthemoon.com/ inurl:tag page
              Even if you want the tag pages indexed, you don't need their paginated pages indexed too

              site:howlatthemoon.com/ inurl:page inurl:category
              A few dozen category pagination pages in the SERPs, many more on the site that have been booted from the index
              Example:
              http://www.howlatthemoon.com/dueling_piano_bar/category/nightlife/page/2/

              I would install the Yoast Wordpress SEO plugin, which should fix these in the following ways:
              Use rel canonical tags
              Use rel Next/Prev tagsNoindex tag pages
              etc...

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Should you use robots.txt for pages within your site which do not have high quality content or are not contributing a great deal so when Google crawls your site the best performing content has a higher chance of being indexed?
                Jacksons_Fencing
                Jacksons_Fencing
                0
                5
                44

              • Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?
                libero_net
                libero_net
                1
                8
                142

              • Why is my site crawled so much more in Bing then in Google?
                EcommerceSite
                EcommerceSite
                0
                5
                204

              • Help Crawl friendliness for large site
                TakeshiYoung
                TakeshiYoung
                0
                4
                144

              • How Often is Site Crawled
                Matt-Williamson
                Matt-Williamson
                0
                5
                1.8k

              • Have a client that migrated their site; went live with noindex/nofollow and for last two SEOMoz crawls only getting one page crawled. In contrast, G.A. is crawling all pages. Just wait?
                Nobody1560986989723
                Nobody1560986989723
                0
                5
                422

              • Site maintenance and crawling
                Daylan
                Daylan
                1
                3
                451

              • Recently revamped site structure - now not even ranking for brand name, but lots of content - what happened? (Yup, the site has been crawled a few times since) Any ideas? Did I make a classic mistake? Any advise appreciated :)
                AlanBleiweiss
                AlanBleiweiss
                0
                9
                844

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy