The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robot.txt error

    Robot.txt error

    Intermediate & Advanced SEO
    13 3 203
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Rubix
      Rubix last edited by

      Thanks Irving,

      www.positivecollections.co.uk is the url

      I've tried to remove everything from the robot file and check again on webmaster, same thing happened  It's just blocking the main link

      1 Reply Last reply Reply Quote 0
      • Rubix
        Rubix last edited by

        When I click on Fetch As Google, i get 'Denied by robots.txt'' error.

        @Jesse: That's the main website, we've got other URLs.Error appears on positivecollections.co.uk

        jesse-landry 1 Reply Last reply Reply Quote 0
        • jesse-landry
          jesse-landry @Rubix last edited by

          That site is also being indexed. Again I ask, what makes you think it is not being indexed? (cause it is)

          1 Reply Last reply Reply Quote 0
          • Rubix
            Rubix last edited by

            When I click on Fetch As Google, i get 'Denied by robots.txt'' error.

            jesse-landry 1 Reply Last reply Reply Quote 0
            • jesse-landry
              jesse-landry @Rubix last edited by

              Okay. But that doesn't mean it isn't being indexed. Here's a fun test: Go to any page on your website and select a string of two or three sentences. Google it. Does the page come up in the SERPs?

              (I did this to 3 pages on your site and it worked for all of them. Therefore, your site is being indexed.) Why do you need to Fetch?

              Rubix 1 Reply Last reply Reply Quote 0
              • irvingw
                irvingw last edited by

                TECHNICAL ISSUE

                It's your cookie policy blocking bots from spidering. Need to fix that at the server level. Easy fix! 🙂

                http://www.positivecollections.co.uk/cookies-policy.aspx

                Your robots.txt is fine.

                Rubix 1 Reply Last reply Reply Quote 1
                • Rubix
                  Rubix @jesse-landry last edited by

                  I couldn't relay on SERPS as the website is old, it's been indexed for quite so i didn't think that SERP results would change that quick. I've been receiving the error since yesterday.

                  It's on SERPS today but would it be there tomorrow? The reason I am saying that is because when i change the Page Title, it doesnt get changed on SERPS instantly, it takes a day or so before i see the changes on SERPS.

                  1 Reply Last reply Reply Quote 0
                  • Rubix
                    Rubix @irvingw last edited by

                    Thanks Irving, I need a little more help, I am not quite sure if I understand it. What is it that needs to be fixed here?

                    irvingw 1 Reply Last reply Reply Quote 0
                    • irvingw
                      irvingw @Rubix last edited by

                      Try to spider your site with this link checker tool

                      http://brokenlinkcheck.com/

                      bots cannot accept cookies and your site is requiring cookies to be enabled in order to be visited so Google cannot access the site because you are not allowing the visit without the cookie being dropped is most likely the issue.

                      Disable cookies on your browser and clear your cache and see what happens when you try to visit your site, are you blocked?

                      These discussions may possibly help

                      http://www.highrankings.com/forum/index.php/topic/3062-cookie-and-javascript/

                      http://stackoverflow.com/questions/5668681/seo-question-google-not-getting-past-cookies

                      Rubix 1 Reply Last reply Reply Quote 1
                      • Rubix
                        Rubix @irvingw last edited by

                        Thanks Irving, it worked 🙂

                        1 Reply Last reply Reply Quote 0
                        • 1 / 1
                        • First post
                          Last post
                        • Robots.txt Allowed
                          GlobeRunner
                          GlobeRunner
                          0
                          4
                          118

                        • Robots.txt question
                          Mobilio
                          Mobilio
                          0
                          2
                          107

                        • Robots.txt
                          Travis_Bailey
                          Travis_Bailey
                          0
                          4
                          107

                        • Robots.txt assistance
                          theLotter
                          theLotter
                          0
                          9
                          280

                        • Robots.txt Syntax
                          MichaelC-15022
                          MichaelC-15022
                          0
                          2
                          118

                        • Robot.txt help
                          evolvingSEO
                          evolvingSEO
                          0
                          23
                          203

                        • Robots.txt error message in Google Webmaster from a later date than the page was cached, how is that?
                          eTundra
                          eTundra
                          0
                          3
                          235

                        • 202 error page set in robots.txt versus using crawl-able 404 error
                          EricaMcGillivray
                          EricaMcGillivray
                          0
                          3
                          870

                        Get started with Moz Pro!

                        Unlock the power of advanced SEO tools and data-driven insights.

                        Start my free trial
                        Products
                        • Moz Pro
                        • Moz Local
                        • Moz API
                        • Moz Data
                        • STAT
                        • Product Updates
                        Moz Solutions
                        • SMB Solutions
                        • Agency Solutions
                        • Enterprise Solutions
                        • Digital Marketers
                        Free SEO Tools
                        • Domain Authority Checker
                        • Link Explorer
                        • Keyword Explorer
                        • Competitive Research
                        • Brand Authority Checker
                        • Local Citation Checker
                        • MozBar Extension
                        • MozCast
                        Resources
                        • Blog
                        • SEO Learning Center
                        • Help Hub
                        • Beginner's Guide to SEO
                        • How-to Guides
                        • Moz Academy
                        • API Docs
                        About Moz
                        • About
                        • Team
                        • Careers
                        • Contact
                        Why Moz
                        • Case Studies
                        • Testimonials
                        Get Involved
                        • Become an Affiliate
                        • MozCon
                        • Webinars
                        • Practical Marketer Series
                        • MozPod
                        Connect with us

                        Contact the Help team

                        Moz logo
                        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                        • Accessibility
                        • Terms of Use
                        • Privacy