The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Robot.txt error

    Robot.txt error

    Intermediate & Advanced SEO
    13 3 203
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jesse-landry
      jesse-landry last edited by

      Are you sure your site isn't being indexed?

      Cause I went to your profile and if http://www.mtasolicitors.com/ is your site, then it is definitely being indexed.. What makes you think it isn't?

      1 Reply Last reply Reply Quote 0
      • Rubix
        Rubix last edited by

        Thanks Irving,

        www.positivecollections.co.uk is the url

        I've tried to remove everything from the robot file and check again on webmaster, same thing happened  It's just blocking the main link

        1 Reply Last reply Reply Quote 0
        • Rubix
          Rubix last edited by

          When I click on Fetch As Google, i get 'Denied by robots.txt'' error.

          @Jesse: That's the main website, we've got other URLs.Error appears on positivecollections.co.uk

          jesse-landry 1 Reply Last reply Reply Quote 0
          • jesse-landry
            jesse-landry @Rubix last edited by

            That site is also being indexed. Again I ask, what makes you think it is not being indexed? (cause it is)

            1 Reply Last reply Reply Quote 0
            • Rubix
              Rubix last edited by

              When I click on Fetch As Google, i get 'Denied by robots.txt'' error.

              jesse-landry 1 Reply Last reply Reply Quote 0
              • jesse-landry
                jesse-landry @Rubix last edited by

                Okay. But that doesn't mean it isn't being indexed. Here's a fun test: Go to any page on your website and select a string of two or three sentences. Google it. Does the page come up in the SERPs?

                (I did this to 3 pages on your site and it worked for all of them. Therefore, your site is being indexed.) Why do you need to Fetch?

                Rubix 1 Reply Last reply Reply Quote 0
                • irvingw
                  irvingw last edited by

                  TECHNICAL ISSUE

                  It's your cookie policy blocking bots from spidering. Need to fix that at the server level. Easy fix! 🙂

                  http://www.positivecollections.co.uk/cookies-policy.aspx

                  Your robots.txt is fine.

                  Rubix 1 Reply Last reply Reply Quote 1
                  • Rubix
                    Rubix @jesse-landry last edited by

                    I couldn't relay on SERPS as the website is old, it's been indexed for quite so i didn't think that SERP results would change that quick. I've been receiving the error since yesterday.

                    It's on SERPS today but would it be there tomorrow? The reason I am saying that is because when i change the Page Title, it doesnt get changed on SERPS instantly, it takes a day or so before i see the changes on SERPS.

                    1 Reply Last reply Reply Quote 0
                    • Rubix
                      Rubix @irvingw last edited by

                      Thanks Irving, I need a little more help, I am not quite sure if I understand it. What is it that needs to be fixed here?

                      irvingw 1 Reply Last reply Reply Quote 0
                      • irvingw
                        irvingw @Rubix last edited by

                        Try to spider your site with this link checker tool

                        http://brokenlinkcheck.com/

                        bots cannot accept cookies and your site is requiring cookies to be enabled in order to be visited so Google cannot access the site because you are not allowing the visit without the cookie being dropped is most likely the issue.

                        Disable cookies on your browser and clear your cache and see what happens when you try to visit your site, are you blocked?

                        These discussions may possibly help

                        http://www.highrankings.com/forum/index.php/topic/3062-cookie-and-javascript/

                        http://stackoverflow.com/questions/5668681/seo-question-google-not-getting-past-cookies

                        Rubix 1 Reply Last reply Reply Quote 1
                        • Rubix
                          Rubix @irvingw last edited by

                          Thanks Irving, it worked 🙂

                          1 Reply Last reply Reply Quote 0
                          • 1 / 1
                          • First post
                            Last post
                          • Robots.txt advice
                            Martijn_Scheijbeler
                            Martijn_Scheijbeler
                            0
                            3
                            105

                          • Robots.txt Help
                            GlobeRunner
                            GlobeRunner
                            0
                            5
                            162

                          • Robots.txt Allowed
                            GlobeRunner
                            GlobeRunner
                            0
                            4
                            118

                          • Robots.txt
                            Travis_Bailey
                            Travis_Bailey
                            0
                            4
                            107

                          • Robots.txt
                            TomRayner
                            TomRayner
                            0
                            5
                            137

                          • Robot.txt help
                            evolvingSEO
                            evolvingSEO
                            0
                            23
                            203

                          • Recovering from robots.txt error
                            BlueprintMarketing
                            BlueprintMarketing
                            0
                            11
                            1.8k

                          • Robots.txt unblock
                            Elchanan
                            Elchanan
                            0
                            5
                            4.3k

                          Get started with Moz Pro!

                          Unlock the power of advanced SEO tools and data-driven insights.

                          Start my free trial
                          Products
                          • Moz Pro
                          • Moz Local
                          • Moz API
                          • Moz Data
                          • STAT
                          • Product Updates
                          Moz Solutions
                          • SMB Solutions
                          • Agency Solutions
                          • Enterprise Solutions
                          • Digital Marketers
                          Free SEO Tools
                          • Domain Authority Checker
                          • Link Explorer
                          • Keyword Explorer
                          • Competitive Research
                          • Brand Authority Checker
                          • Local Citation Checker
                          • MozBar Extension
                          • MozCast
                          Resources
                          • Blog
                          • SEO Learning Center
                          • Help Hub
                          • Beginner's Guide to SEO
                          • How-to Guides
                          • Moz Academy
                          • API Docs
                          About Moz
                          • About
                          • Team
                          • Careers
                          • Contact
                          Why Moz
                          • Case Studies
                          • Testimonials
                          Get Involved
                          • Become an Affiliate
                          • MozCon
                          • Webinars
                          • Practical Marketer Series
                          • MozPod
                          Connect with us

                          Contact the Help team

                          Moz logo
                          © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                          • Accessibility
                          • Terms of Use
                          • Privacy