The Moz Q&A Forum

    • Forum
    • Questions
    • My Q&A
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Q&A Forum
    2. Categories
    3. Intermediate & Advanced SEO
    4. Google attempting crawl of URLs with parameter randomly placed in URL

    Google attempting crawl of URLs with parameter randomly placed in URL

    Intermediate & Advanced SEO
    5 2 1.6k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • AdeccoGlobal
      AdeccoGlobal last edited by

      This post is deleted!
      1 Reply Last reply Reply Quote 0
      • CleverPhD
        CleverPhD last edited by

        This is not a Google issue. You have an issue with the way your server is configured. It is showing a different link for when a bot is visiting your site versus when someone is browsing the site.  This is why you can't see it.    Google can only see what you show it.

        Here is how you find this.   You can run a tool like Screaming Frog or there are plugins for FF or Chrome that lets you change the user agent and how the page is shown

        https://chrome.google.com/webstore/detail/user-agent-switcher/ffhkkpnppgnfaobgihpdblnhmmbodake

        https://addons.mozilla.org/en-us/firefox/addon/user-agent-switcher/

        If you do not want to mess with plugins you can use the good ole rex swain tool

        http://www.rexswain.com/httpview.html

        Start with one of your pages that list jobs - I found this one

        http://www.office-angels.com/en-GB/Pages/all-job-results.aspx?kws=Catering%2c+Restaurant+%26+Bars

        Take that URL and put it into the RexSwain Tool in the URL section.  Dont change anything else and press submit. You will get a page with a bunch of code (shows you the source code that is returned).  Use the search page function in your browser to look for the offending "job-id?/"   you should not find it.

        Click back and go back to the form.   Leave the URL in place and now add "Googlebot/2.1 (+http://www.google.com/bot.html)" as the user agent and submit.  Do the same search with "job-id?/" you will see it is added in front of all the links to your jobs.  See image Rex Swain Crawl as Google Bot attachment below.

        If you do this same exercise with one of the plugins, just change the agent to Google and then hover over the link.  See image Page in Chrome with Google User Agent attachment below.  Bottom left you see the "bad" url and top right you see my User agent set to On to act as google.

        Get with your IT guys as you are technically cloaking right now, showing 1 thing to Google and another to users.  You are also potentially showing duplicate content to Google as well for 2 URLs with the same job.

        Also, I would take out that parameter handing in GWT, that has nothing to do with all of this.

        Good Luck!

        fllIsG4 DvSK6Dp

        AdeccoGlobal 1 Reply Last reply Reply Quote 1
        • AdeccoGlobal
          AdeccoGlobal @CleverPhD last edited by

          You sir, are a prince among men.

          Yes that is exactly it, thank you! I've been tearing my hair out over this for weeks trying to work this out.

          There seems to be some sort of condition built in to the job results page designed to distinguish page edit mode vs. normal view mode that's causing the issue, we'll have to untangle it next week.

          Anyway, thanks again!

          CleverPhD 1 Reply Last reply Reply Quote 1
          • CleverPhD
            CleverPhD @AdeccoGlobal last edited by

            My pleasure!

            CleverPhD 1 Reply Last reply Reply Quote 0
            • CleverPhD
              CleverPhD @CleverPhD last edited by

              One more - once you do fix the issue of the display on the page, you will want to 301 all the rogue URLs to the correct ones just to get that cleaned up for the Goog or anyone else who might visit.  That will also take care of the 404 issues in GWT.

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              • First post
                Last post
              • Google Only Indexing Canonical Root URL Instead of Specified URL Parameters
                Nitruc
                Nitruc
                0
                3
                57

              • We 410'ed URLs to decrease URLs submitted and increase crawl rate, but dynamically generated sub URLs from pagination are showing as 404s. Should we 410 these sub URLs?
                jeffchen
                jeffchen
                0
                3
                189

              • URL Parameter & crawl stats
                katemorris
                katemorris
                0
                4
                232

              • What is the value of Google Crawling Dynamic URLS with NO SEO
                ingageseo
                ingageseo
                0
                3
                300

              • Can I, in Google's good graces, check for Googlebot to turn on/off tracking parameters in URLs?
                john4math
                john4math
                0
                8
                214

              • If I had an issue with a friendly URL module and I lost all my rankings. Will they return now that issue is resolved next time I'm crawled by google?
                Pete79
                Pete79
                0
                3
                226

              • Link to Google Places, or Google Maps?
                MiriamEllis
                MiriamEllis
                0
                4
                463

              • 400 errors and URL parameters in Google Webmaster Tools
                STPseo
                STPseo
                0
                2
                1.7k

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy