Google Adsbot crawling order confirmation pages?

[seoeuroflorist]

Hi,

We have had roughly 1000+ requests per 24 hours from Google-adsbot to our confirmation pages. This generates an error as the confirmation page cannot be viewed after closing or by anyone who didn't complete the order.

1. How is google-adsbot finding pages to crawl that are not linked to anywhere on the site, in the sitemap or linked to anywhere else?

2. Is there any harm in a google crawler receiving a higher percentage of errors - even though the pages are not supposed to be requested.

3. Is there anything we can do to prevent the errors for the benefit of our network team and what are the possible risks of any measures we can take?

This bot seems to be for evaluating the quality of landing pages used in for Adwords so why is it trying to access confirmation pages when they have not been set for any of our adverts?

We included "Disallow: /confirmation" in the robots.txt but it has continued to request these pages, generating a 403 page and an error in the log files so it seems Adsbot doesn't follow robots.txt.

Thanks in advance for any help,

Sam

[katemorris]

Hi Sam,

I can see how this might be concerning. Without knowing your site, I can't confirm anything but answers to your questions:

1. Bots have been known to "fill out forms" before and at least Googlebot has been known to find pages through the use of Chrome (a user using Chrome). There are many ways, but if you are sure that there is no link to it anywhere, I wouldn't worry about it.
2. No. That is what header codes are there for, to let the bots know what is there, what is forbidden, etc.
3. Other than robots.txt, there isn't any way to stop them from sending in requests. If it gets out of hand, you can try talking to AdWords directly, but more than likely, this is not causing an issue.

Overall, I'd just let it happen. Let them get the 403 error and they'll figure it out. As long as this isn't showing in the organic index, you should be fine.