How Google Carwler Cached Orphan pages and directory?

[Guest]

This post is deleted!

[darshit21]

Hi Dan,

No, i am not exclude "demo" directory from robots.txt for any search engine.

I am not using wordpress its simple stattic HTML website (Not using any type of CMS).

[darshit21]

Hi Thetjo,

I know about it.

My question is that how Google Crawl it without any referral link?

Thanks.

[darshit21]

Hi JoelHit,

NO, There is not any single refferal link to "Demo" directory from entire website and also from third party websites.

I am aware about Google Crawling and Indexing Systems.

Thanks.

[StalkerB]

<conspiracy-hat></conspiracy-hat>

Did either you or your client use gmail when you sent him the demo link?

Regardless, Dan's advice to noindex and block the directory from spiders is the future when doing development work.

[darshit21]

Hi Barry,

Yes, We were used Gmail for reporting.

Is it make any sense??

[ChrisMacNaughton]

The <conspiracy hat="">side of things was him commenting that Google is sometimes accused of processing everything in Gmail and could have possibly pulled your link to the demo directory from that.</conspiracy>

[StalkerB]

Yup, correct.

I was certain I'd replied to this

Anyway, you ever notice how the ads in gmail are always relevant to the content of your emails? Google are totally reading them

[darshit21]

Is google crawling our mails?

Is it possible?

[KeriMorgret]

Try putting the URL into Google and see if you find any pages linking to it.

I knew a company that created a test site that was a copy of a live site (made with a specific hosted CMS). Didn't exclude the test site in robots because "we all know we won't link to it so it'll be ok". Site got indexed, and it was because a person at the company was having problems with the implementation of the test site, went to the help forum (which person didn't think would be indexed) and posted the URL to the test site.

I found the above by just putting in the URL of the test site into Google, and I saw the post in the help desk. You might try the same to see if somehow there is a rogue link.