Robots.txt

[remino63]

My campaign hse24 (www.hse24.de) is not being crawled any more ...

Do you think this can be a problem of the robots.txt?

I always thought that Google and friends are interpretating the file correct, seen that he site was crawled since last week.

Thanks a lot

Bernd

NB: Here is the robots.txt:

User-Agent: *
Disallow: /

User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Mobile
User-agent: MSNBot
User-agent: Slurp
User-agent: yahoo-mmcrawler
User-agent: psbot
Disallow: /is-bin/
Allow: /is-bin/INTERSHOP.enfinity/WFS/HSE24-DE-Site/de_DE/-/EUR/hse24_Storefront-Start
Allow: /is-bin/INTERSHOP.enfinity/WFS/HSE24-AT-Site/de_DE/-/EUR/hse24_Storefront-Start
Allow: /is-bin/INTERSHOP.enfinity/WFS/HSE24-CH-Site/de_DE/-/CHF/hse24_Storefront-Start
Allow: /is-bin/INTERSHOP.enfinity/WFS/HSE24-DE-Site/de_DE/-/EUR/hse24_DisplayProductInformation-Start
Allow: /is-bin/INTERSHOP.enfinity/WFS/HSE24-AT-Site/de_DE/-/EUR/hse24_DisplayProductInformation-Start
Allow: /is-bin/INTERSHOP.enfinity/WFS/HSE24-CH-Site/de_DE/-/CHF/hse24_DisplayProductInformation-Start
Allow: /is-bin/intershop.static/WFS/HSE24-Site/-/Editions/
Allow: /is-bin/intershop.static/WFS/HSE24-Site/-/Editions/Root%20Edition/units/HSE24/Beratung/

[NickPateman81]

User-Agent: *
Disallow: /

That is blocking every bot from crawling anything.

User-Agent: * = every robot

Disallow: / = every directory

[krissy-cca]

Yep, You just made your site invisible! >.<

Personally I just disallow areas I don't want indexing and let all bots crawl.

User-Agent: *

Disallow: /whatever I don't want indexed

[Tom-Anthony]

Hallo Bernd,

Of course, I agree with everyone else that you need to fix your robots.txt file.

However I'd also add the suggestion that you setup Google Webmaster Tools for your site. These will help inform you about crawl errors and your robots.txt file and might be helpful for you in future.

Also whilst having a quick look at your site I noticed some duplicate page title issues. Make sure you are tracking your site with SEOmoz's campaign tool. It will really help you find these types of issues.

Viel Glück!