Can anyone tell me why the bot has only picked up one page?

[designsecrets]

www.namebadgesinternational.co.nz

After the 2nd week, I changed the robots.txt file to allow ALL robots on the website, but it still hasn't gone through any pages after another crawl

Any help would be hugely appreciated.

[Aran_Smithson]

Hi,

A quick search on Google for site:namebadgesinternational.co.nz shows other pages are being indexed.

I can see a few other issues too, for instance the canonicalization of your domain and homepage, your homepage can be accessed via www, non-www, index.php and though root /

It may be a good idea to log into your Google Webmaster tools account (assuming you have one) and check Diagnostics for crawl errors and similar.

Also our robots blocks /cgi-bin but leaves 404 and login pages open to indexation (i would personally block these as i don't want my visitors landing on them).

Check out http://www.seomoz.org/learn-seo/canonicalization for more info on canonicals.

best of luck...

Aran

[ShaMenz]

Hi designsecrets,

I'm assuming that you are referring to the SEOmoz bot (Roger) and seeing only one page of your site crawled.

If this is correct, then there has been a similar known issue in the past.

You will need to go to SEOmoz Help and ask the Help Team to look at the campaign and sort out the problem for you. Make sure that you provide them with your campaign number and/or domain to make it easy for them to locate the problem.

You can also access the Help area by clicking the Help link at the top right of any page on the site.

Hope that helps,

Sha

[KeriMorgret]

https://seomoz.zendesk.com/entries/409821-why-isn-t-my-site-being-crawled-you-re-not-crawling-all-my-pages

This post by the help desk about why a full crawl isn't being done may also be helpful.

[designsecrets]

This post is deleted!

[ShaMenz]

Hi again,

Since there are a lot of directives in this that are specific to certain conditions in your site that we cannot see, you should leave them alone.

Your redirect should appear at the beginning of the .htaccess anyway, so your best course of action is:

1. Make a copy of your existing .htaccess file with a different name so that you cannot accidentally lose it by overwriting
2. Place your redirect code at the beginning of the .htaccess (remembering that "RewriteEngine On" should appear only once)
3. Save and upload your new .htaccess file
4. Check in your browser to see if the redirect works (as many known pages as possible)
5. If it works, CELEBRATE!
6. It it breaks, replace your file with your copy of the original and ask the original developer, or one with access to the detail of your site to sort it out for you.

Incidentally, the rule you have above will redirect from www to non-www...just to be sure that is what you intended.

Hope that helps,

Sha