Title errors for pages behind a login
-
On our website we have content which is located behind a members login.
the SEOMoz crawl report has returned these pages with a "no title" error against them.
It appears that these pages are being crawled until the website prompts it to login. I can only presume that it follows the url but doesn't have an opportunity to crawl the meta data.
what is the solution for these pages?
401, so that the bots know these pages are behind a login?
do we implement anything to ensure "no index", "no follow"?
I searched the T'interwebs and couldn't find anything conclusive on this issue.
-
Well, I guess if these are secure pages the correct approach would be the HTTP 403 status code (Forbidden). For example, this is the way apache forbids access to PHPMyAdmin if you have set up IP filtering in PHPMyAdmin.conf. I am not entirely sure how your security is implemented or how the SEOMoz crawler deals with a 403 but it should just back off from these pages.
Alternatively, if your URL structure allows it, you could block the SEOMoz crawler from these pages in robots.txt. Assuming your secure pages were in a directory called /secure/ we would need:
User-agent: rogerbot Disallow: /secure/
Hope that helps!
Marcus