Welcome to the Q&A Forum

Great, thanks.

With those 2 recommendations I have more than enough for the next crawler. Thank you both!

Hi, thanks for answering

Well, it looks doable. Will try t do it on next programmed crawler, trying to minimize exposed time.

Hw, your idea seems very compatible with my first approach, maybe I could also allow rogerbot through htaccess, limiting others and only for that day remove the security user/password restriction (from joomla) and leave only the htaccess limitation. (I know maybe I'm a bit paranoid  just want to be sure to minimize any collateral effect...)

*Maybe could be a good feature for Moz to be able to access restricted sites...

I'm in the middle of site development and wanted to start crawling my site with Rogerbot, but avoid googlebot or similar to crawl it.

Actually mi site is protected with login (basic Joomla offline site, user and password required) so I thought that a good solution would be to remove that limitation and use .htaccess to protect with password for all users, except Rogerbot.

Reading here and there, it seems that practice is not very recommended as it could lead to security holes - any other user could see allowed agents and emulate them. Ok, maybe it's necessary to be a hacker/cracker to get that info - or experienced developer - but was not able to get a clear information how to proceed in a secure way.

The other solution was to continue using Joomla's access limitation for all, again, except Rogerbot. Still not sure how possible would that be.

Mostly, my question is, how do you work on your site before wanting to be indexed from Google or similar, independently if you use or not some CMS? Is there some other way to perform it?
I would love to have my site ready and crawled before launching it and avoid fixing issues afterwards...

Thanks in advance.

Yes... analyzing data would be the best way to start... will try to bring more information for tomorrow and paste it here so we could speak with real numbers.

Thanks!

What happens with sites that have the complete list of products in the home page and in the "Products" section.

Would like to know your opinion as web developers and users and like SEO professionals.

I'm in duality like web developer, somehow my subconscience is telling me that is stupid to have it on both places... And the home page is getting nice results and comments from the users, so, would consider quitting the "Products" teaser view. But of course, eliminate the "Products" section is somehow... un natural....

And what happens with SEO, I read somewhere that it's not very good to have all the "rank" spread in so many links in the home page... I don't even understand it completely or neven sure they said rank or juice..., but it's something that is concerning me...

We are talking about 20 products, that can grow. I'm looking to have a better SEO, but also the UX is very important, even more than SEO (I know good UX is better SEO).

Thanks in advance to any contribution.

I started with one very nice tutorial in https://tutsplus.com/, it's a pity that can't find it now because I have to continue soon with it and it was very useful and free...

Thank you cprasad,

I always miss big and obvious, text messages...

Thank you Sanket, I do have more questions about that, but as I'm still learning how to use all the tools - would like to keep that offer for other moment and form some more precise question

I have set the campaign for the Root domain, hw, I'm receiving issues for both subdomains, demo and www. Our site is set to redirect all non www traffic to www, except demo, who has its own rules.

Is that normal? Should I contact the support team?

Thanks