Cannot crawl website with redirect intalled on subdomain url

[Pascal_Hammerli]

Hi!

I want to crawl this website : http://www.car-moderne.ch.

I tried a got back the crawl just for that one url (not for all the pages of the website). This single line cvs says that the status of the http://www.car-moderne.ch is 200, but in fact it is a redirect 301 to http://www.car-moderne.ch/fr where the live home page is (actually the Moz bar sees the 301, not the 200 as the single-lined crawl does).

How can I proceed in this case (a 301 redirect being installed on the subdomain url) to still be able to have a full-fledged juicy cvs with all the broken links, duplicate content, etc.

Thank you for your help!

Pascal Hämmerli

[ChiarynMiranda]

Hey Pascal,

Sorry for the confusion here! It looks like the subdomain, www.car-moderne.ch, returns a 200 HTTP status to our crawler and to other crawlers, such as the hurl.it tool. In the body of the screenshot I attached from the hurl.it tool, the only code there is the number 404, so basically the site is serving a page with no crawlable data. The page isn't redirecting and it doesn't return any real source code, so there is no data for us to include in the crawl. I would recommend working with your webmaster to resolve this issue and to get the page to correctly serve a 301 redirect to the /fr version of the site to all crawlers.

I can see that the site is correctly responding with a 301 redirect for some crawlers, such as this test I ran as googlebot, but the response doesn't seem to be consistent. One thing you will want to be sure to have your webmaster check is how the site responds to user-agents that are hosted on Amazon Web Services, as some of our crawlers and the hurl.it crawl are both hosted through AWS.

Once the issue of the HTTP response is resolved, you should be able to get much better data from the crawl test tool.

I hope this helps! Please let me know if I can help you with anything else.

Chiaryn

[Pascal_Hammerli]

Dear Chiaryn,

Thank you for your very helpful reply.

This website is hosted on a partner agency who create the website and I only act as a SEO consultant for them. What you say is very helpful because it means their home-made CMS should be corrected to provided better 301 redirection.

I wish you a good day,

Pascal

[ChiarynMiranda]

So glad to help, Pascal!