Welcome to the Q&A Forum

Hi there! Tawny from Moz's Help Team here! I think I can help you figure out what's going on with your robots.txt file. First things first: we're not starting at the robots.txt URL you list. Our crawler always starts from your Campaign URL and goes from there, and it can't start at an HTTPS URL, so it starts at the HTTP version and crawls from there. So, the robots.txt file we're having trouble accessing is khadination.shop/robots.txt. I ran a couple of tests, and it looks like this robots.txt file might be inaccessible from AWS (Amazon Web Services). When I tried to curl your robots.txt file from AWS I got a 302 temporary redirect error (https://www.screencast.com/t/jy4MkDZQNbQ), and when I ran it through hurl.it, which also runs on AWS, it returned an internal server error (https://www.screencast.com/t/mawknIyaMn). One more thing — it looks like you have a wildcard character ( * ) for the user-agent as the first line in this robots.txt file. Best practices indicate that you should put all your specific user-agent disallow commands before a wildcard user-agent; otherwise those specific crawlers will stop reading your robots.txt file after the wildcard user-agent line, since they'll assume that those rules apply to them. I think if you fix up those things, we should be able to access your robots.txt and crawl your site! If you still have questions or run into more trouble, shoot us a note at help@moz.com and we'll do everything we can to help you sort everything out.

Hi Harini, Jo from the Moz help team here. I've had a look at your site and it looks like there is something server side that is blocking our bot. When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1 https://screencast.com/t/J3hhDTCM I'm also seeing this message in this third party tool. "The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)" All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through. I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2 You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2 You'll also want to make sure you are not blocking AWS (Amazon Web Services). Best of luck! Jo

Harini, you're asking a pretty broad question and the answer is likely a long a complex one only to be arrived at after careful analysis and consideration. That said, when I read your question I got the sense that you're focusing on on-page ranking factors only; that you may be forgetting about all the other things that go into boosting your rankings and conversions. For example, your site seems to be light on external backlinks compared to others competing for the same term (mobile app development company in India) and you seem to be using a mix of www and non-www URLs. If I've misunderstood, can you rephrase your question? You might get more specific responses.

I second this! Thanks for all the extra help you've been giving folks in the forum lately, @Hurf.  

According to my Moz bar, you don't have any backlinks.  How many do your competitors have? Do you have a solid plan to get quality backlinks?