Best way to create robots.txt for my website

[zoe.wilson17]

How I can create robots.txt file for my website guitarcontrol.com  ?

It is having login and Guitar lessons.

[Guest]

This post is deleted!

[Mobilio]

I see that you're on WordPress.

This CMS create "virtual" robots.txt. You can see this here:
https://codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_Optimization

But on your website there is error in robots.txt and you should see in web server log files (access and error) why this is happening. Also you may need looking .htaccess because something preventing this text file to be accessed.

There is alternative way for using robots.txt in WordPress. All you need is to create new and blank robots.txt in same folder and put this there:
User-agent: *
Disallow:

Then save file and that's all. Now bad news - WP can't control indexing and crawling anymore.

[rootwaysinc]

Hi,

First you need to understand your website need, you have to decide which part of your website should not be indexed or crawled by SE bots, like your website provides user login and user areas, if you are providing private dashboard for your user then it should be blocked by robots.txt (or you can use meta tag to prevent robots from crawling and indexing your particular page like ) or you can learn more about robots.txt here https://moz.com/learn/seo/robotstxt

Hope it helps