Robots.txt Help

[Michael.Leonard]

I need help to create robots.txt file.

Please let me know what to add in the file. any real example or working example.?

[Martijn_Scheijbeler]

Include sitemaps. Disallow: Pages that you don't want indexed: search pages, login pages, core admin files.

[O2C]

A very basic robots.txt file would look something like the below

User-agent: *
Sitemap: http://www.yourwebsite.com/sitemap.xml
Disallow: http://www.yourwebsite.com/url-you-dont-want-indexed
Disallow: http://www.yourwebsite.com/another-url-you-dont-want-indexed

Hope that helps

[rachaelpracht]

This is the site I used to really get a good understanding of how to create a robots.txt file: http://www.robotstxt.org/ 

[GlobeRunner]

Michael, from what i can tell, your website is built using WordPress. We typically recommend installing the Yoast SEO plugin and using that--which will help with your robots.txt file. If you need more information, take a look here: https://yoast.com/wordpress-robots-txt-example/

Generally, most of your site won't need to be disallowed in the robots.txt file, unless you're using tags and categories on your site. Yoast typically helps disallow the proper directories that you need to disallow.

One thing that you need to be aware of is the fact that you don't want to disallow your .CSS or .JS files on your site, many of the themes nowadays will put those files in your wp-admin folder--which by default typically gets disallowed.