How to Configure Robots.txt File

[seobac]

How to configure correctly the Robots.txt File of the website. Need proper process to follow.Because a lot of my website URLs are excluded by google with the issues in the Robots.txt file.

[effectdigital]

It depends on how your unique website creates URLs and how they are formatted, it also depends upon the current contents of your robots.txt file. If you can share some examples of URLs that are blocked, which you think should not be blocked, and also the contents of your robots.txt file, someone can probably tell you what you did wrong

[jasongmcmahon]

In addition to  effectdigitals comments, it partially depends on the CMS. For example a Standatrd WordPress robots.txt is formatted:

**User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php**

I prefer to add a link to the sitemap.xml:

**User-agent: ***
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.website.com/sitemap.xml

To allow all web crawlers access to all content, just use:

**User-agent: ***
Disallow:

Refer to https://moz.com/learn/seo/robotstxt for further advise.