Welcome to the Q&A Forum

In addition to  effectdigitals comments, it partially depends on the CMS. For example a Standatrd WordPress robots.txt is formatted: **User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php** I prefer to add a link to the sitemap.xml: **User-agent: *** Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.website.com/sitemap.xml To allow all web crawlers access to all content, just use: **User-agent: *** Disallow: Refer to https://moz.com/learn/seo/robotstxt for further advise.

My site also have this problem please help. Social Sathee

Thank you!