If you add a robots.txt file to the root folder of your website, it will aid in indexing by search engines. Search engines such as Google employ website crawlers, or robots, to evaluate all of the material on your website, including images, videos, and documents. Search engines may scan areas of your website that you don't want them to, such as the admin page. You can add these pages to the file so that they are expressly excluded from the search. In the Robots Exclusion Protocol, robots.txt files are used to block robots. Input the sites to be omitted from the file, and this website will produce the file for you in no time!

Please read the following guidelines carefully before you start using this tool:

1. The first and second rows are "Default - All Robots" and "Crawl-Delay", you should choose the "Default" value for all robots. If you don't want to modify them, just leave them as they are.

2. The third row is about Sitemap. Make sure you put Sitemap link of your site in the text box. If you don't have one, let create it with our XML Sitemap Generator tool.

3. Search Robots row. For example, you choose "Google" for search engines.

The first row is for search engines, the second row is for pictures, and the third row is for the mobile version of your website. Set the "Allowed" or any option you want.

Do the same for other search engines.

4. Restricted Directories: As the last option, you might disable crawlers from indexing certain sections of the page. Remember to add the forward-slash before entering the directory or page's URL.