If you add a robots.txt file to the root folder of your website, it will aid in indexing by search engines. Search engines such as Google employ website crawlers, or robots, to evaluate all of the material on your website, including images, videos, and documents. Search engines may scan areas of your website that you don't want them to, such as the admin page. You can add these pages to the file so that they are expressly excluded from the search. In the Robots Exclusion Protocol, robots.txt files are used to block robots. Input the sites to be omitted from the file, and this website will produce the file for you in no time!