11th January 2022
The robots.txt file is one of the main ways of telling a search engine where it can and can’t go on your website. All major search engines support the basic functionality it offers, but some of them respond to some extra rules which can be useful too. This guide covers the ways to use robots.txt on your website. What is a robots.txt file? Crawl directives The robots.txt file is one of a number of crawl directives. A robots.txt file is a text file that is read by search engines (and other systems). Also called the “Robots Exclusion Protocol”, the robots.txt file is […]













