How to allow all Robots.txt works primarily by exclusion. You exclude the files and folders that you don’t want to be accessed, everything else is considered to be allowed. If you want bots to be able to crawl your entire site, then you can simply have an empty file or no file at all. Or you can put this into your robots.txt file to allow all: User-agent: * Disallow: