Home » Free Webmaster Tools » Robot Text File Generator Robot Text File Generator Share on Facebook Share Share on Pinterest Pin it Share on TwitterTweet Share on Google Plus Plus Please select which of the following web crawlers / engines you wish to allow access to your website. If you turn the engine ON, they will crawl and index the website. If you turn it OFF, they will not crawl nor index the website. Googlebot Google Image Google Mobile MSNBot PSBot Slurp Yahoo MM Yahoo Blogs Teoma Scrubby IA Archiver Generate Code Download File Here is the generated code. Please paste this into a blank htaccess file, or click the download button above. Press Generate Code for the code to appear here.