Ping IP / IP Checker Traceroute DNS Checker WHOIS Lookup What is My IP PageRank Checker Reverse DNS

Robot.txt Generator

A robots.txt file is a short text file that resides in your home directory. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see. This tool will make your robots.txt file. As a little 'extra' it also includes a block to many unwanted spiders that only crawl your site in order to collect the email addresses stored on your pages.(SPAM-bots)


Enter the directories of your site that you don't want to be seen by search engines: 1 directory per line.

Leave blank if you don't want to block anything.
For enter private
For enter mp3/A
For and its subfolders enter mp3


Click this box if you want to disallow 'nasty' robots, like email-extractors and robots that copy your whole site to someone's hard drive. (e.g. WGET)

Block nasty bots


Related Questions

Bots!! Follow & Revisit?
Where can I find a spider emulator?

Related Tools

Spider Emulator