robots.txt - What to Put in It
Example of a robots.txt file disallowing several folders and the files within those folders.
Use a Text Editor
As it's a text file it can be edited by any text editor, such as Notepad (PC), TextMate or TextWrangler (Mac).
The "/robots.txt" file contains one or more records, and frequently has just a single record something like this:
User-agent: *
Disallow:
1) Example above allows all robots full access to your site.
User-agent: *
Disallow: /
2) Example above excludes all robots access to your site.
User-agent: Bing
Disallow:User-agent: *
Disallow: /
3) Example above allows just one robot (Bing) to access your site.
User-agent: NastyBot
Disallow: /
4) Example above excludes just one robot from gaining access.
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/
5) Example above excludes all robots from certain parts of the server.
Each URL must have its own "Disallow" line. You cannot put "Disallow: /cgi-bin/ /tmp/ /private/" on one line.
Another important point is that you cannot have blank lines in a record, as they are used to delimit multiple records.
Once you've finished editing, it's time to test your robots.txt file...
Reply
