Build A Robots.txt File

As I look around, many sites still do not contain a robots.txt file. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see. The result of not having the right robots.txt file is often that spiders will run rampant on your site, indexing content you may very well not want public. To ensure good interaction with SE spiders, here are a few interesting robots.txt file builders (all free):

• Robots.txt Builder at Clickability

• 1-Hit.com Robots.txt Generator
  

• McAnerin Netowrks Robot Control Code Generation Tool

Related Posts and Articles:

• Dancing With Robots
• Robots Meta Tag: Proper Use
• Google's Robots.txt Sitemap