Build A Robots.txt File
Posted on 04.24.2007
- share this
As I look around, many sites still do not contain a robots.txt file. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see. The result of not having the right robots.txt file is often that spiders will run rampant on your site, indexing content you may very well not want public. To ensure good interaction with SE spiders, here are a few interesting robots.txt file builders (all free):
Related Posts and Articles:
Discover the Tactics and Techniques of the Top Digital Enterprises
today in Website Magazine's Special Spring Issue - Web 100.
Become a Member
Not already a part of our community? Sign up to participate in the discussion. It’s free and quick.