Skip to Main Content

Build A Robots.txt File

Posted on 4.23.2007
As I look around, many sites still do not contain a robots.txt file. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see. The result of not having the right robots.txt file is often that spiders will run rampant on your site, indexing content you may very well not want public. To ensure good interaction with SE spiders, here are a few interesting robots.txt file builders (all free):

Related Posts and Articles:


Leave Your Comment

Login to Comment

Become a Member

Not already a part of our community?
Sign up to participate in the discussion. It's free and quick.

Sign Up


Leave a comment
    Load more comments
    New code