IAB Bots/Spider List
The Interactive Advertising Bureau has released a new list of spiders, part of its long-standing effort to help advertisers filter out nonhuman visits. The list is to be maintained in conjunction with ABC Electronic in the UK. The joint effort aims to catalogue all the common bots and spiders, in an effort to warn publishers, analytics firms, and third-party ad servers about which ads are being served to automated programs instead of eyeballs. It covers spiders known to visit sites in the US and UK. More in this IAB press release. You have to be an IAB subscriber to receive the list.
My initial thought is that this is certainly a step in the right direction. The success of such a list would depend on how thorough and up to date it is kept. For example, no information on the contents of the list was provided but it could be safely assumed that there would be information on the IP address and user-agent of the spider. Both of these can be altered relatively quickly. If it is a continually updated list it will work but there needs to be a way to either dynamically prohibit said spiders in real time or simply a guideline (possibly set forth by the IAB) about how much to credit advertisers based on the activiity of the list. The IAB suspects between 0.5% and 3% of ad impressions are generated by bots and spiders.