The Interactive Advertising Bureau has released a new list of spiders, part of
its long-standing effort to help advertisers filter out nonhuman visits. The
list is to be maintained in conjunction with ABC Electronic in the UK.
The joint effort aims to catalogue all the common bots
and spiders, in an effort to warn publishers, analytics firms, and third-party
ad servers about which ads are being served to automated programs instead of
eyeballs. It covers spiders known to visit sites in the US and UK. More
in this IAB press release.
You have to be an IAB subscriber to receive the list.
My initial thought is that this is certainly a step in
the right direction. The success of such a list would depend on how thorough and
up to date it is kept. For example, no information on the contents of the list
was provided but it could be safely assumed that there would be information on
the IP address and user-agent of the spider. Both of these can be altered
relatively quickly. If it is a continually updated list it will work but there
needs to be a way to either dynamically prohibit said spiders in real time or
simply a guideline (possibly set forth by the IAB) about how much to credit
advertisers based on the activiity of the list. The IAB suspects between 0.5%
and 3% of ad impressions are generated by bots and spiders.
IAB, interactive advertising bureau, robots, ad impressions