Email Testing Showdown: Holdout vs. Comp.
In Website Magazine's October issue, Daryl Logullo of Kellyco Metal Detectors addressed how the days of pressing send for bulk-and-blast emails are over. The practice damages deliverability, increases complaints, raises bounce rates and can cripple an email program overnight. This is where dynamic targeting and segmentation save the virtual day. Logullo addressed several ways to leverage these effective practices, two of which didn't make it to print, but are included below.
Holdout testing is a surefire way to determine whether a mailing variable is positively or negatively affecting email performance. Let’s say a list contains 100,000 records and your teams wants to test whether changing the mailing frequency impacts sales.
(a) Take 25,000 names at random and split them into a simple A/B files - Group A: 12,500, Group B: 12,500.
(b) For group A, regularly mail twice a week over a defined period (30 days) for a total of 60 email messages a month.
(c) Group B, “hold out” and test a change in the send frequency, emailing them one time a week for a total of 30 messages a month.
(d) Compare performance (sales) and make decisions accordingly.
Comparable (or comp) segment analysis is another important measurement designed to lift performance that originated in offline marketing, including mail and catalog marketing. It can easily be applied to email marketing and should.
(a) Identify all customers who bought exactly three times from your emails last year during Q1/Q2 period. Segment the names.
(b) Do a similar analysis (comparable) for this year Q1/Q2 of names who also bought three times.
(c) Study the revenue performance for these three-time multi-buyers and compare.
(d) Analyze data and make decisions accordingly.