Skip to Main Content

Email Testing Showdown: Holdout vs. Comp.

Posted on 8.31.2013

In Website Magazine's October issue, Daryl Logullo of Kellyco Metal Detectors addressed how the days of pressing send for bulk-and-blast emails are over. The practice damages deliverability, increases complaints, raises bounce rates and can cripple an email program overnight. This is where dynamic targeting and segmentation save the virtual day. Logullo addressed several ways to leverage these effective practices, two of which didn't make it to print, but are included below. 

Holdout testing is a surefire way to determine whether a mailing variable is positively or negatively affecting email performance. Let’s say a list contains 100,000 records and your teams wants to test whether changing the mailing frequency impacts sales. 

(a) Take 25,000 names at random and split them into a simple A/B files - Group A: 12,500, Group B: 12,500. 

(b) For group A, regularly mail twice a week over a defined period (30 days) for a total of 60 email messages a month. 

(c) Group B, “hold out” and test a change in the send frequency, emailing them one time a week for a total of 30 messages a month.

(d) Compare performance (sales) and make decisions accordingly.

Comparable (or comp) segment analysis is another important measurement designed to lift performance that originated in offline marketing, including mail and catalog marketing. It can easily be applied to email marketing and should.  

(a) Identify all customers who bought exactly three times from your emails last year during Q1/Q2 period. Segment the names.

(b) Do a similar analysis (comparable) for this year Q1/Q2 of names who also bought three times. 

(c) Study the revenue performance for these three-time multi-buyers and compare. 

(d) Analyze data and make decisions accordingly.



Today's Top Picks for Our Readers:
Recommended by Recommended by NetLine

Leave Your Comment

Login to Comment

Become a Member

Not already a part of our community?
Sign up to participate in the discussion. It's free and quick.

Sign Up


Leave a comment
    Load more comments
    New code