This is a draft of the
article L. Dintzer and R. Grossman,
Test-Measure-Refine:
80 Years Later - It Still Works,
which later appeared
in DM News, March 5, 2002.
In uncertain times, it is a good idea to go back to the basics. The best introduction to email marketing was written about 80 years ago by Claude Hopkins. In a slim 95 page book called Scientific Advertising, he covers all the ideas essential for successful email campaigns today. Of course, since he wrote it 80 years ago, you won't find discussions of what list server to use, but frankly, someone else in your organization should be worrying about that anyway. The book has been reprinted often, and unlike books today about new media advertising, which become obselete in the three months it takes them to be printed, is still timely and relevant.
One of the fundamental lessons in Scientific Advertising is the importance of testing. Testing and measuring the results is one of the things that separates effective email campaigns from simple email blasts.
It is common today to hear campaign managers dismiss the importance of testing due to the low CPM costs. The problem with this point of view is that without testing response rates do not improve. Campaigns became one time events and not part of a process which improves response rates over time. Low CPM costs simply mean that tests can involve larger samples.
The basic Test-Measure-Refine (TMR) process is simple:
It is important to measure several different alternatives in a single mailing. The reason is simple: each mailing is different and the lesson learned is about which alternative is most effective. It is very difficult to extract this information if the alternatives are tested across mailings. This is one of the most basic mistakes made.
There are many factors that can be tested. It is usually wisest to focus on the most important:
All of these elements are critical to successful email campaigns. The experience of the marketer and the campaign manager should guide the order of the factors to be tested, and the variations tested.
One of our clients used the TMR method for planning a series of e-mail campaigns. The result was a doubling of the response rate, and just as important, a better understanding of the prospect base. With this understanding, future campaigns improved.
Test. Our client is a major retailer of women’s health products. It is usually a good idea first to test the offer. This is generally the most important factor. Three offers were tested. The least expensive was a referral letter. In the middle was was a sweepstakes offer. We also tested a high cost offer consisting of a free product sample. The goal of the test was to learn which offer was most cost effective.
Measure. The response rates from the three offers were:
| Offer | Response |
|---|---|
| referral letter | 3.2% |
| sweepstakes | 3.5% |
| product sample | 7.8% |
Refine. As might be predicted, the free product generated the highest response rate, but was the least cost effective offer. There was not a substantial difference between the letter and sweepstakes response rates. While the referral letter is more cost effective, the marketer’s experience was that sweepstakes do substantially better than referral letters, and was reluctant to abandon sweepstakes offer. The marketing and campaign managers decided to drop product sampling in further campaigns, but continue to test the referral letter and sweepstakes offers.
Test. We tested the positioning of the email piece next. Copy focusing on the health benefits of the product was prepared; copy focused on the nutrition benefits was prepared; and more general copy emphasising both was also prepared. Each was tested using both the referral letter and sweepstakes offer.
Measure. The results of the campaign test are below. Notice that TMR helped refine the effectiveness of both the offer and the positioning. The response rates increased to over 5% from the 3.2%- 3.5% of the initial campaigns.
| Positioning | Referral | Sweepstakes |
|---|---|---|
| Health | 2.8% | 3.3% |
| Nutrition | 3.6% | 5.2% |
| Neutral | 4.9% | 4.4% |
Refine. The health positioning was dropped as relatively ineffective, retaining the nutrition and neutral positionings. The nutrition/sweepstakes combination garnered the highest response rate, but only marginally higher than the neutral/referral combination. Since the cost of the sweepstakes offer was higher, future campaigns used the neutral/referral combination.
Test.We considered targeting next and looked for a segment of the target popultion of women based upon age, geography, income, education, and number of of children in the household. For simplicity, we tested two cells: women under 35 and women over 35.
Measure. The tests showed a sharp difference between women under 35 and over 35. Using the Referral offer and the neutral positioning, the women over 35 were more than twice as responsive:
| Segment | Response |
|---|---|
| Women under 35 | 3.2% |
| Women over 35 | 7.2% |
Refine. This was an easy choice to implement. Future campaigns would target women over age 35.
Test. Now that the TMR approach narrowed the offer type, the positioning, and the target, we tested the source of the names next. This was done to increase the size of the target population. The campaign was sent to the original name provider and two alternative suppliers.
Measure. Using the referral letter offer, the neutral positioning, and females over age 35, the three list sources generated the following response rates:
| List | Response |
|---|---|
| List 1 (original list) | 7.3% |
| List 2 | 5.9% |
| List 3 | 5.2% |
Refine. List 1 had the highest response, and while the responses of the other lists was lower, they were all significantly higher than the orginal unrefined 3.2% response rate prior to the TMR process.
Today, there is a growing belief that targeting doesn't matter, given the low CPM price. Our experience is different. Lists burn out quickly. Good targeting decreases burn out and maintains the value and usefulness of a list. Good targeting can significantly increase response rate or maintain the same response rate, while decreasing the number mailed, saving other names for other days. The test-measure-refine loop is as valuable today as it was 80 years ago when Hopkins described it.
Dr. Dintzer is a direct marketing consultant with over fifteen years of experience in the industry. He has worked at IRI and yesmail.com. Dr. Grossman is the President of Open Data Partners, which provides consulting services, outsourced data services, and litigation support services related to data. Dr. Grossman is also the Director of the Laboratory for Advanced Computing at the University of Illinos at Chicago, which develops internet-based technologies.
For more information, please contact Open Data Partners www.opendatagroup.com.