Our Work

Six Big Mistakes to Avoid When Testing

One of the benefits of direct marketing is that decisions can be driven by data.  Unlike say television advertising, it’s very easy to draw a straight line between “I did this” and “a donor did this.” 

In most cases, we use this to our benefit to make the best use of fundraising dollars.  Unfortunately, there are more than a few ways that data can be misinterpreted and push you to the wrong conclusion.  Here’s a list of some potential mistakes during testing that can result in flawed conclusions:

  • Insufficient Test Quantities: Testing should give you a result that is accurate and repeatable.  Unfortunately, if the quantities that are tested are not large enough, you can wind up with a result that is essentially meaningless.  For simplicity and as a general rule; always shoot for test quantities that will result in 100 or more responses.  Doing so will lead you to results that are statistically significant.
  • Improper Segmentation: When splitting packages into test and control groups, it’s critical that the groups are equivalent to one another.  This is one of those infuriating little details that should go unspoken, but can wreck an otherwise good test.  Most data companies have their ducks in
    a row, but if you are working with someone new, it’s good practice to check output files to confirm that they are split correctly and of equal composition.
  • Ignore Roll-Out Costs: Your control package is likely being produced in higher volume than any of the test packages.  Make sure that you quote all test packages for roll-out costs, so that you have an apples-to-apples comparison.  Otherwise, none of your small quantity tests will ever be able to compete with your control in full volume.
  • Testing More than One Variable:  It’s ok to test completely different packages against one another, but if you are trying to maximize response on your control or “tweak-testing” then testing one variable at a time is the only way to know what changes are making the most impact.  Similarly. . .
  • Extrapolating on Findings:   Testing works because you run A vs. B and get a result.  It’s tempting to take that result and assume it’s a general fact for every situation.  Unfortunately, doing so can lead you to make some very wrong decisions. 
  • Wildcards: A lot can happen between a concept and delivery; unanticipated events in the marketing environment can nullify a test.  Some are obvious, such as when a hurricane strikes or unwanted press.  Others are less obvious, such as what your acquisition list panel is receiving at the same time as your appeal.  If any of your results seem blatantly inconsistent with what you have seen in the past, ask yourself “is there anything that could have impacted these results?”

It always amazes me how consistent donors will behave once you know some basic facts about their past behavior.  Our job as fundraisers is to use that information to make the best decisions possible. 

Email: currents@newrivercom.com



Katapult MarketingSix Big Mistakes to Avoid When Testing