We receive a lot of questions about A/B testing in email marketing, so we prepared for you an infographic with list of 6 crucial rules and most common mistakes. Learn how to use that powerful tool!
Rules of A/B testing
- Start with hypothesis
Don’t do tests on a whim or by chance. Always start with hypothesis, based on research, customer feedback or analytics.
- If I change … from … to …, I will achieve…
That’s how your hypothesis should look like. E.g.: If I change shape of the button from flat to 3D, I’ll achieve higher conversion.
Hypothesis should include:
- Tested element
- How to change it
- What will be achieved
Therefore you’ll be sure that tests are precise and relevant.
- Only 1 difference
The next step is to prepare 2 almost identical creations that vary only in 1 element – the one you mentioned in your hypothesis. Test 1 element at the time.
- Size of test groups
We recommend using calculator of test groups, e.g. Evan Miller’s. It will help you avoid testing on too small groups (lack of significance) or wasting too much part of your mailing list for testing (when you could be sending them right version).
- Success
How you will know if the hypothesis is right? Before sending your two versions you need to establish what will determine success. There are 2 most popular indicators: OR or CTR. Match chosen one to the element you test (OR for subject or sender’s name, CTR for elements of creation itself).
- Continuity
One test doesn’t give you universal knowledge on customers. Test must be run on regular basis and on plan.
Most frequent mistakes
- Testing without hypothesis
- Testing on whim („I’ll just see which one customers like better”), lack of plan
- Hypothesis based on intuition or feeling
- Lack of recording and documenting run test
- Testing many elements at once
- Wrongly defined success criteria (e.g. CTR for subject)
- Generalizing outcomes of test run in specific period (Holidays, sales, season peak)
- Lack of regularity.
Follow
