Plan email experiments with a clear hypothesis, a meaningful metric, an adequate sample, and an honest interpretation of uncertain results.
An email A/B test compares two versions under similar conditions. Its value is the decision it helps you make, not the winner badge in a dashboard.
Start with a question specific enough that either result would change your next email. "Does a screenshot help new users complete setup?" is useful. "Which email is better?" leaves too much undefined.
Use a metric close to the reason for sending. If the goal is activation, a completed product task is more useful than an open. For a newsletter, a relevant click or reply may be enough. Track complaints and unsubscribes as costs, even if they are not the primary metric.
Define the denominator. Clicks per delivered email and clicks per opener answer different questions. Open-based metrics are also affected by privacy features. Apple's Mail Privacy Protection, for example, can load an email's remote content in the background, so an open may be recorded for a message nobody read. Write down the exact calculation before the test starts.
Our email metrics guide explains where tracking can mislead you.
If you are not sure where to start, work down this list. The elements near the top decide whether the reader wants the message at all. A better button will not rescue an offer nobody wants.
| Element | What a clear result tells you |
|---|---|
| Audience or offer | Whether this group wants what you are asking them to do. Test it first, because every later element depends on it. |
| Subject line | Which framing gets the right readers to open and act. Judge it on clicks or the downstream action as well as opens. |
| Sender name | Whether readers recognize and trust the from name, such as a named person compared with the company name. |
| Preview text | Whether the line shown after the subject gives a reason to open. Some email clients shorten or hide it, so the effect depends on which clients your readers use. |
| CTA copy and placement | Whether readers understand what happens after the click and can find the link without hunting for it. |
| Layout | Whether the structure helps readers reach the action. A layout change alters many things at once, so read the result as a package. |
| Send time | When this audience tends to act on this kind of message. Compare over more than one send, because a single day can be unusual. |
Send time is the one test where the versions go out at different times by design. Keep everything else identical, including how you split the audience, and measure both groups over the same length of window.
A hypothesis names the change, the audience, the metric, and the reason you expect a difference. The reason is the part you learn from. If the result goes the other way, it tells you which assumption to question.
If we [change] for [audience], [metric] will [rise or fall] because [reason].Some examples:
Any of these could fail, and a failure still teaches you something about your readers. A vague test such as "try a new template" teaches you very little whichever version wins.
A small change may be real but too small to justify extra production work. Choose the minimum effect that would change your decision, estimate the baseline rate, and use a sample-size calculation with a stated confidence level and statistical power.
There is no universal minimum such as 1,000 recipients. A common action needs a different sample from a rare purchase. A test looking for a very small lift needs more data than one looking for a large change.
For a small list, it may be better to test a substantial change over several comparable sends, or collect qualitative feedback, than to split one campaign into many tiny groups. Repeated campaigns introduce timing differences, so describe that limitation when interpreting the result.
Randomly assign eligible recipients to versions. Keep the audience rules, delivery window, and other material content comparable. If you change the subject, layout, offer, and landing page together, you are testing the whole package and cannot say which part caused the difference.
Avoid counting repeated sends to the same person as independent observations. For account-level outcomes, consider whether people in the same account could influence each other.
Set an observation window that allows the intended action to happen. Do not repeatedly peek and stop at the first favorable fluctuation unless your statistical method explicitly supports that approach.
Suppose one version gets 24 purchases and the other gets 27. Those counts alone do not establish a winner. You need the group sizes, assignment method, and an interval around the estimated difference.
If the result is inconclusive, say so. Keep the simpler version or use what you learned to design a stronger next test. A test that rules out a large improvement can still save you time.
Record the hypothesis, versions, audience, sample, dates, metric, result, and next decision. Link that record to the campaign. When reviewing subject lines, resist turning a single test into a permanent rule for every audience.
