A/B test your marketing email

An A/B test measures engagement for different versions of the same email with a sample of your recipients. After creating your marketing email, you'll create a version B to test how a single variation impacts email opens or clicks.

Depending on how each version of your email performs over a period of time, the tool can pick a winning version to send to the rest of your recipients list.

The instructions below pertain to creating an A/B test with a regular marketing email. If you want to automate your A/B email tests with the workflows tool, check out this article.

Please note: if you're creating a custom report in HubSpot, you can only filter for the Email content ID of variation A of an A/B test email. Analyzing data for variation B in a custom report is not currently supported.

Create an A/B test email

• In your HubSpot account, navigate to Marketing > Marketing Email.

• Click Create email. This will be version A of your A/B test.

• In the top left of the editor, click test Create A/B test.
• In the dialog box, enter a name for the variation B email, then click OK.
• After creating both versions of your A/B test email, you can switch between the two at any time. In the upper left of the email editor, click the name of the version.

• While editing your variations, you can consider varying:

  • Offers: experiment with the medium of the offer. You might test an eBook versus a whitepaper or video.
  • Copy: experiment with the formatting and style of the content. You could test plain paragraphs versus bullet points or a longer block of text vs a shorter block of text.
  • Email sender: try sending the email from an employee's email address instead of a generic department address.
  • Image: try out different images to see how the conversion rate is influenced.
  • Subject line: play around with the length of the subject line or add personalization.
  • Whole email: the fastest way to achieve drastic results and produce a landing page that drives a lot of conversions is to test the entire email. Make iterations to the whole email that affect image placement, subject line, and its copy. Once you have a statistically significant result pointing to the variation that performed better, you can continue optimizing through smaller tweaks. 
• Once you've finished editing your A and B email variations, finish configuring your test:
  • Click the name of the version you're editing in the upper left and switch back to version A.
  • While editing version A of your email, click the name of version A in the upper left again, then click Manage test.

• • In the dialog box, configure each of the following test settings:
    • A/B distribution: use the slider to decide what percentage of contacts will receive Variation A and Variation B. You can also set the slider to send Variation A to 50% and Variation B to 50%. This will send one email variation to half of the contacts in the list, and the other email to the other half of contacts. This lets you analyze the statistics of the two emails after the send to see which variation was more successful.
    • Winning metric: if you've opted to send variations A and B to a smaller group, then send the winning version to the remaining recipients, click the Winning metric dropdown menu to select the metric that determines how the winning variation will be chosen: Open rate, Click rate, or Click through rate.
    • Test duration: use the slider to set a time period to gather data before choosing a winner and sending the winning variation. If the results of the test are inconclusive after the test duration, HubSpot will automatically fallback to sending Variation A to the remaining recipients.

• • When you're done, click Save changes.
• Review your email sending settings and configure the recipients for the A/B test. Note that for A/B tests, you must select at least one contact list for your recipients.
• Click Review and send in the top right of the email editor.

Please note: you can run a full-split (50/50) A/B test using any sample size. All other sample splits require a list that includes at least 1,000 recipients, excluding any contacts that previously hard bounced or unsubscribed.

Cancel an A/B test

To cancel an A/B test before the test is complete:

• In your HubSpot account, navigate to Marketing > Marketing Email.
• Click the name of the A/B email you want to cancel.
• Based on whether the test has already begun, you can either end the test or cancel the test to make further edits:
  
  • If HubSpot is still processing your email, or the test has begun but the winner hasn't been determined yet, you can click End A/B test in the upper right of the email editor to cancel the A/B test. HubSpot will not pick a winning variant, and emails of either variation will not be sent to additional recipients.
  • If you scheduled your email for a future date, you can click Cancel scheduled email in the upper right of the editor to cancel the A/B test and make further edits to your email. To cancel a scheduled A/B test without making additional edits, you can click Cancel scheduled email instead.

Send your A/B test email and gather results

After your email is sent, wait enough time to gather results. You can analyze the performance of your email variations in the email details:

• In your HubSpot account, navigate to Marketing > Marketing Email. 
• Click the name of your A/B email. The results of the A/B test will be summarized on the Performance page. The winning email version and metric will be highlighted in green.

• In the upper left, click the Email version dropdown menu to analyze the email results for your email overall, or for either of the variations.

• When analyzing the overall results or the performance of either variant, you can also click See details at the top of the page to review the original A/B test configuration.

Please note: the Test distribution percentage in the Email details panel refers to the combined sample split of both versions. For example, if you configured your test to send Variation A to 10% and Variation B to 10%, then the Test distribution percentage would be 20%.