Run A/B tests for marketing emails

An A/B test measures engagement for different versions of the same email with a sample of your recipients. The best performing version is then sent to the rest of the recipients. 

This article covers how to create an A/B test with a regular marketing email. Learn more about creating A/B tested automated emails using workflows.

Understand limitations & considerations

• Tests split on a different ratio than 50/50 require at least 1000 recipients, which excludes contacts that previously hard bounced or unsubscribed. If fewer than 1000 recipients are selected, only version A of the email will be sent. 
• If you're creating a custom report, you can only filter for the Email content ID of variation A of an A/B test email. 

Create an A/B test email

1. In your HubSpot account, navigate to Marketing > Email.

2. Click the name of a drafted email, or click Create email. 

3. In the top right of the editor, click A/B test.

4. In the right panel, enter names for versions A and B. It's recommended to use names that help you differentiate between versions. 
5. In the How many recipients should receive the A/B test field, enter a percentage of recipients to enroll in the A/B test. All other recipients will receive the winning version. 
6. Click the Winning metric dropdown menu and select a metric for how the winning version will be determined. You can choose from Open rate, Click rate, and Click through rate.
7. In the Test length field, enter a number of hours over which the test will take place. After this time, the remaining recipients will be sent the winning version. 
8. Click the Fallback version dropdown menu and select a version that will be sent if the results are inconclusive. 
9. Click Create test. 
10. Edit both versions of your email. To switch versions, click the dropdown menu in the top right and select a version. 

11. While editing your variations, you can consider varying one of the following aspects:

    • Offers: experiment with the medium of the offer, such as an eBook versus a whitepaper.
    • Copy: experiment with the formatting and style of the content. You could test plain paragraphs versus bullet points, or a longer block of text vs a shorter block of text.
    • Email sender: try sending the email from an employee's email address instead of a generic department address.
    • Image: try out different images to see how the conversion rate is influenced.
    • Subject line: experiment with the length of the subject line or add personalization.
12. To change the parameters of the A/B test before sending the email: 
    • Click the dropdown menu in the top right and select Manage test. You must be viewing version A to select this option. 
    • In the dialog box, make any changes, then click Update. 
13. To cancel the A/B test: 
    • Click the dropdown menu in the top right and select Manage test. You must be viewing version A to select this option. 
    • In the right panel, click Delete test. 
    • Type delete, then click Delete. This will permanently delete version B of the email. 
14. In the top right, click Review and send. 

Cancel an A/B test

To cancel an A/B test before the test is complete:

1. In your HubSpot account, navigate to Marketing > Email.
2. Click the name of the A/B email you want to cancel.
3. Based on whether the test has already begun, you can either end the test or cancel the test to make further edits:
   
   • If HubSpot is still processing your email, or the test has begun but the winner hasn't been determined yet, you can click End A/B test in the upper right of the email editor to cancel the A/B test. HubSpot will not pick a winning variant, and emails of either variation will not be sent to additional recipients.
   • If you scheduled your email for a future date, you can click Cancel scheduled email in the upper right of the editor to cancel the A/B test and make further edits to your email.

Analyze A/B test performance

After your email is sent, wait enough time to gather results. You can analyze the performance of your email variations in the email details:

1. In your HubSpot account, navigate to Marketing > Email. 
2. Click the name of your A/B email. The results of the A/B test will be summarized on the Performance page. The winning email version and metric will be highlighted in green.
3. In the upper left, click the Email version dropdown menu to analyze the email results for your email overall, or for either of the variations.