An A/B test measures engagement for different versions of the same email with a sample of your recipients. After creating your marketing email, you'll create a version B to test how a single variation impacts email opens or clicks. Depending on how each version of your email performs over a period of time, the tool can pick a winning version to send to the rest of your recipients list.
Guidelines for effective A/B testing
- Decide what you want to test - As you optimize your email, there are a number of variables you can test. You don’t have to limit yourself to testing only images or text size. Look at the various elements on your marketing resources and their possible alternatives for design, wording, layout. In fact, some of the areas you can test might not be instantly recognizable. For instance, you can test different subject lines, senders, or ways to personalize the message.
- Test one variable at a time - In order to evaluate how effective an element on your page is, you have to isolate that variable in your A/B test. Only test one element at a time. Remember that even a simple change, say, switching the image in your email, can drive big improvements. In fact, these sorts of changes are usually easier to measure than the bigger ones.
- Set up control and treatment - When conducting an A/B test, set up your unaltered version as your “control,” the email you would normally use. From there, build a variations, or “treatments,” the email you’ll test against your control. For example, if you are wondering whether including a testimonial in an email would make a difference, set up your control email with no testimonials. Then create your variation with the testimonial.
- Split your sample group randomly - In order to achieve conclusive results, you need to test with two or more audiences that are equal. With HubSpot, we automatically split traffic to your variations so that each variation gets a random sampling of visitors. You can decide the size of the sample group using the slider in the A/B test pane.
- Decide which metrics to test - Before you launch your test, think about how significant your results should be in order for you to decide that the change should be made to your website or email campaign. You can choose a winning email version based on opens by delivered, clicks by delivered, and clicks by opens.
Please note: You won't be able to create an A/B test with an email that is saved for automation for your workflows. Make sure your email is set to send at a specific date and time or immediately (under the Sending tab in your email).
Set up an A/B test in HubSpot
Create your A/B test emails
- In your HubSpot account, navigate to Marketing > Email.
- Click Create email to set up a new email. This will be version A of your A/B test.
- After you've created your email, click the A/B tab on the left, then click Create A/B Test.
- In the dialogue box that appears, give your version B email a name, then click OK. By default, HubSpot will populate this with the name of your existing email with (Variation) at the end.
Now that you've created both versions of your A/B test email, you can switch between the two by clicking on A or B at the top left-hand side of the editor.
While editing your variations, here are some things to consider for testing:
- Offers - Experiment with the medium of the offer. You might test an eBook versus a whitepaper or video.
- Copy - Experiment with the formatting and style of the content. You could test plain paragraphs versus bullet points or a longer block of text vs a shorter block of text.
- Email Sender - Try sending the email from an employee's email address instead of a generic department address.
- Image - Try out different images to see how the conversion rate is influenced.
- Subject Line - Play around with the length of the subject line or add personalization.
- Whole Email - The fastest way to achieve drastic results and produce a landing page that drives a lot of conversions is to test the entire email. Make iterations to the whole email that affect image placement, subject line, and its copy. Once you have a statistically significant result pointing to the variation that performed better, you can continue optimizing through smaller tweaks.
Choose the criteria for your A/B test
Once you've finished editing your A and B email variations, you'll configure your testing options by clicking on the same A/B icon on the left.
First, you'll need to decide on the size of your test group. You can move the slider to decide what percentage of contacts will receive Variation A and Variation B.
If you'd like, you can also set the slider to send Variation A to 50% and Variation B to 50%. This will send one email variation to half of the contacts on the list, and the other email to the other half of contacts. This lets you analyze the statistics of the two after the send to see which was more successful.
Please note: you can do a 50/50 test using any sample size. All other sample splits require a list that includes at least 1,000 recipients, excluding any contacts that previously hard bounced or unsubscribed.
Choose the winning metric for your test
If you've opted to send variations A and B to a smaller group and then send the winning version to the rest, next you need to select the metric that will decide how the winning variation will be chosen. You can choose the winner based on Opens by delivered, Clicks by delivered, or Clicks by opens.
Set up the test duration
Choose how long you want to gather results before sending the winning variation. Drag the test duration slider button to set how long to gather data before choosing a winner. Your A/B test can run for up to 100 hours.
Choose which version to send if your test results are inconclusive
If the results are inconclusive after the test duration has concluded, the version selected from this dropdown will decide which variation gets sent to the remainder of contacts.
Send your email and gather results
Once you've finished setting up the criteria for your A/B test, you're ready to send your emails. Click Send in the top-right corner of the email editor.
You can see the performance of your email variations by navigating to Marketing > Email and clicking on the name of your A/B email. The results of the A/B test will be summarized on the Performance screen. The winning email version and metric will be highlighted on your results in green.
You can use the AB Overview dropdown menu in the upper-left to analyze the email results for your total email send or for either of your email variations.