Email

Why were my A/B email test results inconclusive?

Last updated: November 9, 2016

Available For:

Marketing: Pro, Enterprise
Sales: N/A

HubSpot performs a statistical test on your A/B test email. If neither email is statistically better, HubSpot will then send the version of the email chosen when creating the A/B test to the rest of the list. 

It's important you give your A/B test a decent amount of time to gather results, so the HubSpot email tool can have enough time to gather statistics and take actions on sending the winning variation. This will default to 1 hour when you create a new A/B test but it's recommended to extend this in order to allow enough time to gather helpful data.

The validity of the statistical test is based on on three factors:

  • The overall difference in rates (opened, clicked, etc.)
  • The overall total sample size.
  • The rates themselves need to be high enough to be significant.

To select a variation to send if your A/B test results are inconclusive, navigate to the A/B tab in your email editor > choose the preferred variation under If the results are inconclusive, send option.

User-added image

For more information on using A/B testing, check out this user guide article. To learn more about how to determine your test sample size and time frame in order to achieve statistical significance, take a look at this HubSpot blog post