Step Gain Optimization

In a normal A/B test, we examine “statistical significance,” which is normally set to 95%. In essence, what this means is that 95% of the results were above the control or that there is a 95% chance that one variation was, in fact, better than the other. These simplified explanations of the p values in a chi-squared test work for our purposes.

Instead of doing that, we will be looking at the chance that our small test, when scaled, will meet our threshold requirements. For example, let’s say we want to improve an email series with a conversion rate of 20% to 30%. The step gain option would be to test many different variants and find one test that can immediately “step” to the desired conversion rate.

One of the most efficient ways to test these variants is with small groupings of 100 users. This audience would be large enough to produce the guiding results we seek while still small enough to have minimal effects on the business if they fail. Returning to our example of developing a system that achieves a 30% conversion rate, we can run several tests and examine the chances of that test producing the desired result below.

Here, we see several different tests, out of 100 entries each, and the number of conversions they produced. We also see the likelihood that these tests were of systems with a 30% conversion rate or better and only accidentally had this low conversion rate.

As you can intuitively see, a test with 40 out of 100 conversions is very likely to be of a system that averages a 30% conversion rate, and a test with 15 is very unlikely.

Looking at a visual representation, we see that if our test produced 30 conversions, the variation when scaled, could go in either direction, making us uncertain if it will succeed. However, if we observed 45 conversions, even if there was significant variation, it would still surpass our 30% threshold. Thus increasing the confidence.

At the most basic level, that is enough information to go off of. If the strategy for the organization is risk aversion or the ability to scale is incredibly resource intensive, then we already have the answer. Run many tests until you find one that is at least 95% likely to meet the desired goal.

However, from a pure efficiency viewpoint, this is not the optimal way to run this system. Ideally, what you would do is identify any test that, let’s say, has at least a 1 in 3 chance of succeeding, so in our case, meets 27 or 28 conversions, and then run a scaled test to determine if that measure is accurate so we can make that step gain as fast as possible.

With this methodology, you can easily understand the expected value of a new system to better determine whether or not implementation is worth it for your organization.

With this additional piece of information giving us the range of conversion rates that a specific test could fall into, we can utilize our pre-test modeling strategies as an additional support mechanism for making our decisions. Perhaps even at the lower end of the bound, an organization would still see enough value to make a scaled test worth running.