Not everything needs to be an A/B test
Generating results is one of the main responsibilities of product professionals. The focus is always on improving the product and enhancing success metrics, but the paths to achieving these results can vary. While A/B testing is a valuable tool for learning, it's equally important to make strategic decisions that directly impact the business—and these decisions don’t always need to be based on A/B testing.
This article invites you to reflect on when A/B testing is truly necessary and when other approaches might be more effective.
Experimentation vs. Maturity
Companies and teams with low experimentation maturity tend to apply A/B tests to everything. This often fails to solve real problems and wastes time, money, and resources. It’s common for less mature companies to rely on A/B testing as the main—or even only—learning tool. This can create the false impression that everything can be tested and that meaningful insights or continuous innovation will always follow. In reality, many of these tests end up being merely “disguised releases” or ineffective experiments.
Companies face a variety of challenges, and each situation is unique. For example, if a company is doing a full product redesign, A/B testing may not be the best option. Similarly, when introducing a new mandatory component on the homepage, A/B testing might not be ideal. However, later on, testing which content or call to action within this new component converts better compared to the original version could be a great way to understand user perception and behavior. To do this, it's essential to have historical data, understand what comparisons to make, and have clear success metrics.
Release or Run an A/B Test?
At this point, it's crucial to assess certain elements that may guide your decision-making process.
Uncertainty in the delivery
Evaluate whether the delivery carries high uncertainty. One of the great advantages of experimentation is mitigating risks and efforts on initiatives that might not achieve the expected results.
Well-defined hypothesis
Check if there’s a well-defined hypothesis that can be tested and validated through A/B testing. If the hypothesis isn't clearly formulated, it may be necessary to revisit the method or explore the topic further to create a hypothesis that outlines what will be done and what you expect to validate.
Necessity of the delivery
Review the priority of the delivery. Ensure that it aligns with the product strategy and key metrics. Important note: not every delivery without a clear hypothesis will be dismissed, as there are structural needs that enable other products and journeys. These should be or are often connected to the broader product strategy.
Results
One of the main advantages of A/B testing is its ability to produce comparative results, allowing you to understand how one version performs relative to another. These results enable more informed decisions, such as whether to expand the new version to a larger user base, adjust the current iteration, or recognize that the test did not meet expectations. Sometimes, results may be inconclusive, in which case reviewing or adjusting the test format is recommended.
Decision-Making Flowchart
Flowchart scenarios
Make a release
In this case, the feature or enhancement in question does not involve significant risks, and the results will not be used for comparisons or future decision-making. The objective is clear, and the demand is important. Making a release does not mean that the feature should not be monitored; rather, the purpose and comparison of an A/B test do not apply here. Therefore, it is recommended to proceed with the release.
Point of Attention: It’s crucial to understand which release strategy is most suitable for the current stage of the product and the company. Generally, releases occur gradually, following criteria such as users receiving the new version of the app, regional expansions, or segmentation by user groups, among others.
Review the method or reformulate the hypothesis
In this scenario, the demand may have a degree of uncertainty, but there is no clear hypothesis to test, nor is there an evident need for this delivery to be made. At this stage, it is essential to reflect on the phase of discovery the product is in and identify which questions still need to be answered. It may be necessary to explore other learning methods that do not involve an A/B test. Alternatively, the demand may indeed be important and risky, but the hypothesis is not well formulated, complicating the decision to make a release or conduct an A/B test. Therefore, a review and reflection are warranted.
Run an A/B Test
In this scenario, the delivery has a degree of uncertainty, and the hypothesis is well formulated. The objectives and expected metrics are clear, and the collected results will impact the next steps and decisions regarding the product.
Additionally, there are basic requirements for executing and analyzing the test:
Technological Capacity: Adequate infrastructure to efficiently conduct tests, including tools for version segmentation and result analysis.
Legal Conditions: Compliance issues or legal requirements do not impede the execution of the test.
Testing Duration: Sufficient time for the results to be valid and statistically relevant.
User Base: There are enough users to validate the test results. This scenario may differ for newly launched products with low traffic, which can be a barrier.
Conclusion
A/B testing is a valuable way to generate insights and learnings for the product. However, like any tool, it does not solve all problems. The most important aspect is ensuring that the right problem is being addressed with the most appropriate tool to achieve the expected results.
Sources