You must calculate the results obtained for the previously selected research metric for both traffic segments. The difference between the obtained values will be the answer to the questions posed at the beginning of the testing. Its absence is the result of the meaninglessness of innovations, even if it is present when considering individual sections of the study. For some parts of the testing, the version being studied may show better or, conversely, worse results.
The success of the analysis is not as important as the experience you gain from the research. Even negative or neutral results can be useful for future A/B tests. Update the hypothesis, defendable positions, change the central areas of study. There are many ways to use the data obtained.
Analysis and evaluation of A/B testing results
After completing the A/B test engineer database within the expected timeframe, you receive bare statistics. Let's start working on it: we separate the indicators that carry high significance and those caused by a number of accidents. We enter a new stage: assessing the significance of the final results.
Randomness is revealed through A/B testing hypotheses: null and alternative. In the first case, the discrepancies will be invisible or barely noticeable. In the second: significant.
To test a hypothesis, special statistical tests are used. Their choice directly depends on the control metric. The most commonly used is the Student's t-test. It can be used to analyze many quantitative indicators. The Student's t-test is ideal for working with data of not very large volumes.
To make a final decision on the positive results of A/B testing, the significance level should be in the range of 90 to 99%. If this indicator is on a low tier, it is impossible to draw conclusions about the effectiveness and, accordingly, about the results of the study of the changes made.
Although this characteristic is very important, many developers completely forget about the significance level when publishing the results of A/B testing. In fact, we have a situation in which 80% of all studies are statistically insignificant.
A large volume of tested traffic in parts of the audience makes the difference in the average daily results of the significance level smaller, smoothing it out. Small traffic due to the results of random variables gives low accuracy, which leads to an increase in the time of conducting the study to obtain high-quality results.
Once the results have been tested for statistical significance and random errors have been excluded, you must make a final decision on which variant to implement.
After completing A/B testing, after analyzing its results, when changes have been made to the current product, you need to think about future prospects. Start calculating weak or pain points, set other goals, formulate new hypotheses. All this will certainly lead to more efficient functioning of the project and improve its quality.
When thinking about further steps to optimize your product's sales funnel, it's worth remembering the methods of unit economics. It will help determine the degree of profitability of a project or model based on the total revenue from one client or from one specific product.
Possible Mistakes When Conducti