Statistics & methods
The current production A/B surface reports observed analytics. It does not expose a configurable statistical testing method.
What the results report
For each complete-video variant, the results can show:
- Sessions and conversions
- The observed rate for the selected goal
- Lift versus the control when it can be calculated
- Traffic share and average retention
- An observed leader
A test with no player events shows No data yet rather than synthesized zeroes.
Controls that are not available
The current UI has no:
- Bayesian/Frequentist selector
- Sequential-testing option
- Alpha, power, or minimum-sample inputs
- Sample-size calculator
- Significance gauge, p-value, or confidence interval
- Confidence-over-time chart
- Confidence threshold for promotion
Do not interpret the Leader label as a claim of statistical significance. It identifies the highest observed goal rate among the available results.
How promotion works
Manual promotion is an operator decision. Optional automatic promotion is a per-test schedule: after 1–336 hours, the current goal-rate leader is promoted. It is not gated by a documented confidence threshold or minimum sample size.
Best practices
- Compare sessions and rates alongside lift; small samples can move sharply.
- Keep live traffic changes in mind because editing allocation does not reset observations.
- Use an external statistical review if your decision requires a formal confidence or power calculation.
Related docs
Frequently asked questions
Which statistical engine does the UI use?
The current UI does not expose or identify a Bayesian/Frequentist engine. It reports observed sessions, rates, lift, retention, traffic share, and leader.
Does Leader mean statistically significant?
No. Leader reflects the strongest observed goal rate; the product does not expose confidence, significance, or p-values.
Is there a sample-size calculator?
No. Sample-size, alpha, power, and minimum-sample controls are not available in the current production UI.