The Two Schools of Thought

When running A/B tests, you have two main statistical frameworks to choose from: frequentist (the traditional approach) and Bayesian (increasingly popular in industry). Both have merits, and understanding when to use each can improve your testing program.

Frequentist A/B Testing: The Classical Approach

How It Works

Frequentist testing asks: "If there were no real difference (null hypothesis), how often would we see results this extreme?"

Key Concepts:

• P-value: Probability of observing results this extreme if null is true
• Statistical Significance: Typically p < 0.05 (5% threshold)
• Confidence Interval: Range where true effect likely falls
• Power: Probability of detecting a real effect (typically 80%)

Sample Calculation

For a conversion rate test (5% baseline, 10% minimum detectable effect):

• Control: 5.0% conversion rate
• Variant: 5.5% conversion rate (10% relative lift)
• Required sample: ~31,000 per group for 80% power

Strengths

1. Well-established: Decades of theoretical foundation
2. Simple decision rule: p < 0.05 = significant
3. Easy to explain: "95% confident the effect is real"
4. Pre-registration: Clear upfront commitment to sample size

Limitations

1. Binary output: Significant or not—no probability of improvement
2. Fixed sample size: Can't peek at results without inflation
3. No prior information: Treats each test as if we know nothing
4. Misinterpretation: P-values are widely misunderstood

Bayesian A/B Testing: The Probabilistic Approach

How It Works

Bayesian testing asks: "Given the data we observed, what's the probability that variant B is better than A?"

Key Concepts:

• Prior: What we believe before seeing data
• Likelihood: How well each hypothesis explains the data
• Posterior: Updated belief after seeing data
• Probability of Being Best: Direct answer to "which is better?"

Sample Output

Instead of p-values, Bayesian analysis might report:

• Probability B beats A: 94%
• Expected lift: +8% (credible interval: +2% to +14%)
• Risk of choosing B if wrong: 0.3% revenue loss

Strengths

1. Intuitive output: "94% probability B is better"
2. Continuous monitoring: Update beliefs as data arrives
3. Decision-focused: Directly answers business questions
4. Incorporates prior knowledge: Use historical data wisely
5. Risk quantification: Understand downside of wrong decisions

Limitations

1. Prior selection: Subjective choice that affects results
2. Computational complexity: More sophisticated calculations
3. Less familiar: Requires education for stakeholders
4. No universal stopping rule: Flexibility can become lack of discipline

Practical Comparison

| Aspect | Frequentist | Bayesian |

|--------|-------------|----------|

| Question Answered | "Is this statistically significant?" | "What's the probability B is better?" |

| Output | P-value, confidence interval | Probability of improvement, credible interval |

| Early Stopping | Invalid without correction | Valid with caveats |

| Prior Information | Not used | Incorporated |

| Interpretation | Requires training | More intuitive |

| Implementation | Simpler | More complex |

When to Use Frequentist Testing

Best for:

1. Regulatory contexts: When you need defensible, standard methods
2. Large organizations: Where consistent methodology matters
3. High-stakes decisions: Where statistical rigor is scrutinized
4. Simple tests: Where sophistication isn't needed

Example Scenario:

You're running a pricing test for a public company. The board will review results. A traditional frequentist test with pre-registered sample size provides defensible, auditable results.

When to Use Bayesian Testing

Best for:

1. Rapid iteration: When you're running many tests quickly
2. Business decisions: When you need probability of improvement
3. Limited traffic: When samples are small
4. Sequential testing: When you want to monitor continuously
5. Mature testing programs: When you have historical priors

Example Scenario:

You're testing ad creative on Meta, running multiple variants per week. You want to quickly identify winners and reallocate budget. Bayesian testing lets you make probability-based decisions without waiting for fixed sample sizes.

Practical Implementation

Frequentist Setup

Most A/B testing tools (Google Optimize, Optimizely, VWO) default to frequentist:

1. Define hypothesis and minimum detectable effect
2. Calculate required sample size
3. Run test to completion
4. Report results with p-value and confidence interval

Bayesian Setup

Some tools support Bayesian (Dynamic Yield, some Optimizely features), or you can calculate:

1. Define prior belief (skeptical, informed, or non-informative)
2. Collect data and update posterior
3. Report probability of improvement and expected lift
4. Make decision based on probability threshold (e.g., 95%)

Python Example (Simplified):

```python

import pymc as pm

import numpy as np

Observed data

control_conversions = 120

control_visitors = 2400

variant_conversions = 145

variant_visitors = 2400

Bayesian model

with pm.Model():

Priors (weakly informative)

p_control = pm.Beta('p_control', alpha=1, beta=1)

p_variant = pm.Beta('p_variant', alpha=1, beta=1)

Likelihoods

pm.Binomial('control', n=control_visitors, p=p_control, observed=control_conversions)

pm.Binomial('variant', n=variant_visitors, p=p_variant, observed=variant_conversions)

Probability variant is better

pm.Deterministic('prob_variant_better', p_variant > p_control)

Sample posterior

trace = pm.sample(2000)

Result: Probability variant beats control

prob_better = (trace.posterior['prob_variant_better'].values.mean())

print(f"Probability variant is better: {prob_better:.1%}")

```

The Hybrid Approach

Many practitioners use elements of both:

1. Pre-register sample size and test duration (frequentist discipline)
2. Monitor continuously with Bayesian probability updates
3. Don't stop early unless probability is overwhelming (>99%)
4. Report both p-values and probabilities for different audiences

Common Mistakes to Avoid

With Frequentist Testing

• Peeking without correction: Inflates false positive rate
• Stopping at significance: Wait for planned sample size
• Ignoring practical significance: A 0.1% lift might be significant but useless
• Multiple comparisons: Testing many variants without adjustment

With Bayesian Testing

• Bad priors: Using overly strong priors that dominate data
• Premature stopping: Making decisions with very wide credible intervals
• Overconfidence: Treating 90% probability as certainty
• Complexity theater: Using Bayesian methods for simplicity's sake

Recommendations for Marketing Experiments

For Paid Media Creative Testing

• Use Bayesian: Quick iteration, probability-based budget allocation
• Set decision threshold: 90-95% probability to declare winner
• Accept uncertainty: Some tests won't have clear winners

For Website Conversion Testing

• Either works: Choose based on organizational preference
• Pre-register: Commit to sample size regardless of method
• Consider business impact: Use expected value calculations

For Pricing/High-Stakes Tests

• Use Frequentist: More defensible for major decisions
• Increase sample size: Aim for 95% confidence and 90% power
• Get statistical review: Have methodology vetted

Conclusion

Both frequentist and Bayesian approaches are valid tools for A/B testing. The choice depends on your context, organizational preference, and decision-making needs.

What matters most is running well-designed experiments—proper randomization, sufficient sample sizes, and clear hypotheses. The statistical framework is secondary to experimental rigor.

Start with whichever approach your tools and team support. As your testing program matures, you can experiment with alternatives and find what works best for your decision-making process.