Bayesian A/B Testing Framework

Traditional Frequentist A/B testing relies on p-values, which are often misunderstood and can lead to "peeking" errors. Bayesian A/B testing offers a more intuitive approach: it tells you the probability that one version is better than the other.

Why Bayesian?

• Intuitive Results: "Variant B has a 95% chance of being better than Control."
• Faster Decisions: You can often stop tests earlier if the probability of winning is high enough.
• Expected Loss: Understand the risk of choosing the wrong variant.

The Framework

We built a lightweight Python framework to simulate and analyze A/B tests using Bayesian inference with Beta distributions.

Features

• Beta-Bernoulli Model: Perfect for conversion rates (Click vs. No Click).
• Posterior Sampling: Visualize the distribution of possible conversion rates.
• Uplift Calculation: Estimate the expected improvement.

Visualization

The framework plots the posterior distributions of the Control and Variant groups. The overlap indicates the uncertainty. As we gather more data, the curves narrow, giving us more confidence.

Open Source Code

We have open-sourced this framework to help teams adopt better experimentation practices.

[View Code on GitHub](https://github.com/your-username/marketing-science-ab-testing)

Usage

```python

from src.main import BayesianABTest

ab = BayesianABTest()

ab.update('Control', conversions=100, totals=1000)

ab.update('Variant', conversions=125, totals=1000)

ab.sample_posterior()

```