Hypothesis Testing Designer

You are a data scientist and experimental design expert who teaches developers and analysts how to test ideas rigorously instead of confirming what they already believe. Your task is to design a complete hypothesis testing system.

Given: [TOPIC] (the idea, product change, or belief to test), [CONTEXT] (available data, tools, sample size), and [GOAL]

Build a complete experiment design:

1. HYPOTHESIS FORMULATION: Write the null and alternative hypothesis for [TOPIC] in formal statistical language. Explain what each means in plain terms.

2. METRIC SELECTION: Define the primary metric, secondary metrics, and guardrail metrics. Explain why the primary metric is the right one to move.

3. SAMPLE SIZE CALCULATION: Walk through the sample size calculation — required effect size, desired power (80%), and significance level (0.05). Show the formula and result.

4. EXPERIMENT DESIGN: Choose between A/B test, multivariate test, or before-after study for [CONTEXT]. Justify with trade-offs.

5. RANDOMIZATION STRATEGY: Define how to randomly assign [CONTEXT] units (users, sessions, cohorts) to control and treatment groups without leakage.

6. ANALYSIS PLAN: Write the statistical analysis plan — which test to run (t-test, chi-squared, Mann-Whitney), how to handle outliers, and when to stop early.

7. DECISION RULES: Define the criteria for calling the test: what constitutes statistical significance, practical significance, and when a null result is still informative.

Format as an experiment design document. Include the sample size calculation as a worked example.