Experiments

An Experiment is an optional, pre-registered measurement wrapper around Change variants. It exists to stop early, noisy results from quietly becoming confident claims — whether the reader is a human or an agent.

Pre-registration and the stats plan

You can ship a Change without an Experiment. Torno calls that ship-and-watch and labels it a weaker evidence class rather than mislabeling it an Experiment. When you want causal evidence, you register a plan before launch. The plan pins:

• the Change and its variants, and the assignment mode — Torno’s own Subject bucketing, or externally assigned arms you declare;
• the target Metric, pinned to a specific version, plus guardrail Metrics;
• the audience Cohort, the layer, and the traffic share;
• the minimum detectable effect, planned and minimum duration, sample floor, and novelty window;
• the auto-pause policy.

torno experiment plan estimates duration from current traffic and warns when a plan is underpowered — before launch, while the warning is still cheap. Launch runs through policy and validates readiness, exposure and layer placement, the Metric version, and power. Lifecycle controls cover create, launch, pause, resume, ramp, and stop.

Layers and assignment

Layers are mutually exclusive traffic groups. Two Experiments in the same layer never share a Subject, so they cannot contaminate each other’s results. Cross-layer overlap is tracked in an exposure registry, interaction findings must be acknowledged, and two Experiments colliding on the same Metric can trigger a human tiebreak.

Assignment is deterministic and sticky: the same Subject always resolves to the same variant, evaluated locally from a signed assignment bundle rather than by a per-request server call. An assignment is recorded exactly once, conflicting records are flagged as integrity problems rather than averaged away, and a running Experiment interlocks arbitrary rollout changes to its Change. See Changes and SDKs for delivery mechanics.

Readouts and the evidence vocabulary

The engine uses sequential Bayesian inference, so you can look at a readout at any time without invalidating it. What you will not see is a naked p-value or an early “winner.” Every readout carries one explicit evidence state from a closed set:

Readouts also carry lift bounds or posterior intervals, probability and expected-loss context, runtime and sample constraints, and the reason behind any integrity flag. Revenue targets add a bounded contribution block. Agents reading through the API, CLI, or MCP get this full schema, not a simplified verdict.

Integrity checks and exclusions

• Sample-ratio mismatch detection flags arm traffic that drifts from the planned split.
• Assignment conflicts are surfaced, not silently resolved.
• The novelty window keeps first reactions from being read as lasting effects.
• Events from bot-quarantined Subjects are excluded, and retroactive quarantine revokes their arm membership.
• Every per-arm read is k-gated through Cohorts: arms below the Workspace’s k-anonymity minimum are suppressed, not reported. See Identity and privacy.

Guardrails and auto-pause

Guardrail Metrics carry declared harm directions. The engine monitors them while the Experiment runs and, under the pre-registered auto-pause policy, can pause a breaching Experiment on its own. A guardrail_breach readout states which guardrail moved and why the Experiment stopped.

Program holdout

An optional Property-level holdout keeps a slice of traffic outside the entire growth program, so you can measure whether the program as a whole moves anything — a question most growth tooling prefers not to ask.

From readout to Decision

A finished readout becomes a Result on the Loop. A Result is not a Decision. Someone — human or agent, within policy — must record ship, kill, iterate, or extend, with the evidence attached, and the Decision log is append-only. Torno publishes no performance numbers for the stats engine; the claim is narrower and more durable: a readout never tells you more than the data supports.