Skip to main content

​
Experiments and metrics

Superflag connects a feature decision to an immutable experiment iteration and versioned metric definition. Assignment uses the same deterministic evaluator as ordinary flags; there is no second bucketing system.

​
Current model

  • an experiment references one flag and immutable metric revisions;
  • starting creates an immutable iteration snapshot;
  • assignment is deterministic for the configured randomization unit;
  • safe allocation ramps preserve existing bucket boundaries;
  • a new iteration uses a new salt and population;
  • lifecycle, ramp, revision, and configuration updates require version preconditions, while creation uses uniqueness and idempotency; and
  • optional SDK telemetry records exposure and configured outcomes asynchronously.
Lifecycle is explicit:
Archived is terminal.

​
Metric revisions

Metric definitions support conversion, count, sum, mean, and derived ratio kinds. Each definition declares an owner, stable event key, visibility, desired direction, randomization unit, and bounded attribution window. Running iterations pin revisions instead of following mutable names.

​
Manage from the CLI

Use metrics validate|plan|apply|recover and experiments validate|plan|apply|recover for agent-authored or automated changes. Plans bind immutable target identity, current revisions, the full resolved definition, and an integrity hash.

​
What the result means

Superflag’s feature-decision data answers which variation was assigned, whether it was actually exposed, which configured outcome followed, and whether the data is complete enough to inspect. It is not a general event, profile, funnel, session replay, or arbitrary SQL product. The first inference engine is a fixed-horizon contract for binary conversion and numeric means. It exposes sufficient statistics, effect intervals, sample size, analysis window, freshness, readiness, and health reasons including sample-ratio mismatch, crossover, missing attribution, late data, stale watermarks, and low power. It uses family-wise multiple-comparison correction and publishes its engine version with every result. Winner selection remains deliberately disabled pending formal statistical review. Invalid or incomplete analysis cannot trigger an action. Inspect the authoritative result directly:
Feature insights are bounded rather than general analytics: allow-listed feature dimensions and measures, a 90-day aggregate window, seven-day redacted event pages, disclosed approximate-unique error, freshness, query cost, and opaque cursor checkpoints. Cells below 20 approximate subjects are suppressed. There is no arbitrary SQL, user profile, funnel, cohort, or raw targeting-context explorer.

Feature telemetry

Understand decisions, exposures, outcomes, privacy, and delivery behavior.