Note 05Proving groundsvalidation · 3 min read
Validation
Hypothesis-first validation: every empirical claim stated with its falsifier and bound to numbered experiments E1 through E7, completed or scheduled.
The validation program is organized hypothesis-first: each load-bearing claim is stated at full strength with its falsifier, and bound to numbered experiments, completed or scheduled.
Results below follow the two evidence modes and pre-registration discipline of methods: mock mode validates machinery, real mode measures agents, never conflated.
The hypotheses
H1: Mint integrity. Completion read from environment-captured state mints no fiction: false claims settle at rate zero, while any self-report channel settles them at approximately the false-claim rate. Falsifier: the environment arm settling any incomplete unit. Experiments: E1, E2 (mock: complete, held) · E4 (real: in progress).
H2: Ledger identification. On real agents, cost per quirq falls with unit index toward a stable floor in persistent environments and stays at cold-start in fresh contexts, holding model, prompt, and task family fixed. Falsifier: flat or rising persistent-arm trajectories, or equal decay in fresh contexts. Experiments: E3 (mock consistency: complete, held) · E4 (real: in progress).
H3: Predictive validity. Audit-corrected QER trend predicts real business outcomes (renewal, expansion, P&L attribution) better than token spend, task counts, or benchmark scores. Falsifier: QER* trend failing to outperform those baselines in instrumented deployments. Experiments: E5, E6 (scheduled).
H4: Budget governance. Under payer-set budgets, benchmark anchoring, and sampled value audits, budget inflation stays within audit tolerance under sustained optimization pressure. Falsifier: systematic budget drift in long-running ledgers despite the mitigations. Experiments: E7 (scheduled).
Completed experiments (E1 to E3, mock mode)
E1: verification source (H1). 200 identical units, an agent that falsely claims done with probability 0.05.
| Arm | False claims | Silently settled | Rate |
|---|---|---|---|
| Self-report | 7 | 7 | 3.5% |
| Environment snapshot | 7 | 0 | 0.0% |
E2: hardening (H1). 50 units, an agent that guts the test instead of fixing the code.
| Arm | Checks green | Intent actually met |
|---|---|---|
| Gameable definition of done | 100% | 0% |
| Plus one verification-surface check | 0% (all caught) | 0% |
One hash-equality check on the test file converts total silent failure into total detection.
E3: tenure (H2). Twelve similar units, persistent vs fresh environment: cost decays 85.6% to the execution floor (~800 tokens) by unit six in the persistent arm; the fresh arm stays flat at ~5,540. Mock-mode caveat in full: the scripted agent implements the cost model, so this arm demonstrates harness consistency, not agent behavior.
The roadmap (E4 to E7)
- E4 (H1, H2): real-mode replication of E1 to E3 with a production coding agent and measured tokens. The false-claim rate becomes a measurement, and the tenure curve becomes evidence. In progress.
- E5 (H3): pilot ledgers: the dashboard instrumented on real work across at least three unit types.
- E6 (H3): predictive study: QER* trend vs token spend, task counts, and benchmark scores as predictors of renewal, expansion, and P&L attribution.
- E7 (H4): longitudinal budget-drift audit under the gaming mitigations.
Results and per-run data are published here as they land.
The claims are on the record, and every one is tiered against its evidence in claims and limitations.
Adapted from the XO research program · canonical version at docs.xo.builders
