Back to explorer

We measure the tool, not the agent. Most agent benchmarks hold the task fixed and vary the model. This benchmark inverts that: one constant reference agent, the same task per category, five trials — so a difference in the numbers is attributable to the tool and how the agent was taught it, not the setup.

Agent Ergonomics — Measurement Methodology v0

Status: frozen · Version: v0 · the hash of this file is recorded in docs/methodology.lock and stamped into every profile's provenance.methodology. A profile whose hash does not match the current file was produced under a different methodology and is not comparable — regenerate it.

Why this document exists: a benchmark is only a benchmark if it is reproducible and its rules are disclosed before the numbers. This is the Artificial-Analysis discipline applied to Agent Experience: fix the harness, vary the subject, prefer ground-truth signals to judgement, keep axes separate, report variance.

1. What is measured

The subject is a tool/skill/MCP-service an agent uses to produce an artifact. The subject is the ONLY thing that varies between profiles. Everything else — the agent, the task difficulty, the trial count, the scoring — is held constant so a difference in the numbers is attributable to the tool, not the setup.

Output is the surfaces × lenses matrix (5 × 6 = 30 cells) plus roll-ups. See ax-model.md for the model. A cell is emitted only when it was actually measured; unmeasured cells are reported as gaps, never filled from nothing.

2. The reference agent (the constant)

  • Reference agent: the PI coding agent (@earendil-works/pi-coding-agent), model z-ai/glm-5.2 over OpenRouter, run non-interactively (pi --print --mode json). One agent scores every profile.
  • Rationale: it reports real token telemetry (input/output/cost per turn), which anchors the economy lens — the one lens with the widest real spread.
  • The reference agent is the constant. Measuring "does AX hold across different agents?" is a separate study (ax deep … --ensemble), whose cross-model runs never feed the scored matrix.

3. Subjects & provisioning contract

  • A subject key is <category>-<tool> (e.g. diagrams-d2). No oota: prefix.
  • Resolution order: provisioned skill (skills.sh / official, via npx skills add) → provisioned MCP server → the tool's own source.
  • Provisioning must succeed for the mode it claims. A subject that silently falls back to source while its peers ran from an installed skill is not comparable and must be flagged, not scored alongside them. (provisioned.json records the mode; profiles record skillMode.)

4. Trials

  • N = 5 trials per tier (--n overrides; never below 2 for any tier whose reading needs variance).
  • Per-trial isolation: each trial runs in a fresh copy of the grounded cwd. Trials never share mutable state, so run-to-run determinism is a real measurement rather than an artifact of one directory being mutated in place.
  • Every reading carries std across the N trials. Variance is reported, not hidden.

5. Grounded briefs

A brief = (category task-spec) × (a real seed entity supplying the subject matter). Same category → same seed by default (controlled comparison). The task-spec is checkable (see §7). Briefs are generated deterministically (no LLM), so the bar is identical for every tool in a category.

6. The probe battery (what each tier ACTUALLY does)

Each tier is a prompt sequence (± a cwd mutation), not the same one-shot intent relabelled. Prompts run over a persistent cwd (PI: sequential invocations, state persists on disk; ACP: one open session). intentTemplate is threaded; scenarios live in ax/probes/scenarios.ts.

Tier Interaction Executable when Cells (dropped if not executable)
T0 cold-call from memory, no docs, no tools always interface.prior_alignment
T1 guided one prompt, docs available always loop.verifiability/economy/determinism, interface.verifiability
T2 multi-turn base + 2 evolving follow-ups over persistent cwd always recursion.coherence (needs checker), recursion.economy (needs tokens)
T3 failure-injection a broken artifact is seeded; agent must recover it broken template exists for the tool loop.safety
T5 compaction context inflated + long iterative task only if real compaction occurs recursion.determinism — dropped when compactionObserved = 0
T6 refinement base + "push past acceptable" ×2 needs checker recursion.verifiability

Rule: if a tier cannot execute what it claims against the reference agent, its cells are deleted, not filled from a run that didn't do the thing. (Concretely: T5 determinism requires observed compaction; goal-based cells require a checker; token-based cells require token telemetry.)

7. The ground-truth hierarchy (checker → telemetry → judge → rater)

Signals are trusted in this order; a lower tier only fills what a higher tier can't:

  1. Programmatic checker (ax/instruments/checker.ts) — deterministic, no LLM. Given the category + the produced artifact, it computes which of the brief's checkable requirements are met. Its score (fraction met, 0..1) is goalProgress — the trajectory signal AND the anchor the rater is calibrated against (never the reverse). Checkers wired (the CHECKERS registry):

    • diagrams — d2 · mermaid · plantuml · dot · python-diagrams (renders + node/ edge/cluster/title counts); excalidraw (JSON: shapes, bound arrows, frames, title). Live-validated at N≥5.
    • data-visualization — a figure was produced, its source runs clean, and (when SVG) it carries labels + multiple series colours. Built; pending live validation.
    • documents — compiles to final format (tectonic/pdflatex/pandoc/typst) + 3+ sections + table + figure + citation. Built; pending live validation.
    • markup — renders + title + nested sections + table + code block + link + image. Built; pending live validation.
    • image-processing — WebP + JPEG outputs, cropped ~1.91:1, each <200KB (dims via sips/identify, size on disk). Built; pending live validation.

    A category is v0-validated once it has been run live at N≥5 with clean provisioning; until then its checker is active but the category's numbers are provisional. Whole-file hash of THIS doc pins the checker inventory, so adding a checker is a deliberate, versioned change.

  2. Telemetry — tokens, turns, success, compaction, run-to-run variance. Real, parsed from the agent's event stream. No estimation.

  3. Artifact judge (judge.ts) — a modality-aware vision panel that SEES the artifact, used where a programmatic checker doesn't exist. Secondary.

  4. Rater panel (rater.ts) — LLM judges fill cells with no hard instrument, from the real transcript/docs/telemetry. Independence: the panel MUST NOT include the reference model (a model grading its own work is not independent); the reference z-ai/glm-5.2 is excluded. The panel is handed the checker's verdict as ground truth and told to defer to it on verifiability/determinism.

A category with no checker returns hasChecker:false → its goal-based cells stay soft (judge/rater only), at reduced confidence, and are flagged.

8. Calibration & validity

  • Convergent validity: where a hard instrument and the independent rater score the same cell, the delta is recorded (profile.validation). Per-lens rater bias is measured and subtracted from that lens's soft cells (anchoring soft to hard).
  • Peer calibration: absolute −1..+1 scores read as "everything green" (scoring-audit.md). The published standing is percentile vs. the measured field, with low-variance ("dead") cells down-weighted. This sharpens as N subjects grows and requires like-vs-like peers to be trustworthy.
  • Construct validity: ax perturb degrades one dimension (e.g. strips docs) and checks the matrix moves in the right cells and stays put elsewhere.

9. Reproducibility & cost

  • Same subject + same methodology hash + same seed → same brief and same scoring procedure. Randomness in the agent is bounded by N and reported as std.
  • Transcripts + per-trial telemetry are persisted for re-analysis without re-running.
  • Cost controls: Screen (static + T0/T1) before Deep; cap N at 5 unless a comparison demands more.

10. Known limitations (honest)

  • Checkers exist only for diagrams so far. Every other category's goal-based cells are soft (judge/rater). Breadth without a checker multiplies rater noise — see §11.
  • Compaction (T5) is hard to force reproducibly with a one-shot reference agent; those cells are frequently (honestly) dropped.
  • Peer percentiles need N subjects and like-vs-like peers. At small N they are coarse. A field of weak tools makes a mediocre one look green; the checker + perturbation guards the absolute anchor.

11. Scaling gate (Part 5 — deferred on purpose)

Adding services/skills multiplies whatever the instrument is. Breadth is unlocked per category only after that category has: (a) a programmatic checker, (b) a provisioning path that never silently falls back to source (enforced — provisionFallback/comparable flags in provenance), (c) N ≥ 2 with reported variance. Diagrams clears this gate (live-validated).

Checkers now exist for data-visualization, documents, markup, image-processing (§7) — the (a) requirement is met for these. Remaining before their numbers are promoted from provisional to v0-validated: a clean provisioning entry in skills.curated.json for at least one tool per category, and a live N≥5 deep run. Only then does skills.curated.json expand further across the 29 categories, and MCP services beyond excalidraw get added. Order stays: checker → provisioning → live-validate → expand, never expand-first.

Changelog

v0.1 (2026-07-06) — measurement sharpened

Changes that alter default scoring, hence a new hash. Profiles under the prior hash (57cb8d0be70fb17a, the first diagram batch) are a valid v0 baseline but are NOT comparable to v0.1 numbers — regenerate to compare.

  • Determinism = pass^k (ax/instruments/reliability.ts) — the τ-bench estimator P(k random trials all pass), replacing the old success/turn-variance heuristic. Sharper, standard, checker-grounded where available.
  • Human surface instrumented (ax/instruments/human.ts) — human.safety (reversibility/oversight), human.verifiability (transparency), human.coherence (steerable control surface) are now scored from checkable patterns (Microsoft Agentic Design Principles) instead of resting entirely on the rater.
  • OTel GenAI adapter (ax/harness/otel.ts, docs/otel-telemetry-contract.md) — any OpenTelemetry-instrumented harness can feed the matrix; PI is one adapter.
  • Recursion.determinism via FRESH-CONTEXT CONTINUATION — the compaction-survival signal now comes from T2: each segment is a cold agent (no conversation history) seeing only the cwd, so if the tool's artifacts aren't self-describing, a cold agent breaks prior work and goalProgress regresses. Stability across segments = the score. This retires T5's compaction-forcing entirely (it cost ~43 min/tool for compaction=0 on a one-shot agent — pure waste). T5 remains defined/opt-in (--tiers T5) for genuinely session-continuous agents, but is out of the default.
  • Parallel trials — the N independent trials of a tier run CONCURRENTLY (OOTA_TRIAL_CONCURRENCY, default 5), not serially; PI switched from blocking spawnSync to async spawn. A tier's wall-time drops from N× to ≈ the slowest single trial. (The remaining floor is per-call agent latency — configurable cap OOTA_PI_TIMEOUT_MS, with timeouts logged, not silently absorbed.)
  • Merge preserves evidence — collapse() no longer discards raw/std/n when two instruments fill the same cell (pass^k curve, token counts, checker detail now survive).
  • OTel ingest is live — ax otel <trace.otlp.json> <subject> [--cwd dir] scores telemetry cells from any OpenTelemetry GenAI trace (not just a bare adapter).
  • Incremental persistence — ax deep writes a partial profile after every tier, so a long/killed run is never a total loss.
  • Task-suite quarry (docs/task-suites.md) — where a validated external benchmark has a programmatic verifier, adapt it (verifier→checker, tasks→briefs); reference only, gated per the rules above.

Teaching sources — the source is a variable of AX

A tool doesn't serve an agent in the abstract; it serves the agent through a teaching source — an official skill, an official MCP, a library-specific community skill, or the tool's official documentation. The same tool taught two different ways gives two different ergonomics. So the AX subject is a (tool × source) pair, and comparing sources is a first-class goal, not noise.

Source kinds (all must be tool-specific)

Ranked by authority, but all are legitimate as long as they are specific to that library — never generic ("a diagram skill") and never our own hand-rolled wrapper:

  1. official-skill — a skill published by the tool's vendor (e.g. Resend's react-email).
  2. official-mcp — the tool's official MCP server (e.g. the Excalidraw MCP).
  3. official-docs — the tool's own documentation, pulled into markdown (fixtures/official-docs/<tool>.md). The always-available baseline when no official skill/MCP exists — the official information is present, not absent.
  4. community-skill — a library-specific skill from skills.sh, used when no official skill exists. Must be for that library (e.g. d2-diagram-creator for D2), highest-adoption preferred, and its rationale recorded in skills.curated.json.

Explicitly excluded: generic/catch-all skills, and any hand-rolled wrapper of ours (the former OOTA wrappers). Those measure our implementation, not the tool.

Addressing a source

diagrams-d2          # curated primary (here: the tool-specific community skill)
diagrams-d2@docs     # the official D2 documentation
diagrams-d2@skill    # the tool-specific skill, explicitly
diagrams-d2@mcp      # the official MCP, explicitly

Each (tool × source) produces its own profile, keyed by the suffixed id (diagrams-d2@docs), so multiple sources for one tool coexist and rank independently on the leaderboard.

Comparing sources (the "over time" goal)

Because the harness, reference agent, briefs, and N are held constant, a difference between diagrams-d2@docs and diagrams-d2@skill is attributable to the source — i.e. which way of teaching the tool serves the agent better. Two uses:

  • Per-source ranking — "for D2, the community skill beats the raw docs on Disclosure (+0.4) but is even on Loop." Actionable: it tells you whether a skill is worth its maintenance, and where docs fall short.
  • Tool-level roll-up — a tool's AX can be summarized as the best available source (what a well-set-up agent would actually use) and/or the mean across sources (how the tool tends to land however it's taught). Report both; never silently average away a great skill or a broken one.

As more skills accrue for a tool over time, this becomes a longitudinal study: the spread across sources is itself a signal (a tool that only works well via one bespoke skill is more fragile than one that's ergonomic from its official docs).

Status

  • official-docs pulled for d2, mermaid, plantuml, matplotlib (the diagram + data-viz tools that have no official skill).
  • Resolution + mounting live (resolveSubject, <key>@docs).
  • Cross-source comparison instrument: designed here, not yet built — the data (per-source profiles) is what it consumes; wire it once ≥2 sources per tool have been run.