excalidraw
grade
reliability
overall
Ground truth
4 of 5 requirements met · deterministic checker, no model judgment- valid .excalidraw file — 41 elements
- 6+ shapes — 30 shapes
- 8+ arrows bound to shapes — 9/11 bound
- 2+ frames/containers — 0 frames
- a title text — present
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone sometimes regressed prior work.
Virtual whiteboard for sketching hand-drawn like diagrams
Excalidraw represents canvas diagrams as structured JSON data containing arrays of geometric elements and coordinate definitions. Handing this tool to an agent requires it to manually compute spatial dimensions, reference matching IDs for connections, and define styling parameters textually. Without runtime documentation or validation libraries, the agent must rely entirely on pre-trained knowledge to generate the diagram schema.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Determinism | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 47% of this matrix is hard-measured.
The experience
Generating individual elements such as rectangles, text labels, and basic line parameters works adequately. The agent successfully outputs standard shapes and varied line styles, producing file structures that render without parsing crashes.
However, defining arrow-to-shape bindings works poorly. The agent must calculate relative visual offsets and generate matching identifier strings inside the JSON manually, which is highly sensitive to minor coordinate errors. Because the environment does not provide visual preview capabilities, the agent operates blindly and cannot check if its connectors align with target shapes.
Structuring complex container frames or grouped sections works poorly. Because there is no local schema verification, the agent outputs incorrect container properties, resulting in zero recognized grouping frames in the rendered artifact.
Iterative editing across separate sessions works poorly. When a fresh agent resumes work from a saved JSON canvas, the lack of structural metadata or inline schema guidance makes the raw dimensions difficult to interpret, often causing the agent to regress prior work when applying updates.
Fit
This tool is an adequate fit for agents generating simple, flat flowcharts or block layouts where only basic shapes and annotations are needed in a single turn. It is a poor fit for production-grade diagrams demanding custom container frames, exact arrow alignments, or collaborative, multi-step design updates.
Notes
The following diagram illustrates the open-loop generation process where the lack of an interactive rendering loop or schema linter leads to undetected structural errors:
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 5 of 14 cross-checked cells (76%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| loop.determinism | +0.70 | -0.30 | 1.00 |
| recursion.verifiability | -1.00 | +0.00 | 1.00 |
| recursion.determinism | +0.94 | +0.00 | 0.94 |
| interface.prior_alignment | -0.60 | +0.00 | 0.60 |
| human.verifiability | +0.60 | +0.00 | 0.60 |
| interface.coherence | +0.50 | +0.00 | 0.50 |
| loop.safety | +0.70 | +0.20 | 0.50 |
| interface.verifiability | +1.00 | +0.50 | 0.50 |
| loop.verifiability | +0.56 | +0.10 | 0.46 |
Models & runs
2026-07-06| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
| | 25 | 17/25 | 23 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- Claude Code (claude-code-acp)
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- hard-measured cells
- 47%
- teaching source
- excalidraw-mcp (official)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.