mermaid
grade
reliability
overall
Ground truth
5 of 5 requirements met · deterministic checker, no model judgment- renders without error — exit 0
- 6+ labeled nodes — 17 nodes
- 7+ directed edges — 11 edges
- 2+ clusters/groups — 3 clusters
- has a title — present
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
Generation of diagrams like flowcharts or sequence diagrams from text in a similar manner as markdown
The mermaid command-line tool, mmdc, compiles a text-based definition file into localized SVG or PNG diagrams. This allows agents to generate complex structured schemas programmatically through simple text modifications. The local execution model provides immediate validation feedback and prevents external API dependencies.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Determinism | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
Teaching sources compared
Teaching source barely matters for mermaid: sources land within a grade of each other (docs A−, skill A−).
The choice of teaching source has minimal impact on the agent's performance with this tool. Both the general documentation and the specialized agent skill file enable the agent to render error-free diagrams on the first attempt. This consistency occurs because the Mermaid syntax is rigid and well-documented. An agent can learn the required notation conventions, such as subgraph syntax and link types, equally well from standard official documentation or localized guides.
| Source | Grade | Renders | Disclosure | Interface | Loop | Recursion | Human |
|---|---|---|---|---|---|---|---|
| official docs ★ | A− | ✓ | +0.23 | +0.62 | +0.63 | +0.36 | +0.35 |
| skill | A− | ✓ | +0.31 | +0.64 | +0.55 | +0.38 | +0.31 |
Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.
The experience
The agent runs the mmdc CLI tool in a single-step, local write-and-compile loop. The compiler operates with very low computer resource overhead. Syntax errors do not corrupt existing files or damage the workspace, which allows the agent to safely refine output code.
The structured syntax of Mermaid matches the sequential patterns that agents naturally produce. The compiler parses node shapes, subgraphs, direction markers, and custom style classes. Edge declarations with specific syntax rules, such as using uppercase node labels or quoting special characters, prevent execution errors when followed exactly.
The agent defines relationships using primary solid arrow --> or secondary dotted arrow -.-> paths, which compile cleanly. However, because the layout engine is automated, the agent lacks precise control over the exact placement of individual nodes or lines. The agent must rely on subgraph blocks and class definitions to separate components, resulting in some layout congestion when multiple edges cross cluster boundaries.
Fit
This tool serves as an excellent fit for agents generating technical specifications, software architecture charts, flowcharts, or structural wireframes. The deterministic compiler output translates raw logical definitions into highly readable, consistent diagrams.
It does not fit tasks requiring fine-grained aesthetic control, custom vector positioning, or artistic layout. Because the renderer dictates pixel layout and routing paths, the agent cannot programmatically resolve visual clutter or enforce pixel-perfect spacing.
Notes
The diagram illustrates the write-compile-fix loop used by an agent to generate diagrams cleanly with the mmdc compiler.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 9 of 16 cross-checked cells (75%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| disclosure.verifiability | -0.40 | +0.70 | 1.10 |
| recursion.coherence | -0.46 | +0.60 | 1.06 |
| human.verifiability | -0.20 | +0.80 | 1.00 |
| human.coherence | -0.20 | +0.70 | 0.90 |
| recursion.verifiability | -0.08 | +0.70 | 0.78 |
| interface.coherence | +0.07 | +0.70 | 0.63 |
| loop.economy | +0.97 | +0.50 | 0.47 |
Models & runs
2026-07-06| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 25 | 25/25 | 11 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.7794
- hard-measured cells
- 53%
- teaching source
- mermaid (official docs) (official-docs)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.