plantuml
grade
reliability
overall
Ground truth
5 of 5 requirements met · deterministic checker, no model judgment- renders without error — exit 0
- 6+ labeled nodes — 15 nodes
- 7+ directed edges — 20 edges
- 2+ clusters/groups — 2 clusters
- has a title — present
Reliability (pass^k): one attempt succeeds 100% of the time · three in a row 100% · all runs 100%. A fresh agent resuming from the files alone never broke prior work.
PlantUML is a command-line tool that compiles plain-text markup into visual diagrams. For an AI agent, the tool processes structured styling directives to generate raster or vector files such as PNG and SVG. It operates locally with high input-to-output predictability when provided with a Java runtime environment.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Determinism | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
Teaching sources compared
The skill source produces a rendering artifact; docs does not — the teaching source is decisive for plantuml.
The teaching sources provided to the agent produced starkly different results, making the source material of critical importance. The official documentation served the agent poorly because it focused exclusively on plain-text and Unicode ASCII generation layouts. When limited to these documents, the agent could not easily produce high-fidelity graphical images and had to rely heavily on prior training to execute visual renders.
In contrast, the skill resource served the agent well. It provided complete graphical context and detailed visual styling parameters that allowed the agent to render standard SVG and PNG outputs. The structured skill templates gave the agent direct instructions for configuring graphical parameters, bypassing the limitations of the text-only documentation.
| Source | Grade | Renders | Disclosure | Interface | Loop | Recursion | Human |
|---|---|---|---|---|---|---|---|
| skill ★ | A− | ✓ | +0.35 | +0.62 | +0.58 | +0.38 | +0.48 |
| official docs | D | ✗ | +0.30 | +0.31 | +0.28 | +0.45 | +0.41 |
Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.
The experience
The tool performs well when generating deterministic graphical diagrams from raw specifications. The agent writes declarative syntax with distinct directives for styling layout clusters and arrow aesthetics, which compiles directly into SVG or PNG formats. The clean text-to-file pipeline allows the agent to iteratively verify and refine complex layouts using predictable command-line arguments.
However, the tool performs poorly if the local system path lacks a pre-configured Java platform wrapper. The agent faces compile-time failures from missing executable dependencies, requiring it to locate the correct OpenJDK path on the local operating system and manually export the path to the configuration variable before executing compilation runs. This path-resolution step represents a friction point where the execution flow is interrupted.
Additionally, the tool performs adequately when handling documentation discrepancies. When the provided documentation is restricted to text-based ASCII output options, the agent must draw upon its internal pre-trained understanding of graphical syntax parameters. The agent utilizes undocumented configuration flags such as the png output flag and color hex styling syntax to achieve graphical outcomes not standardly detailed in the ASCII manual.
Fit
PlantUML is a strong fit for agents that require local, predictable execution of diagrams with zero network dependency. Because the tool operates purely on text files and local compilers, the input and output processes are extremely token-efficient and safe. There is low risk of corrupting external system directories since errors remain isolated within the local file compilations.
It is a poor fit for agents operating in restricted environments where installing Java runtimes or managing binary dependencies is prohibited. In such setups, the agent cannot resolve missing Java compilation dependencies, causing execution pipelines to fail entirely.
Notes
The diagram below illustrates the write-compile-fix loop used by the agent to configure the environment and compile the diagram.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 11 of 16 cross-checked cells (82%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| human.verifiability | -0.20 | +0.90 | 1.10 |
| recursion.verifiability | -0.32 | +0.50 | 0.82 |
| recursion.economy | +0.98 | +0.20 | 0.78 |
| recursion.coherence | -0.11 | +0.50 | 0.61 |
| recursion.determinism | +1.00 | +0.50 | 0.50 |
Models & runs
2026-07-06| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 25 | 22/25 | 13 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.8987
- hard-measured cells
- 53%
- teaching source
- plantuml-ascii (skills.sh)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.