matplotlib
grade
reliability
overall
Ground truth
2 of 2 requirements met · deterministic checker, no model judgment- figure produced — gapminder_comparison.png
- chart source runs clean — exit 0
Reliability (pass^k): one attempt succeeds 80% of the time · three in a row 40% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
matplotlib: plotting with Python
Matplotlib is a Python plotting library that generates static, vector, or raster visualizations via script execution. It provides an object-oriented programming interface that the agent writes to a script, executes in Python, and saves to local files. While the library produces high-quality, customized output figures, repeated executions within the same workspace often suffer from run-to-run instability.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Determinism | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
Teaching sources compared
skill (A−) serves the agent better than docs (B) for matplotlib. The gap is widest on Human (0.17).
The specialized skill material serves the agent better than standard documentation. The gap in performance between these teaching sources is widest during human-agent interactions. The skill material succeeds because it guides the agent directly to the explicit, object-oriented plotting interface and enforces layouts like constrained placement. In contrast, the standard documentation presents multiple interface models, including both the implicit state-machine and explicit object-oriented API, which can cause the agent to generate mixed-paradigm code that fails during runtime execution.
| Source | Grade | Renders | Disclosure | Interface | Loop | Recursion | Human |
|---|---|---|---|---|---|---|---|
| skill ★ | A− | ✓ | +0.39 | +0.59 | +0.33 | +0.25 | +0.15 |
| official docs | B | ✓ | +0.30 | +0.42 | +0.49 | +0.30 | +0.32 |
Same harness, same reference agent, same briefs, same trial count — the only variable is how the agent was taught the tool. ★ = best source.
The experience
The agent interacts with the tool by writing and executing a custom Python script that parses data and configures axes, legends, scales, and annotations. This script-based workflow works well for determinism and safety because the plotting operation is local, self-contained, and idempotent. Writing to a dedicated file ensures that running the script repeatedly does not corrupt global system states.
State portability works exceptionally well during task handoffs. A fresh agent resuming the task from workspace files alone does not break or regress previous progress. This success occurs because all visual parameters, text bounding boxes, and axis scales are explicitly defined in the saved Python code.
However, repeated execution of the plotting commands within a single session is highly unreliable. If the agent needs to refine or re-run the script multiple times, the tool frequently fails. The agent struggles to handle the verbose specifications required for fine styling, leading to formatting and runtime compilation errors in later turns.
Positioning layout elements also poses difficulty for the agent. Configuring multiple legends, drawing exact annotations, and dividing charts with log scales requires repeated turns of script editing and command-line execution to fix overlapping boundaries. The lack of automatic, failure-proof text overlapping features forces the agent into expensive save-and-inspect loops.
Fit
Matplotlib is a good fit for analytical agents that must produce publication-quality static charts with highly customized visual parameters. It works well when the execution environment allows the agent to edit, run, and verify local scripts through iterative visual feedback.
The library is a poor fit for agents operating under strict execution budgets or in environments without image inspection capabilities. Because the agent relies on repeated runs to correct overlapping text and axis margins, using this tool in expensive or text-only environments is inefficient.
Notes
The following diagram outlines the agent's loop when generating and verifying figures with this tool.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 11 of 16 cross-checked cells (79%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| human.verifiability | -0.20 | +0.80 | 1.00 |
| recursion.verifiability | -0.20 | +0.70 | 0.90 |
| recursion.economy | +0.88 | +0.00 | 0.88 |
| loop.economy | +0.63 | +0.00 | 0.63 |
| recursion.coherence | -0.30 | +0.20 | 0.50 |
Models & runs
2026-07-06| Model | Harness | Runs | Solved | Turns | Knows it |
|---|---|---|---|---|---|
 glm-5.2 |  PI | 25 | 23/25 | 12 | 100% |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn).
Hover a model — here or in the graph — to isolate its runs.
“Knows it” = a closed-book quiz (12 questions, no docs) on whether the model already knows this tool from pre-training — 100% across the panel.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.8924
- hard-measured cells
- 53%
- teaching source
- matplotlib (skills.sh)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.