latex
grade
reliability
overall
Ground truth
0 of 5 requirements met · deterministic checker, no model judgment- compiles to final format — no compiler
- 3+ sections — 0 sections
- a data table — no
- a generated figure — no
- a bibliography citation — no
Reliability (pass^k): one attempt succeeds 0% of the time · three in a row 0% · all runs 0%. A fresh agent resuming from the files alone never broke prior work.
The latex-paper-en tool is a command-line assistant designed to audit, format, and compile existing LaTeX manuscripts. It operates through specific Python utility scripts that analyze bibliography keys, evaluate grammar, check venue formatting, and diagnose compile-time errors. When tasked with composing a document or figure from scratch, the assistant fails to maintain state or compile any document, as its structure is strictly optimized for diagnosing pre-existing files.
Surfaces × lenses
| surface ↓ lens → | Coherence | Economy | Determinism | Verifiability | Safety |
|---|---|---|---|---|---|
| Disclosure | |||||
| Interface | |||||
| Loop | |||||
| Recursion | |||||
| Human |
Click any cell for what was measured and why. A corner dot marks a cell filled by a hard instrument (checker, telemetry); undotted cells are judged by the rater ensemble. 53% of this matrix is hard-measured.
The experience
The agent interacts with the tool by invoking specific Python utility scripts corresponding to modules like compile, format, and bibliography. The documentation works well because it clearly outlines tool capabilities and explicitly lists forbidden use cases, such as drafting papers from scratch. However, when the agent receives a task that violates these boundaries, the workflow breaks.
Iterative generation and multi-file dependency management work poorly. The agent fails to manage the sequence of executing a Python script to output a vector image and then compiling a LaTeX file that references that image. Because the tool offers no mechanics to handle these generative dependencies, the agent loses track of the execution state across turns.
The code output quality works poorly when compiling from scratch. The agent's loop is unable to resolve compilation errors and eventually writes a truncated, non-functional Python script. This failure loop consumes a large amount of input characters and financial expense while completely failing to render a PDF.
Workspace safety works well. The interface does not perform destructive actions on existing files in the directory, and a fresh agent resuming from the workspace can read the files without prior work being corrupted. When used on existing documents, the compiler error logs work adequately to guide precise corrections.
Fit
This tool is a good fit for agents tasked with editing, proofreading, translation, formatting compliance, and build repair of pre-existing LaTeX manuscripts. The modular python verification scripts and clear compiler diagnostics allow the agent to pinpoint and resolve syntax or citation issues reliably. It is also suitable for agents working within strict conference template venues like IEEE or ACM, where formatting constraints are highly structured.
The tool is a poor fit for agents trying to generate new academic research papers, synthesize data figures dynamically, or manage multi-file compilation chains. It is also unsuitable for workflows based on other document formats like Typst or Word document formats that do not use LaTeX source files. The lack of generative template mechanics and state management will cause the agent to enter loop failures and produce truncated files.
Notes
The following diagram illustrates the failed feedback loop during a generative compilation task.
Validation
Convergent check. Where a hard instrument and the independent rater panel measure the same cell, they agree on 5 of 16 cross-checked cells (68%). Two independent methods landing on the same number is our accuracy signal — not just the raters agreeing with each other.
| Cell where they disagree | Instrument | Rater | Δ |
|---|---|---|---|
| recursion.determinism | +1.00 | -0.50 | 1.50 |
| loop.economy | +0.69 | -0.50 | 1.19 |
| human.coherence | +0.50 | -0.50 | 1.00 |
| recursion.economy | +0.36 | -0.60 | 0.96 |
| interface.coherence | +0.45 | -0.50 | 0.95 |
| recursion.verifiability | +0.40 | -0.50 | 0.90 |
| loop.safety | +0.70 | +0.00 | 0.70 |
| interface.prior_alignment | +0.40 | -0.30 | 0.70 |
| human.verifiability | +0.10 | -0.50 | 0.60 |
| loop.verifiability | -1.00 | -0.50 | 0.50 |
| disclosure.verifiability | +0.47 | +0.00 | 0.47 |
Models & runs
2026-07-07| Model | Harness | Runs | Solved | Turns |
|---|---|---|---|---|
 glm-5.2 |  PI | 22 | 12/22 | 6 |
Each path is one run through the shared semantic space; nodes are turns (red = an errored turn). Hover a model — here or in the graph — to isolate its runs.
How this was measured
- reference agent
- pi:z-ai/glm-5.2
- trials
- 25 (n=5/tier)
- methodology
- v0.1 3e74a3c6
- cost
- $0.8316
- hard-measured cells
- 53%
- teaching source
- latex-paper-en (skills.sh)
Same brief, same reference agent, and the same trial count for every tool in a category — a difference in the numbers is attributable to the tool (and its teaching source), not the setup. Profiles carry the methodology hash; results under different hashes are not compared.