A flat CLI that starts shallow, returns prose (never JSON), and remembers what it already knows in a sidecar next to each PDF — paired with a unified document model that takes a MathPix™ lines.json or an extended version of pdfminer.six as its input and turns it into a typed, queryable graph.
The button opens drillspace in a prebuilt GitHub Codespace — the drillui terminal comes up on a sample paper, nothing to install. Needs a GitHub account.
01 — what it is
pdfdrill returns prose, never JSON, and persists what it learns in a sidecar next to each PDF — so a re-run answers instantly and a higher-level command never repeats a lower one. It wraps the heavy extraction tools so an LLM can reach for the cheapest command that still answers the question, escalating to OCR and the full document model only on demand.
There are five ways to drive it: the CLI itself; a drillui browser terminal for chatting with one drilled document; a stdio MCP server and a Streamable-HTTP MCP endpoint that hand drill results back as clickable resources inside Claude Desktop, Claude Code, or the claude.ai web custom connector; and a batch driver that walks a list of URLs or arXiv ids shallow→deep. Underneath, three packages do the work:
The flat CLI + sidecar state machine + capture layer. Wraps the heavy extraction tools so you never call them by hand. Every command returns one sentence of prose and records cumulative facts — re-runs return instantly.
The typed Document: DocObjects with anchor-based Streams, Realizations across streams, Alignments between them, and Region geometry. MathPix lines.json is the only format that lets you compare a LaTeX string against the image MathPix actually rendered.
Operators over a Document: Mutators modify in place (dehyphenate, Stanza NLP), Projectors emit artifacts — plaintext, LLM-compact markdown, TiddlyWiki tiddlers, the comparison table, and the full formula report.
Anchors are opaque identities, not positions — inserts and deletes in one stream never invalidate references elsewhere. Source streams stay immutable; modules only add objects, realizations, and alignments. The raw MathPix payload is always recoverable verbatim.
02 — the killer case
A paper's page-1 text says only “Our code is available here.” — where here is a hyperlink with no separate anchor text. Plain extraction, MathPix Markdown, and a chatbot PDF upload all drop the annotation layer, so the URL simply does not exist in anything an LLM reads. pdfdrill links reads the annotation layer directly and surfaces it in ~50 ms.
$ pdfdrill links 2605.12061.pdf # ~0.06 s · pure `pdfinfo -url` Found 1 code/data host among external links: https://anonymous.4open.science/r/Unified-Representation-A9D9/ (page 1, no visible anchor text) $ pdfdrill urls 2605.12061.pdf # ~6 s · scans every page to recover anchor text Same link re-derived — only reach for this when you need the visible text. $ # MathPix / Markdown / chatbot upload: the link is gone — annotation layer never consulted.
links answers “where is the code?” in 50 ms by reading the annotation layer. urls re-derives the same answer in ~6 s. A MathPix pass misses it entirely. Powerful ≠ right — escalate only when a cheaper command genuinely can’t answer.
03 — drill depth
Every command records facts in the sidecar, so a higher level never repeats a lower one. pdfdrill size is always step zero — free, ~40 ms, and it tells you whether the auto-extracted text is even enough.
size · pdfinfo · links · dests — metadata, text-layer vs. scanned, annotation-layer URLs, named anchors. No extraction.abstract · toc · fonts — “what is this about / what are the sections / is there math?” Usually all you need.md · page N · drill — real extraction. Full Markdown to the sidecar, a single page, or chained size→fonts→abstract→toc→md.model · compare · report · tiddlers — the unified docmodel + QC artifacts. Offline from an existing lines.json.$ pdfdrill size paper.pdf 18-page PDF, 0.2 MB, letter, produced by xdvipdfmx, has a text layer, not encrypted. $ pdfdrill abstract paper.pdf Abstract: For a finite planar set P, let ν(P) be the number of …
size probes the text layer at L0: a born-digital PDF has extractable text on page 1 and fonts; a scan has neither. Page-1 char count is authoritative (a stray stamp font won’t flip an image PDF to “has text”). When there’s no text layer it says “NO text layer — scanned, OCR required” and sets needs_ocr.
04 — command catalogue
Run as PYTHONPATH=src python3 -m pdfdrill <command> <pdf> [args], or via the installed pdfdrill console script. Click a command to see what it returns.
New — beyond math papers
pdfdrill now reads scanned commercial mail, not just arXiv math. The scans & entities tab adds continuity (recover Seite N von M markers MathPix crops away), segment (split a shuffled bundle into ordered documents), entities (offline-validated IBAN / BIC / address / Aktenzeichen), qr (decode GiroCode payment QR), elements (a layout GNN), and semantic (a cross-document evidence graph) — all keyless and self-contained.
05 — the qc pipeline
MathPix lines.json is the only format that ships, for each equation, both the recognized LaTeX and the CDN crop it was read from. pdfdrill puts them side by side — and adds competing readings (MathPix Snip, an LLM on the crop, the author’s own .tex) as extra columns, each scored against a normalized LaTeX form. Below is a real row from report --embed on arXiv 2004.05631v1, eq (1.1):
| LaTeX source | KaTeX render | MathPix CDN crop |
|---|---|---|
a(x):=\{y \in Y \mid R(x, y)=1\}
\quad b(y):=\{x \in X \mid R(x, y)=1\}
|
|
Each crop’s <img> links to the full page it was cut from (page_url() strips the region query). --embed base64-inlines every crop so the artifact has no live-CDN dependency — ideal for a preview that won’t load remote images. The page click-through stays live even when embedded.
pdfdrill snip re-OCRs each crop through MathPix Snip /v3/text → a candidate LaTeX + a per-line confidence that becomes a score signal.
candidates exports the crops for an LLM to read (no API key); vision runs GPT-4o over them. Both attach as extra provenance columns.
pdfdrill latex aligns the author’s e-print .tex to each MathPix equation — the gold reference versus OCR, preamble-expanded so TikZ/macros resolve.
A test input path renders tables straight from LaTeX rather than KaTeX: latex → DVI → dvisvgm emits a font-faithful vector SVG that drops into the HTML table — useful for full tabular layouts KaTeX can’t express. It stays keyless: latex already ships in the Claude.ai sandbox and dvisvgm (3.2.1) is apt-installed by the bootstrap, so it runs there as readily as on a local machine.
Scoring flags shaky equations; escalate exports only those for a second reading; after ingest, relearn re-scores and reports resolved / improved / still-shaky. On 2605.12061: 9 flagged → 7 resolved, 1 retained (the hardest multi-line equation, correctly kept).
.tex, each a scored column.06 — live from the model
These are pulled verbatim from the extracted model of arXiv 2004.05631v1 (At the Interface of Algebra and Statistics) — the Galois connection at the heart of the thesis. Source LaTeX shown beneath each render; nothing was hand-typed.
07 — the document model
The unified docmodel resolves a paper into typed DocObjects. Projected to TiddlyWiki, each becomes one tiddler — equations carry their LaTeX, reference number, page, the MathPix crop, region geometry, and every competing reading as latex_<provenance> + score_<provenance>.
In-text (N) references rewrite to {{<eq>||FREF}} transclusions; citations link straight to their bibliographic tiddler by number or citekey. The persisted model.docmodel.json is the complex memory that survives between LLM/tool calls.
{ "title": "2004.05631v1_EQ0001_p012", "text": "<$latex text={{!!latex}} displayMode=true />", "tags": "equation 2004.05631v1", "kind": "Equation", "latex": "a(x):=\{y \in Y \mid R(x,y)=1\} …", "refnum": "1.1", "equation_number": "(1.1)", "page": "012", "canonical_uri": "https://cdn.mathpix.com/cropped/…-012.jpg?…", "width": "994", "height": "67", "top_left_x": "259", "top_left_y": "666", "latex_tex": "a(x):=\{y\in Y\mid R(x,y)=1\}…", "score_tex": "0.992", "latex_snip": "a(x):=\{y \in Y \mid …\}", "score_snip": "1" }
pdfdrill semantic lifts the model into a constraint graph: every extractor becomes a sensor emitting evidence, and entities — Company / Person / BankAccount on a letter, Symbol / Claim / Reference on a paper — accumulate it, across documents when you pass --store graph.json. The pass itself is reified: each reading is a Question node, each transform a first-class Transformation node, and a derived belief column carries a deterministic best-reading tiebreak that a compiler type-checks for grounding and consistency. A federated vocabnet layer (MSC · PhySH · GND · STW) backs pdfdrill classify, while stex / scikgtex project the graph back out to enriched LaTeX — glossary, Table-of-Symbols and index, or an ORKG-annotated PDF whose contribution metadata rides along as XMP/RDF.
It’s a single self-contained HTML file — no server, perfect for GitHub Pages — that holds the whole graph and renders the LaTeX widgets live through the KaTeX plugin. The artifact below is exactly that: pdfdrill tiddlers output, built into a real wiki you can open and explore.
docmodel.json.08 — claude.ai & claude code
You don’t run any of this yourself. Make pdfdrill available once — attach the bundle in a Claude.ai chat, or install the skill in Claude Code — then ask in plain language. Claude carries the skill (SKILL.md + the 89-command manifest), so it knows which commands to run, and it reads that manual — not you. The package is pure Python under src/ with no pip step, which is why the sandbox (the extraction tools and pydantic already aboard) runs it as-is. Everything in the terminal below is what Claude does for you — not a checklist you type.
you ▸ “Here’s pdfdrill (attached). Install it and load its skill, then tell me whether this paper’s equations extracted cleanly.” # Claude — now holding the skill — does the rest. You watch; you don’t type any of this: $ tar xzf pdfdrill.tgz # unpack the bundle you attached $ cat src/pdfdrill/skill/SKILL.md # the skill IS Claude’s manual — it reads this, you don’t $ PYTHONPATH=src python3 -m pdfdrill size paper.pdf $ PYTHONPATH=src python3 -m pdfdrill report paper.pdf --embed # → self-contained formula-report.html # In Claude Code the skill is installed once, so step one is simply your question. # (the attachable bundle is one `git archive --format=tar.gz HEAD` of the repo.)
mathpix, snip, bibfetch read MATHPIX_APP_ID/KEY & PERPLEXITY_API_KEY from env or a git-ignored .env — never committed, never asked for mid-task.
The structural path — model, compare, report, tiddlers, folder, latex — runs entirely from an existing lines.json. No key, no network.
New: when there is no API key but pdfdrill is being driven by Claude — the Claude Code CLI locally, or Claude.ai in the sandbox — it routes the prompt-driven sub-tasks (vision, bibtex) to that Claude, using pdfdrill’s own prompts. The prompt is the knowledge: a vision crop returns as recompilable LaTeX, a BibTeX record web-searched correct — no OpenAI or Perplexity key required. pdfdrill llm <pdf> --runtime reports the transport it detected: cli (synchronous claude -p --output-format json) or sandbox (a deferred request file the agent answers, then re-run to ingest). The attached result is byte-identical to the keyed API path. But it is not free: delegating a whole document to the running Claude can take more wall-clock time and burn more tokens than a single keyed call — so for steady use, setting up MathPix pay-as-you-go is often cheaper and faster.
pdfdrill now ships as a Claude skill. A single commands.yaml manifest is the source of truth for the whole 89-command surface — --help, the SKILL.md tables and the external drillui TUI are all generated from it, and a CI gate fails on any drift. A declarative prerequisite planner (pdfdrill steps, --ensure) chains the right commands toward a goal, so the agent asks for an outcome and pdfdrill walks the path.
In the settings menu, the Claude.ai sandbox can be set to allow network access to all hosts. Only after that change are the external services (MathPix — mathpix, snip; ChatGPT / GPT-4o — vision; Perplexity — bibfetch; and Gemma), fetching a PDF by URL, and even apt-installing dvisvgm at bootstrap possible. The keyless offline path — model, compare, report, tiddlers, ocr — touches no host and needs no change.
Attach a PDF in a Claude.ai chat and it is silently flattened to Markdown — math turns to Unicode and the annotation layer (every hyperlink with no visible anchor text) is discarded before pdfdrill ever sees it. So don't hand Claude the bare PDF. Do one of these instead: (1) share a URL to the PDF — pdfdrill fetches it byte-for-byte; or (2) put the PDF in a .zip and upload that — the archive sidesteps the auto-Markdown conversion, so the real bytes reach the tool.
If a PDF is already sitting in the sandbox, tell Claude to run pdfdrill on that file — not the converted text: pdfdrill model <uploaded.pdf>, then pdfdrill llmtext <uploaded.pdf>. The llmtext dump is written to the doc's drill folder as <name>.drill/<name>.llm.txt. pdfdrill does not feed that file into the chat on its own, so explicitly ask Claude to read the .llm.txt file from the drill folder before it answers.
Prefer your own environment over the Claude.ai sandbox? A one-shot bash cocalc-setup.sh brings the CoCalc standard image up to spec — sudo apt-get for poppler-utils, dvisvgm and the LaTeX toolchain, then bun, uv/uvx and the Python deps into your user dirs. The drillui ask-the-document web terminal then runs there via bun run tools/drillui_bridge.ts, reached through CoCalc's port-forward. One CoCalc quirk: the terminal's WebSocket must carry the proxy path — paste wss://<host>/<project-id>/server/8787/ws into its Connect box. Full walkthrough: COCALC.md.
Click, wait for the codespace, and the drillui terminal opens in your browser on a sample paper that is already drilled — nothing installed, no command to type. Ask it a question, or type a pdfdrill command name to run it on the open document.
The container is prebuilt, so the toolchain is already baked in rather than installed on launch. It needs a GitHub account and spends its free-tier quota.
09 — live artifacts
All produced by pdfdrill and served straight from this repository — no build step, no server.
arXiv 2607.02234 · Purified OPSD: On-Policy Self-Distillation new · docmodel inspector
Transformer Circuits Thread · Verbalizable Representations Form a Global Workspace in Language Models new · distill reading view
arXiv 2004.05631v1 · At the Interface of Algebra and Statistics
A self-hosted TiddlyWiki with the KaTeX plugin carrying the complete extracted structure — sections, equations, formulas, sidenotes, footnotes, references. Click any equation to see its full record and live render.
open the wiki →The full pdfdrill report: every inline formula and display equation as LaTeX source │ live KaTeX render │ the MathPix CDN crop, each linking back to its full page.
bibkey heimUFT · Burkhard Heim — Unified Field Theory (study notes; derived model published with permission)
The MBB Lectures (1976) & Metron calculations as a self-hosted TiddlyWiki — 367 sections, 312 equations, 1020 formulas, 15 tables. Gruvbox palette, KaTeX live render; every tiddler prefixed with the bibkey heimUFT.
The full pdfdrill report for heimUFT: every inline formula and display equation as LaTeX source │ live KaTeX render │ the MathPix CDN crop, each linking back to its full page.
The pdfdrill compare table: each display equation as LaTeX │ KaTeX │ the MathPix image side by side, scored across provenances.
The raw TiddlyWiki JSON behind the wiki — the unified document model projected to tiddlers, every title/tag/transclusion keyed to heimUFT.
bibkey kolbe2018hubbard · Kolbe (2018) — Korrelation im Hubbard-Modell (German BA thesis, published with the author's permission)
Alexander Kolbe's BA thesis (Universität zu Köln) as a self-hosted TiddlyWiki — 38 sections, 61 equations, 238 formulas, 13 figures, all images embedded. With the author's own .tex ingested as a gold tex provenance column.
The German wiki run through pdfdrill translate --from DE --to EN-US: 176 prose tiddlers DeepL-translated in place, the original German kept under org_text. Equations and figures are untouched.
The full pdfdrill report for kolbe2018hubbard: every inline formula and display equation as LaTeX source │ live KaTeX render │ the MathPix CDN crop.
The pdfdrill compare table: each equation as MathPix LaTeX │ KaTeX │ image, with the author's gold .tex as a scored extra column.
The raw TiddlyWiki JSON behind the wiki — every title/tag/transclusion keyed to kolbe2018hubbard, images base64-embedded.
The DeepL-translated tiddler set — each prose field holds the English text, the German source preserved under org_<field>, translated tiddlers tagged translated.