PDF to Markdown benchmark: who keeps the structure?
Five real, public PDFs. Three tools with default settings. Headings, code blocks and tables checked against an independent ground truth — with the script to rerun it yourself. Try your own file below.
Your files are never stored: documents convert in your browser where possible, and anything sent to our AI travels encrypted (HTTPS) and is discarded after processing. Privacy Policy
“Keeps the heading hierarchy” is the claim every converter makes. This page measures it. We took five public PDFs from five different genres, converted each with CleanMD 0.86.0, MarkItDown 0.1.7 and pymupdf4llm 1.28.2, and scored the Markdown against a ground truth built from the documents themselves: the official table of contents, or the geometry of the page — never from any tool's output.
Pandoc is not in the table because it cannot read PDF (PDF is an output format for it). Everything here is reproducible: the corpus is public, and the benchmark kit (a small zip: run.sh, the ground-truth builder, the scorer, the ground truth itself, the results, and the raw CleanMD outputs so every sentence below can be checked against the actual Markdown) rebuilds every number in about two minutes. Last run: 2026-09-11, CleanMD 0.86.0.
The corpus
| Document | Why it is hard | Ground truth |
|---|---|---|
| RFC 9110 — HTTP Semantics IETF · 194 pages | 291 numbered sections, ABNF grammar and ASCII diagrams, running footer with page numbers | the 291 numbered sections from the table of contents of the official rfc9110.txt |
| BERT: Pre-training of Deep Bidirectional Transformers arXiv 1810.04805 · 16 pages | two-column paper with results tables and captions next to them | the 29 numbered section headings (lines set in the Medium weight that start with the section number) |
| Think Python, 2nd edition Green Tea Press · 292 pages | technical book: table of contents, 500+ code listings, operator tables | the 126 entries of the book's own table of contents (pages 13–17) |
| Loper Bright Enterprises v. Raimondo (opinion of the Court) Supreme Court of the United States · 114 pages | legal opinion: parts I/II/III and sub-parts A/B/C set as single centered characters at body size | the 31 part markers: single-token lines (I, II, A, B, 1…) centered on the page at 11pt. Caveat: this geometric definition mirrors the heuristic CleanMD itself uses for this genre — see Method |
| NIST Cybersecurity Framework 2.0 (CSWP 29) NIST · 32 pages | standard with table of contents, appendices and tables with multi-line cells | the 8 top-level entries of the document's table of contents (5 sections + 3 appendices) |
RFC 9110 — HTTP Semantics
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Numbered sections recognised as headings | 291 / 291 | 0 / 291 | 290 / 291 |
| …of which at the right depth | 291 / 291 | 0 / 291 | 275 / 291 |
| Fenced code blocks (ABNF, ASCII diagrams, examples) | 158 | 0 | 0 |
| Collected ABNF appendix as one block | yes | no | no |
| Running footer lines left in the text (“Standards Track … Page N”) | 0 | 187 | 194 |
| Markdown tables emitted (informative: more is not better) | 22 | 22 | 18 |
| Table cells containing a sentence of prose (contamination) | 9 | 0 | 12 |
| Seconds (same laptop, single run) | 1.46 | 5.46 | 10.85 |
- →Sections: CleanMD 291 of 291, pymupdf4llm 290, MarkItDown 0. History, because it matters: the first run (2026-09-11, CleanMD 0.84.0) scored 280 of 291 and pymupdf4llm won this row; the 11 misses were bold headings whose glyphs came from two font subsets, fixed the same day and re-measured here.
- →Fenced code blocks: CleanMD 158, MarkItDown 0, pymupdf4llm 0. Only CleanMD keeps the collected ABNF appendix as one block (yes / no / no).
- →Running footer lines (“Fielding, et al. Standards Track Page N”) left in the text: CleanMD 0, MarkItDown 187, pymupdf4llm 194. MarkItDown emits 0 headings on this document.
BERT: Pre-training of Deep Bidirectional Transformers
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Numbered sections recognised as headings | 26 / 29 | 0 / 29 | 26 / 29 |
| …of which at the right depth | 26 / 29 | 0 / 29 | 26 / 29 |
| Markdown tables emitted (informative: more is not better) | 12 | 134 | 9 |
| Table cells containing a sentence of prose (contamination) | 0 | 0 | 0 |
| Seconds (same laptop, single run) | 0.31 | 0.84 | 2.03 |
- →Section recall is a tie: 26 of 29 for CleanMD and 26 for pymupdf4llm, both at the right depth. The misses are the same for both (A, B, C: the appendix openers).
- →MarkItDown turns the two-column layout into 134 pseudo-tables and emits 0 headings — the worst failure mode for a downstream LLM.
- →CleanMD extracts 12 tables with 0 prose cells; the two tables that sit side by side on page 6 are still merged into one — a known limit.
Think Python, 2nd edition
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Numbered sections recognised as headings | 122 / 126 | 0 / 126 | 113 / 126 |
| …of which at the right depth | 122 / 126 | 0 / 126 | 0 / 126 |
| Fenced code blocks | 569 | 0 | 657 |
| …of which single-line fences (informative) | 52 | 0 | 328 |
| Operator listing (x != y … x >= y) as a fence | yes | no | no |
| Markdown tables emitted (informative: more is not better) | 17 | 683 | 17 |
| Table cells containing a sentence of prose (contamination) | 1 | 0 | 35 |
| Seconds (same laptop, single run) | 0.95 | 3.64 | 11.26 |
- →CleanMD finds 122 of 126 sections, 122 at the right depth. pymupdf4llm finds 113 but 0 at the right depth: every section comes out at the same level, so the hierarchy is flat.
- →pymupdf4llm emits more fences (657 vs 569), but 328 of them are single-line fences (CleanMD: 52); the operator listing that opens §5.2 is a fence in CleanMD (yes) and not in pymupdf4llm (no).
- →MarkItDown emits 683 tables and 6 headings on a 292-page book: the code listings become tables.
Loper Bright Enterprises v. Raimondo (opinion of the Court)
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Part markers recognised as headings | 28 / 31 | 0 / 31 | 18 / 31 |
| Part markers swallowed into a paragraph (roman and letters) | 2 | 17 | 4 |
| Running page headers promoted to headings | 0 | 0 | 80 |
| Headings emitted (5 opinion titles + the part markers are expected) | 33 | 0 | 104 |
| Markdown tables emitted (informative: more is not better) | 0 | 123 | 0 |
| Table cells containing a sentence of prose (contamination) | 0 | 2 | 0 |
| Seconds (same laptop, single run) | 0.42 | 2.44 | 5.29 |
- →CleanMD turns 28 of 31 part markers into headings and leaves 2 fused in the text; pymupdf4llm gets 18 and emits 104 headings, 80 of which are the running page header (“Syllabus”, “LOPER BRIGHT ENTERPRISES v. RAIMONDO”) promoted to a heading.
- →MarkItDown: 0 headings, 17 markers fused into paragraphs, 123 pseudo-tables.
- →Caveat: the ground truth for this document is geometric (single centered token at body size) and mirrors the rule CleanMD uses to detect these markers — read its numbers with that in mind; the fused-marker count is what any reader can grep for in the output.
NIST Cybersecurity Framework 2.0 (CSWP 29)
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Numbered sections recognised as headings | 8 / 8 | 0 / 8 | 8 / 8 |
| …of which at the right depth | 8 / 8 | 0 / 8 | 8 / 8 |
| Markdown tables emitted (informative: more is not better) | 4 | 4 | 5 |
| Table cells containing a sentence of prose (contamination) | 5 | 0 | 12 |
| Seconds (same laptop, single run) | 0.35 | 0.95 | 2.04 |
- →CleanMD and pymupdf4llm find all 8 top-level entries as headings at the right depth; MarkItDown finds 0 and emits 0 headings.
- →Table cells containing a sentence of prose: CleanMD 5, pymupdf4llm 12, MarkItDown 0. The Tier table (multi-line cells) is split into blocks by every tool; CleanMD also swallows the sentence below Table 1 as a last row — an open issue.
Where CleanMD loses
A benchmark published by a vendor is only worth reading if it shows the losses. These are the metrics where another tool does strictly better than CleanMD in this run:
- →RFC 9110 — HTTP Semantics: table cells containing a sentence of prose (contamination) — MarkItDown (0 vs CleanMD 9)
- →Think Python, 2nd edition: fenced code blocks — pymupdf4llm (657 vs CleanMD 569)
- →Think Python, 2nd edition: table cells containing a sentence of prose (contamination) — MarkItDown (0 vs CleanMD 1)
- →NIST Cybersecurity Framework 2.0 (CSWP 29): table cells containing a sentence of prose (contamination) — MarkItDown (0 vs CleanMD 5)
Changelog of this page: the first run (2026-09-11, CleanMD 0.84.0) had CleanMD at 280 of 291 RFC sections, behind pymupdf4llm. The 11 misses were bold headings split across two font subsets; the fix shipped the same day and the numbers above are the re-run. We keep this note so the comparison cannot be read as if it had always looked this way.
Method
Sections recognised. A ground-truth section counts as found when a Markdown heading (any level) carries its number and the first 18 characters of its title (letters and digits, case-insensitive). “At the right depth” means the heading level matches the numbering depth (1 → Hn, 1.1 → Hn+1, …) with a constant offset per document, so a tool that starts at H1 or H2 is not penalised, but a tool that flattens everything to one level is.
Code fences. Closed ``` blocks. For RFC 9110 we also check that the collected ABNF appendix survives as one block; for Think Python, that the relational-operator listing is a fence rather than a table or prose.
Tables. GFM tables emitted (informative only: more is not better), their rows, and “prose cells”: cells holding a sentence (seven or more words with three consecutive lowercase words — a lower bound, Title Case sentences are not counted). A table with a sentence in a cell is either fake or contaminated by the text next to it — for a downstream LLM that is worse than no table.
Legal opinion. The parts of a US court opinion are single centered characters (I, II, A, B) at body size. We count how many become headings and how many roman/letter markers are swallowed into the following paragraph (“…ambiguous laws. I Our…”; digits are excluded from that grep because they collide with footnote references), plus how many running page headers become headings. Caveat: the ground truth here is geometric — a single token, centered, at body size — and that is also the rule CleanMD uses to detect these markers. It is the only definition we found that does not depend on any tool, but it is closer to our engine than the tables of contents used for the other four documents; weigh the SCOTUS rows accordingly.
Timing. Wall-clock seconds on the same laptop, one run, default options. CleanMD ran in Node with the exact engine that runs in the browser; in the browser expect roughly the same order of magnitude.
Reproducing CleanMD's numbers. CleanMD's engine is not open source, so the kit ships the exact Markdown it produced for the five files; you can also convert the same PDFs on cleanmd.dev (free, no signup) and drop the result into the kit's out/ folder before scoring.
What this benchmark does not measure. Scanned PDFs (none in the corpus: all five have a text layer), reading order inside dense multi-column layouts, math, and the quality of prose. Five documents are five documents: they show failure modes, not a ranking for every PDF in the world.