PDF to Markdown benchmark: who keeps the structure?
Five real, public PDFs. Three tools with default settings. Headings, code blocks and tables checked against an independent ground truth — with the script to rerun it yourself. Try your own file below.
Your files are never stored: documents convert in your browser where possible, and anything sent to our AI travels encrypted (HTTPS) and is discarded after processing. Privacy Policy
“Keeps the heading hierarchy” is the claim every converter makes. This page measures it. We took five public PDFs from five different genres, converted each with CleanMD 0.93.0, MarkItDown 0.1.7 and pymupdf4llm 1.28.2, and scored the Markdown against a ground truth built from the documents themselves: the official table of contents, or, for the court opinion, the markup of its Cornell LII edition — never from any tool's output.
Pandoc is not in the table because it cannot read PDF (PDF is an output format for it). Everything here is reproducible: the corpus is public, and the benchmark kit (a small zip: run.sh, the ground-truth builder, the scorer, the ground truth itself, the results, and the raw CleanMD outputs so every sentence below can be checked against the actual Markdown) rebuilds every number in about two minutes. The same kit lives on GitHub as dylan82/cleanmd-benchmark (MIT), where the ground truth can be reviewed and disputed in the open. Last run: 2026-09-15, CleanMD 0.93.0.
The corpus
| Document | Why it is hard | Ground truth |
|---|---|---|
| RFC 9110 — HTTP Semantics IETF · 194 pages | 291 numbered sections, ABNF grammar and ASCII diagrams, running footer with page numbers | the 291 numbered sections from the table of contents of the official rfc9110.txt |
| BERT: Pre-training of Deep Bidirectional Transformers arXiv 1810.04805 · 16 pages | two-column paper with results tables and captions next to them | the 29 numbered section headings (lines set in the Medium weight that start with the section number) |
| Think Python, 2nd edition Green Tea Press · 292 pages | technical book: table of contents, 500+ code listings, operator tables | the 126 entries of the book's own table of contents (pages 13–17) |
| Loper Bright Enterprises v. Raimondo (opinion of the Court) Supreme Court of the United States · 114 pages | legal opinion: parts I/II/III and sub-parts A/B/C set as single centered characters at body size | the 31 part markers (I, II, A, B, 1…) of the opinion, the Gorsuch concurrence and the Kagan dissent, read from the markup of the Cornell LII edition, where each marker is its own centered heading element; the page geometry is only a cross-check (it matched 31 for 31) and supplies the page numbers |
| NIST Cybersecurity Framework 2.0 (CSWP 29) NIST · 32 pages | standard with table of contents, appendices and tables with multi-line cells | the 8 top-level entries of the document's table of contents (5 sections + 3 appendices) |
RFC 9110 — HTTP Semantics
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Numbered sections recognised as headings | 291 / 291 | 0 / 291 | 290 / 291 |
| …of which at the right depth | 291 / 291 | 0 / 291 | 275 / 291 |
| Fenced code blocks (ABNF, ASCII diagrams, examples) | 158 | 0 | 0 |
| Collected ABNF appendix as one block | yes | no | no |
| Running footer lines left in the text (“Standards Track … Page N”) | 0 | 187 | 194 |
| Markdown tables emitted (informative: more is not better) | 19 | 22 | 18 |
| Table cells containing a sentence of prose (contamination) | 11 | 0 | 12 |
| Seconds (same laptop, single run) | 1.31 | 5.46 | 10.85 |
- →Sections: CleanMD 291 of 291, pymupdf4llm 290, MarkItDown 0. History, because it matters: the first run (2026-09-11, CleanMD 0.84.0) scored 280 of 291 and pymupdf4llm won this row; the 11 misses were bold headings whose glyphs came from two font subsets, fixed the same day and re-measured here.
- →Fenced code blocks: CleanMD 158, MarkItDown 0, pymupdf4llm 0. Only CleanMD keeps the collected ABNF appendix as one block (yes / no / no).
- →Running footer lines (“Fielding, et al. Standards Track Page N”) left in the text: CleanMD 0, MarkItDown 187, pymupdf4llm 194. MarkItDown emits 0 headings on this document.
BERT: Pre-training of Deep Bidirectional Transformers
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Numbered sections recognised as headings | 26 / 29 | 0 / 29 | 26 / 29 |
| …of which at the right depth | 26 / 29 | 0 / 29 | 26 / 29 |
| Markdown tables emitted (informative: more is not better) | 11 | 134 | 9 |
| Table cells containing a sentence of prose (contamination) | 0 | 0 | 0 |
| Seconds (same laptop, single run) | 0.52 | 0.84 | 2.03 |
- →Section recall is a tie: 26 of 29 for CleanMD and 26 for pymupdf4llm, both at the right depth. The misses are the same for both (A, B, C: the appendix openers).
- →MarkItDown turns the two-column layout into 134 pseudo-tables and emits 0 headings — the worst failure mode for a downstream LLM.
- →CleanMD extracts 11 tables with 0 prose cells; the two tables that sit side by side on page 6 are still merged into one — a known limit.
Think Python, 2nd edition
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Numbered sections recognised as headings | 122 / 126 | 0 / 126 | 113 / 126 |
| …of which at the right depth | 122 / 126 | 0 / 126 | 0 / 126 |
| Fenced code blocks | 569 | 0 | 657 |
| …of which single-line fences (informative) | 52 | 0 | 328 |
| Operator listing (x != y … x >= y) as a fence | yes | no | no |
| Markdown tables emitted (informative: more is not better) | 17 | 683 | 17 |
| Table cells containing a sentence of prose (contamination) | 1 | 0 | 35 |
| Seconds (same laptop, single run) | 0.93 | 3.64 | 11.26 |
- →CleanMD finds 122 of 126 sections, 122 at the right depth. pymupdf4llm finds 113 but 0 at the right depth: every section comes out at the same level, so the hierarchy is flat.
- →pymupdf4llm emits more fences (657 vs 569), but 328 of them are single-line fences (CleanMD: 52); the operator listing that opens §5.2 is a fence in CleanMD (yes) and not in pymupdf4llm (no).
- →MarkItDown emits 683 tables and 6 headings on a 292-page book: the code listings become tables.
Loper Bright Enterprises v. Raimondo (opinion of the Court)
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Part markers recognised as headings | 28 / 31 | 0 / 31 | 18 / 31 |
| Part markers swallowed into a paragraph (roman and letters) | 2 | 17 | 4 |
| Running page headers promoted to headings | 0 | 0 | 80 |
| Headings emitted (5 opinion titles + the part markers are expected) | 33 | 0 | 104 |
| Markdown tables emitted (informative: more is not better) | 0 | 123 | 0 |
| Table cells containing a sentence of prose (contamination) | 0 | 2 | 0 |
| Seconds (same laptop, single run) | 0.43 | 2.44 | 5.29 |
- →CleanMD turns 28 of 31 part markers into headings and leaves 2 fused in the text; pymupdf4llm gets 18 and emits 104 headings, 80 of which are the running page header (“Syllabus”, “LOPER BRIGHT ENTERPRISES v. RAIMONDO”) promoted to a heading.
- →MarkItDown: 0 headings, 17 markers fused into paragraphs, 123 pseudo-tables.
- →Ground truth note: the 31 markers come from the markup of the Cornell LII edition of the opinion, not from any tool and not from page geometry; the geometric rule that the first run used (a single centered token at body size, which is also how CleanMD detects these markers) is kept only as a cross-check, and it agreed with the LII list 31 for 31. The fused-marker count is what any reader can grep for in the output.
NIST Cybersecurity Framework 2.0 (CSWP 29)
Best value per row highlighted; ties highlight every winner.
| Metric | CleanMD | MarkItDown | pymupdf4llm |
|---|---|---|---|
| Numbered sections recognised as headings | 8 / 8 | 0 / 8 | 8 / 8 |
| …of which at the right depth | 8 / 8 | 0 / 8 | 8 / 8 |
| Markdown tables emitted (informative: more is not better) | 3 | 4 | 5 |
| Table cells containing a sentence of prose (contamination) | 23 | 0 | 12 |
| Seconds (same laptop, single run) | 0.36 | 0.95 | 2.04 |
- →CleanMD and pymupdf4llm find all 8 top-level entries as headings at the right depth; MarkItDown finds 0 and emits 0 headings.
- →Table cells containing a sentence of prose: CleanMD 23, pymupdf4llm 12, MarkItDown 0. Read CleanMD's count with care: since v0.87 it keeps the Tier table (Appendix B, four paragraph-length cells per row) as one table per page instead of one block per Tier, so its 66 table rows include cells that are sentences by design; the sentence below Table 1, previously swallowed as a last row, is back in the text.
Where CleanMD loses
A benchmark published by a vendor is only worth reading if it shows the losses. These are the metrics where another tool does strictly better than CleanMD in this run:
- →RFC 9110 — HTTP Semantics: table cells containing a sentence of prose (contamination) — MarkItDown (0 vs CleanMD 11)
- →Think Python, 2nd edition: fenced code blocks — pymupdf4llm (657 vs CleanMD 569)
- →Think Python, 2nd edition: table cells containing a sentence of prose (contamination) — MarkItDown (0 vs CleanMD 1)
- →NIST Cybersecurity Framework 2.0 (CSWP 29): table cells containing a sentence of prose (contamination) — MarkItDown (0 vs CleanMD 23)
Changelog of this page: the first run (2026-09-11, CleanMD 0.84.0) had CleanMD at 280 of 291 RFC sections, behind pymupdf4llm. The 11 misses were bold headings split across two font subsets; the fix shipped the same day and the numbers above are the re-run. We keep this note so the comparison cannot be read as if it had always looked this way.
Method
Sections recognised. A ground-truth section counts as found when a Markdown heading (any level) carries its number and the first 18 characters of its title (letters and digits, case-insensitive). “At the right depth” means the heading level matches the numbering depth (1 → Hn, 1.1 → Hn+1, …) with a constant offset per document, so a tool that starts at H1 or H2 is not penalised, but a tool that flattens everything to one level is.
Code fences. Closed ``` blocks. For RFC 9110 we also check that the collected ABNF appendix survives as one block; for Think Python, that the relational-operator listing is a fence rather than a table or prose.
Tables. GFM tables emitted (informative only: more is not better), their rows, and “prose cells”: cells holding a sentence (seven or more words with three consecutive lowercase words — a lower bound, Title Case sentences are not counted). A table with a sentence in a cell is either fake or contaminated by the text next to it — for a downstream LLM that is worse than no table.
Legal opinion. The parts of a US court opinion are single centered characters (I, II, A, B) at body size. We count how many become headings and how many roman/letter markers are swallowed into the following paragraph (“…ambiguous laws. I Our…”; digits are excluded from that grep because they collide with footnote references), plus how many running page headers become headings. Ground truth: the 31 markers are read from the markup of the Cornell LII edition of the opinion, where each marker is its own centered heading element. The first run of this benchmark defined them geometrically instead (a single token, centered, at body size), which is also the rule CleanMD uses to detect them; that definition is now only a cross-check, and it agrees with the LII list 31 for 31.
Timing. Wall-clock seconds on the same laptop, one run, default options. CleanMD ran in Node with the exact engine that runs in the browser; in the browser expect roughly the same order of magnitude.
Reproducing CleanMD's numbers. CleanMD's engine is not open source, so the kit ships the exact Markdown it produced for the five files; you can also convert the same PDFs on cleanmd.dev (free, no signup) and drop the result into the kit's out/ folder before scoring.
What this benchmark does not measure. Scanned PDFs (none in the corpus: all five have a text layer), reading order inside dense multi-column layouts, math, and the quality of prose. Five documents are five documents: they show failure modes, not a ranking for every PDF in the world.