Security Research
Delphos outscores commercial decompilers on DecBench
Delphos Lantern takes first place on the overall metric and first place on elusive types metric
DecBench, a new benchmark from the Noelo Lab at the University of Georgia, measures how well a decompiler recovers from the perturbations made by a compiler, including shuffles of the control flow and discarding the type. The benchmark measures the fraction of functions a decompiler recovers perfectly.
We ran Delphos Latern on the DecBench sample-set. On the leaderboard's normalize-failures view, which restricts scoring to the 91 functions every decompiler produced output for, Delphos scored first on the "union" (an aggregate of all three submetrics) and "types" metrics, ahead of Anthropic's claude-code, OpenAI's codex, and Ventris v8.0.0. Historically, commercial decompilers have struggled at recovering variable types, and this is where Delphos excels.
What does DecBench measure?
The DecBench eval assesses decompilers against original functions. A perfect match occurs when at least one of three properties matches the source: the control-flow graph, the variable and argument types, and the recompiled machine bytes. Every function is scored on each axis, and the union aggregates all three.
Structure (Graph Edit Distance). Joern, an open source static analysis platform for source code, lifts both the decompiler's C output and the ground-truth source to control-flow graphs and counts the fewest node and edge insertions/edits/deletes needed to make them isomorphic. A perfect score means the shapes and the control flow match. Matching labels is excluded from the calculation.
Types. DWARF debug information gives DecBench ground-truth types for every recoverable variable and argument. Arguments are matched by ABI position, stack locals by calibrated frame offset, the rest by exact name. A perfect score means every recoverable variable is typed correctly. Variables the compiler optimized out are excluded from the metric calculation.
Recompile bytematch. DecBench rebuilds the decompiled code with the target's original toolchain and diffs the resulting assembly line by line against the original binary, normalizing call targets and other link-time operands. A perfect score means the bytes are the same. A compilability fixup adds what the compiler reports missing, so a decompiler that emitted valid code gets a fair shot at building.
Each metric measures a different kind of correctness, and each is silent about the other two.
DecBench also publishes a sample-set slice of 250 functions across 224 binaries from 38 projects. Running an LLM agent over the full 94,575-function corpus is not practical, so the sample-set is the slice on which claude-code, codex, and Delphos are scored. Coverage varies across decompilers, so the leaderboard's normalize-failures view restricts scoring to the 91 functions for which every decompiler produced output. Of the 91, 60% are compiled with optimization, and 43 of them are O2-noinline, the hardest configuration in the benchmark. Every system on the leaderboard collapses to about 27% structural accuracy on that slice.
Where does Delphos land on the leaderboard?
Delphos scored 52.7% on Union: perfect on at least one of the three metrics for 48 of the 91 functions. That places it first, ahead of claude-code (51.6%), codex (50.5%), and Ventris v8.0.0 (50.5%).
The gap to the frontier LLM agents is narrow, and n=91 is a small sample. The 2σ noise band on this configuration is ±4.6 percentage points, which means that the differences between Delphos and claude-code on the Union metric is within the margin of error. We did four full runs of the configuration to characterize variance. Single runs of this configuration range from 47.3% to 52.7% Union. The claim is a leaderboard rank, it holds, and the leaderboard is close at the top.
Traditional decompilers performed significantly worse. IDA Pro 9.20 scored 20.9% on Union, Binary Ninja scored 19.8%, and Ghidra 12.1 scored 12.1%. Thus, Delphos produces perfectly-recovered output on about 2.5 times as many functions as any commercial decompiler on the board.
The four columns side by side:
Decompiler | Union | Structure (GED) | Types | Recompile (bytes) |
Delphos | 52.7% | 46.7% | 12.6% | 15.4% |
claude-code | 51.6% | 46.2% | 8.8% | 14.3% |
codex | 50.5% | 44.0% | 9.9% | 17.6% |
Ventris v8.0.0 | 50.5% | 47.3% | 5.5% | 6.6% |
glaurung | 26.4% | 18.7% | 12.1% | 2.2% |
fission | 24.2% | 19.8% | 7.7% | 2.2% |
angr | 20.9% | 16.5% | 6.6% | 1.1% |
IDA Pro 9.20 | 20.9% | 18.7% | 5.5% | 0.0% |
phoenix | 20.9% | 17.6% | 6.6% | 1.1% |
Binary Ninja | 19.8% | 15.4% | 7.7% | 0.0% |
kuna | 16.5% | 15.4% | 3.3% | 2.2% |
manifold | 13.2% | 12.1% | 2.2% | 3.3% |
Ghidra 12.1 | 12.1% | 9.9% | 5.5% | 0.0% |
r2dec | 11.0% | 11.0% | 1.1% | 0.0% |
dewolf | 2.2% | 2.2% | 0.0% | 0.0% |
Delphos leads on Union, and it does so on the metric that best separates decompilers from each other: types.
Why are types an important metric?
Delphos recovers types perfectly on 12.6% of functions, claude-code on 8.8%, codex on 9.9%, and Ventris on 5.5%. Ghidra 12.1 scores 5.5%, Binary Ninja scores 7.7%, IDA Pro 9.20 scores 5.5%. The Delphos margin over the next LLM agent is 43% in relative terms and more than double the best commercial decompiler on this benchmark.
Types are hard in decompilation because the compiler throws them away. Once C source becomes x86-64 machine code, there is no size_t, no struct stat, no char *. There is a 64-bit register, a stack slot, and a memory access with an 8-byte width. A decompiler recovering types is guessing, using data-flow analysis, calling-convention signals, memory-access widths, and knowledge of common APIs. The state of the art has been poor for a long time. The best commercial decompiler on this benchmark recovers types perfectly on 7 of the 91 functions.
Structure and byte equivalence can both be derived from the binary. Structure is re-computable from the assembly with any control-flow analysis, and bytes can be re-checked by recompiling and diffing. Names and semantic types cannot be derived from the binary at all. They are the information the compiler discards. That is what makes them the hardest axis for a decompiler and the axis most likely to reveal how much prior knowledge a system brings to the problem.
Delphos brings prior knowledge to the problem deliberately. A neural stage that has seen a lot of C source during training proposes identifiers and types. A second stage that has the binary in hand verifies each proposal against the assembly and discards the ones that disagree. Classical decompilers have no prior knowledge to propose from, and their type scores show it. Single-agent LLM entries see only the assembly, so their type guesses draw from one source.
For one case: for read_shadow in base-passwd, the neural stage produces identifiers like _PATH_SHADOW, closeitup, and exitpoint. Those are names from the original source and don't appear in the binary, although the model has seen them during training. The verification stage only keeps the names the assembly supports and drops the rest.
How does Delphos work?
Delphos decompiler consists of four stages. Two run locally on a GPU, and two call an external model.
We disassemble the target function into assembly. An LLM trained on decompilation reads that assembly and proposes a candidate C reconstruction that supplies the identifiers and semantic types the assembly no longer carries. The candidate goes to a Claude Opus agent with the binary in hand and access to standard analysis tools. The agent treats the candidate as a hint, checks each construct against the binary, and discards what fails. The prompt body is the same DecBench text used to score claude-code and codex, so the comparison holds at the prompt layer.
Selection. The agentic stage runs twice. For each function, the candidate is recompiled with the target's toolchain, disassembled, and diffed against the target binary. The candidate with the smallest link-normalized divergence wins. Ties break toward fewer gotos. No ground truth is used in the selection. The binary is the input to the problem, and the selector asks the same question a reverse engineer asks: does my reconstruction rebuild to the same bytes?
Selection is vital. Two runs of the identical configuration agree on 83% of their structure-perfect functions by Jaccard, and across four runs the Union score ranged from 43 to 48 out of 91. Selection converts that spread into a stable output: random choice among samples scores 45 on average, selection scores 48. The disagreement is stochastic. Sampling one configuration N times works, and prompt variation across configurations adds nothing on top.
The pipeline is the neural first pass supplying prior knowledge, then an agent with the binary in hand deciding what to keep.
What's next
The success on DecBench indicates a decompiler that treats prior knowledge and binary evidence as complementary inputs, and lets each one correct the other. The neural stage recovers what the compiler discards. The verification stage refuses to accept what the binary does not support. The leaderboard rank on types is the first published measurement of that architecture against every other approach the field has produced.
DecBench is a living benchmark. The Noelo Lab plans to add decompilers and refresh the projects as the field moves. We plan to keep our entry current, and we plan to publish the numbers each time they move. This is the problem we built Delphos Labs to work on.
Related reading: DecBench methodology and dataset