Delphos Lantern scores #1 on BinaryAudit backdoor detection: by mean, peak, floor, and reproducibility

Delphos Lantern’s harness with Opus 5.0 outperforms every published configuration on Quesma's independent BinaryAudit backdoor-detection benchmark, that hides backdoors in roughly 40MB stripped binaries of real network infrastructure (web servers, DNS servers, SSH servers, proxies, load balancers) and asks an agent to find them using security research methods, with no access to source. On Claude Opus 5.0, Lantern's mean pass rate of 74% is four points above the strongest configuration on Quesma's public leaderboard; with the best run getting a score of 77%. Lantern on Claude Opus 4.6 also beats benchmark harness, at 61%.

Every one of three identical Lantern runs beats every published configuration on Quesma's leaderboard. Rounds 1, 2, and 3 on Opus 5.0 scored a peak of 77%, a mean of 74%, a floor of 71%. Even the floor is above the strongest published configuration on Quesma's public leaderboard.

Compared against Quesma's public leaderboard, Delphos Lantern beats every published configuration. Lantern on Opus 5.0 at 74% mean (range 71–77% across three runs) is four points above the strongest leaderboard entry. Lantern on Opus 4.6 at 61% is above every leaderboard entry except Opus 5.0 through the benchmark harness (70%). The next-strongest published configurations, gemini-3.1-pro and Opus 4.6 through the benchmark harness, tie at 49%.

If Lantern and the benchmark harness are using the same Claude models then why does Lantern win?

Lantern and Quesma's benchmark harness both run the same Claude models. That makes the harness comparison clean: same model, different harness, different result. Delphos Lantern on Opus 5.0 hits 74% mean; the same model running through Quesma's benchmark harness (as shown on the leaderboard) scores 70%. Delphos Lantern on Opus 4.6 hits 61%; the same model through the benchmark harness scores 49%. The two apples-to-apples comparisons that hold the model fixed and vary only the harness both favor Lantern, by four points on Opus 5.0 and by twelve points on Opus 4.6. Two tasks, caddy-backdoor-detect and pingora-backdoor-detect, ship broken upstream and are excluded from every count in this post.

The model contributes the reasoning. The harness contributes everything else. What the harness has to solve, on a benchmark like BinaryAudit, is the analyst's problem: pick a starting function, follow control flow, decode strings, form a hypothesis about what a function is doing, revise it when the disassembly does not fit, and keep going until either the backdoor surfaces or the binary comes back clean. Each of those steps is a decision about where to look next, and the harness is what turns the model's raw reasoning capacity into a search that actually converges.

Three specific differences show up in the pass rate

Lantern’s first strength is tooling. Lantern runs a suite of static and dynamic analysis tools that surface backdoor behavior the model would otherwise have to infer from raw disassembly. For example, decryption, deobfuscation, and other static analysis tools. This allows the models to have much stronger data from which to work. For example, on the four obfuscated dnsmasq variants, Lantern passes three of them in every one of three runs, including cases where the backdoor's strings are XOR-encoded or built character by character to defeat pattern matching. The benchmark harness's Ghidra workflow does the same reconstruction, but by hand: the model gets the raw output and has to work backward.

The second advantage is anchoring. In a stripped 40MB binary there are thousands of anonymous functions, and the wrong one is a wasted budget. Lantern narrows the model's attention to functions where the backdoor would plausibly live given the target software. The pattern is visible in the family breakdown: dnsmasq with six of seven tasks fully reproducible, dropbear with three of four, and the two clean-binary tasks in sozu both correct in every run. Anchoring is what keeps the search from wandering across the binary and running out the budget before committing a verdict.

The third is persistence. BinaryAudit tasks are set up so the obvious pattern rarely fits. When the first hypothesis about how a backdoor is hidden does not pan out, Lantern forms the next hypothesis and continues. That behavior is what keeps the false-positive rate at 1.7 per round: the harness does not flag a binary just because the first suspicious function turned out to be innocuous, and it does not clear a binary just because the first check came back negative.

The model is fixed. The harness is the difference.

How does Lantern score?

BinaryAudit sets a strict pass rule: on detection tasks the agent must return the correct YES/NO verdict and the start address of the function containing the backdoor. That address requirement is what makes the result usable in a real reverse-engineering workflow.

The floor clears the field by one point; the mean beats it by four points; the peak beats it by seven.

Four wins for Delphos Lantern on BinaryAudit follow.

The peak beats the field. Round 1 scored 24/31 (77%), higher than any published configuration on Quesma's public leaderboard. That is a demonstration of the strength of the harness.

The mean beats the field. 23.0/31 (74%) across three runs sits four points above Opus 5.0 through the benchmark harness (70%), and 25 points above the strongest non-Claude configuration on Quesma's public leaderboard. Every one of the three runs is above the benchmark-harness ceiling.

False positives are low. Wrong-YES verdicts on clean binaries average 1.7 per round (1/2/2 across three rounds). Both clean-binary tasks in the benchmark pass in all three runs. When Lantern says YES, you can act on it.

More than half the benchmark is fully reproducible. 18 of 31 scored tasks pass in every one of three runs; another 2 give a consistent NO verdict every run. That is 20 of 31 tasks, a 65% three-run agreement rate, that a security team can plan against.

Where is Lantern strong?

Two categories carry the strongest results.

The clean-binary tasks, sozu-backdoor-detect-negative and sozu-backdoor-detect-negative2, where the correct answer is NO, pass in every one of three runs. Across the full 31-task set, wrong-YES verdicts average 1.7 per round (1/2/2). False positives are the failure mode security teams pay for most heavily: every hallucinated backdoor costs an analyst hours of triage on a clean binary. A detector that keeps that number low is one whose YES a team can act on.

The obfuscated dnsmasq variants, where the backdoor's strings are XOR-encoded, built character by character, or routed through syscall / posix-spawn instead of obvious library calls, pass in every one of three runs on three of the four variants. Obfuscation is what an attacker does to hide malicious behavior in a binary, and the Lantern harness can read through it.

A detector that misses backdoors is a problem. A detector that hallucinates them is a bigger one, because it wastes the time of the security team it's supposed to help.

Where is Lantern weaker?

The multi-binary tasks, where the agent has to reason about a set of binaries together instead of one at a time, are the hardest category, at 60% pass rate. Two multi-binary tasks (caddy-backdoor-multiple-arch-binaries-detect and sozu-timebomb-multiple-binaries-detect) fail in every run; the remaining eight pass an average of 2.25 times out of three. Reasoning about a coordinated cross-binary attack is strictly harder than catching a single-binary backdoor, and it is the unsatisfied part of the benchmark for all models and harnesses.

The remaining single-binary misses are scattered across the harder obfuscated variants, dnsmasq-syscall-obfuscated and lighttpd-syscall-obfuscated each pass only 1/3 runs, and lighttpd-authentication-harvester-detect passes only 1/3. These are the cases where obfuscation reroutes the backdoor through system calls in ways that make the malicious operation hard to name from disassembly alone.

Why did we do three runs?

The BinaryAudit leaderboard is single-run. Any single-run number carries per-task uncertainty that the number itself hides. Under three-run reporting, Lantern hits 24, 22, and 23 out of 31, a peak of 77%, a floor of 71%, and a mean of 74%. Every one of those numbers is above the strongest published configuration in the field. The three-run agreement rate, how often the harness gives the same verdict on the same binary across three tries, is 65%, meaning two out of every three scored tasks return the same answer every time.

Two out of every three scored tasks give Lantern the same verdict on every run, a floor a security team can plan against.

Why does analyzing compiled software need an agent, not just a model?

Capable models are increasingly common. The Lantern result is Claude Opus 5.0 running through a harness that keeps the analysis anchored on the right functions and gives the model somewhere to keep looking when the first three hypotheses don't pan out. On dnsmasq's obfuscated variants, where the backdoor's strings are XOR-encoded or built character by character, the harness's decryption tooling turns tasks that could easily read as no-verdict into tasks the model can actually confirm.

The model supplies the reasoning. Lantern supplies the persistence and the tools. A verdict becomes a property of the system, not an accident of one execution.

Bring us a binary you think Lantern will miss.

Request Access.

Appendix

Where does Lantern work and where doesn't it, by family?

The pass-rate averages hide a real difference in how the harness handles different targets. Grouping tasks by the software family they belong to, dnsmasq, lighttpd, dropbear, caddy, sozu, and counting only tasks that passed in all three runs makes the picture concrete.

dnsmasq is where the harness is strongest: six of seven tasks solved in every one of three runs, including three of four obfuscated variants. dropbear is the next-strongest single-binary family at three of four fully reproducible. sozu carries the two clean-binary tasks (both 3/3), which drive the low false-positive number. Multi-binary tasks in caddy and sozu are the harder end of the benchmark; reasoning about a coordinated cross-binary attack is strictly harder than catching a single-binary backdoor. The full per-run breakdown is in the appendix.

Per-task record (Lantern on Opus 5.0, three runs)

P is a pass: correct verdict, and on detection tasks the correct function address (or a correct per-binary answer on multi-binary tasks). F is a failed attempt with a committed but wrong verdict. – is a run that produced no usable answer (budget exhaustion or no committed verdict). Only P counts as solved, which is the strict bar behind every count in this post.

Task	Type	Run 1	Run 2	Run 3	Passes
caddy-backdoor-multiple-arch-binaries-detect	multi	F	F	F	0/3
caddy-backdoor-multiple-binaries-detect	multi	P	P	P	3/3
caddy-backdoor-simple-detect	single	P	P	P	3/3
caddy-timebomb-multiple-arch-binaries-detect	multi	F	F	P	1/3
caddy-timebomb-multiple-binaries-detect	multi	P	F	P	2/3
dnsmasq-backdoor-detect	single	P	P	P	3/3
dnsmasq-backdoor-detect-execvp-obfuscated	single	P	P	P	3/3
dnsmasq-backdoor-detect-obfuscated	single	P	P	P	3/3
dnsmasq-backdoor-detect-posix-spawn	single	P	P	P	3/3
dnsmasq-backdoor-detect-posix-spawn-obfuscated	single	P	P	P	3/3
dnsmasq-backdoor-detect-syscall	single	P	P	P	3/3
dnsmasq-backdoor-detect-syscall-obfuscated	single	P	F	F	1/3
dropbear-brokenauth-detect	single	P	P	P	3/3
dropbear-brokenauth-detect-hint	single	P	P	P	3/3
dropbear-brokenauth-detect-nologline	single	P	P	P	3/3
dropbear-brokenauth2-detect	single	F	–	–	0/3
lighttpd-authentication-harvester-detect	single	F	P	F	1/3
lighttpd-backdoor-detect	single	P	P	P	3/3
lighttpd-backdoor-detect-open	single	F	P	P	2/3
lighttpd-backdoor-detect-proc	single	P	P	F	2/3
lighttpd-backdoor-detect-proc-obfuscated	single	P	F	P	2/3
lighttpd-backdoor-detect-syscall	single	P	P	P	3/3
lighttpd-backdoor-detect-syscall-obfuscated	single	P	F	F	1/3
lighttpd-backdoor-multiple-arch-binaries-detect	multi	P	P	P	3/3
lighttpd-backdoor-multiple-binaries-detect	multi	P	P	P	3/3
lighttpd-timebomb-multiple-binaries-detect	multi	F	P	F	1/3
sozu-backdoor-detect-negative	single (clean)	P	P	P	3/3
sozu-backdoor-detect-negative2	single (clean)	P	P	P	3/3
sozu-backdoor-multiple-arch-binaries-detect	multi	P	F	P	2/3
sozu-backdoor-multiple-binaries-detect	multi	P	P	P	3/3
sozu-timebomb-multiple-binaries-detect	multi	F	F	F	0/3
Round total		24/31	22/31	23/31

References

BinaryAudit benchmark repository

BinaryAudit results and leaderboard

Quesma, We hid backdoors in binaries

Harbor framework