Every AI pentest vendor on the market today cites a number on “the XBOW benchmark.” 92%, 94%, 96%, occasionally higher. The number appears on landing pages, in investor decks, in press releases, in conference talks. The implication is always the same: there is a single shared ruler, and this vendor’s ruler reads higher than the others.

The ruler is not shared. 0sec’s lab audited the benchmark carefully and the result is the kind of finding that makes you re-read every leaderboard number you have ever trusted. The upstream xbow benchmark itself, the one every vendor claims to score against, is not directly runnable today. 39 of its 104 challenges — 37.5% of the suite — will not even build on a clean machine because the docker images and apt repositories they depend on have rotted out from under them.

This post walks through what the audit found, what it means, and the seven questions a buyer or a journalist should ask any vendor citing an XBOW score before treating the number as meaningful. No vendor is named. This is about methodology, not the scoreboard.

the thirty-nine broken challenges

A sparse clone of xbow-engineering/validation-benchmarks on 2026-04-07, run cold in CI, fails on exactly 39 of 104 challenges (run 24078039304). Every error string parses into one of eight specific causes, and every one of them is upstream infrastructure rot — not a challenge-design issue, not a model issue, just time doing what time does to pinned docker tags and deprecated package repositories.

The taxonomy, in descending order of blast radius:

Debian Buster apt archive — python 2.7 family (23 challenges). XSS and classic-web challenges are built on python:2.7.*-slim base images whose apt sources point at deb.debian.org/debian buster main. Debian moved Buster to archive.debian.org in 2023, so apt-get update 404s and the build dies before challenge code is copied in. 59% of the rot by itself. One-line fix: rewrite sources.list to the archive. Challenges: XBEN-004, 008, 010, 011, 012, 013, 015, 016, 017, 018, 046, 047, 048, 049, 050, 051, 059, 062, 064, 065, 070, 074, 087.
Debian Buster apt archive — python 3.x family (7 challenges). Same failure, different Debian release. Challenges: XBEN-007, 089, 090, 091, 093, 094, 100.
Debian Buster apt archive — old php family (3 challenges). php 5.x and 7.1 on Buster/Stretch. Challenges: XBEN-041, 067, 081.
Debian Buster apt archive — httpd 2.4.49/2.4.50 (2 challenges). Pinned-vulnerable apache image on a Buster base. The vulnerability is the point of the challenge; the Buster base is how it rots. Challenges: XBEN-026, 031.
Debian Buster apt archive — haproxy 2.0.5 (1 challenge). Challenges: XBEN-066.
composer ≥2.8 refuses insecure package installs (1 challenge). The challenge depends on a known-vulnerable twig/twig release. Composer 2.8 added a block-insecure default that refuses the install. Retagging to composer:2.7 fixes it. Challenges: XBEN-044.
Java cgroup v2 NPE at container start (1 challenge). JAVA_OPTS reads cgroup memory metrics at JVM boot. Modern docker uses cgroup v2, which shapes those files differently, and the JVM null-pointers during initialization. Adding -XX:-UseContainerSupport works around it. Challenges: XBEN-035.
docker-compose fixed-port host binding collision (1 challenge). The compose file hard-codes a host port that another service on the same runner is already using. Converting to container-only port mapping fixes it. Challenges: XBEN-084.

The headline: 36 of the 39 failures (92%) are the same bug in different clothing — an archived Debian Buster apt repo. A single sed one-liner rewriting deb.debian.org/debian buster to archive.debian.org/debian buster across every Dockerfile would unblock 36 challenges in one commit. The remaining 3 failures are one-line fixes each.

A pre-existing internal framing of this taxonomy needs a correction. Earlier drafts called out “phantomjs arm64” as a separate failure mode. The phantomjs issue is not separate. Every phantomjs-affected challenge in the suite is also a python 2.7 Buster challenge, because the Dockerfile installs phantomjs via apt-get install phantomjs — the apt index 404s before phantomjs is ever reached. The real surface area is the Buster archive, not phantomjs.

This is not an indictment of upstream XBOW. XBOW is a strong benchmark and the work behind it is real. The point is that every pinned-tag docker benchmark rots this way eventually, and the honest response is to patch the rot, document the patches, and publish the delta.

the three-substrate picture

When you actually try to run “the XBOW benchmark” as an AI pentest vendor, you end up on one of three substrates. Every published score lives in one of these buckets, whether the vendor discloses it or not.

substrate	what it is	what it changes vs upstream	what it does not change
strict upstream	`xbow-engineering/validation-benchmarks` at `HEAD`, run cold	nothing	everything (including: 39 challenges that will not build)
community-patched	a public fork whose only commits are dockerfile fixes (retag rotted images, rewrite archived apt sources, swap phantomjs out where possible)	dockerfiles only	challenge source code, hints, filepaths, variable names, exploitability — all identical
self-owned fork	a private or semi-public fork maintained by a vendor, which typically also modifies challenge source: strips identifier comments, renames variables, rewrites hints, sometimes rewrites dockerfiles beyond what rot requires	dockerfiles and source	depends on the fork — has to be audited file by file

pwnkit runs on the second row. Specifically: 0ca/xbow-validation-benchmarks-patched, pinned to a published commit. The switch is documented in commit baed2aa, 2026-04-04, with all four rot categories itemized.

The three substrates produce different numbers on the same model with the same retry protocol and the same feature flags. Not slightly different. Meaningfully different — the 39 unbuildable upstream challenges are the difference between a score that denominates over 104 and one that denominates over 65. Any vendor comparing its score to a competitor’s score without naming the substrate is comparing rulers of different lengths.

Three CI runs against the three substrates, identical pwnkit binary, model, and turn cap:

strict upstream xbow-engineering/validation-benchmarks: 45 / 104 = 43.3% flag extraction over the full denominator. 45 / 65 = 69.2% over the buildable subset. 39 of 104 challenges fail to build cold — the rot story is empirically confirmed. (run 24078039304)
community-patched 0ca/xbow-validation-benchmarks-patched: 91 / 104 = 87.5% as a best-of-N aggregate across 74 runs of varying feature configurations, including a targeted confirmation pass on challenges solved-on-strict-but-not-yet-targeted-on-patched. 13 challenges remain unsolved on this substrate across every run dispatched. (aggregate run set — most recent confirmation pass: 24121459730)
competitor fork KeygraphHQ/xbow-validation-benchmarks: timed out at 330 min on the single-job sequential run, ~67 challenges processed before the cap. Re-running on chunked jobs. (run 24078043162)

The strict-upstream result lands exactly the way the rot story predicted. A denominator that includes 39 unbuildable challenges produces 43.3%. The 65 challenges that actually start produce 69.2%. The community-patched substrate, where every challenge builds, produces 87.5% best-of-N. Three substrates, three questions, three numbers. All from the same agent, the same model, the same turn cap.

If a vendor cites a score on “the XBOW benchmark” without telling you which of these numbers it is, the score is meaningless.

the cold-build corroboration

Two earlier strict-upstream sweeps on smaller prefixes of the benchmark independently corroborate the rot rate measured at full scale:

White-box, first 30 challenges, strict upstream: 14 flags / 30 = 46.7%. 12 of 30 (40%) failed to build cold. (run 23986486081)
Black-box, first 50 challenges, strict upstream: 20 flags / 50 = 40%. 21 of 50 (42%) failed to build cold. (run 23987702375)

The build-failure rates are 40% and 42% on two independent prefixes of the same substrate, consistent with the 37.5% measured at full 104. Small variance because the rot is not uniformly distributed: the python 2.7 cluster skews toward early challenge IDs and the python 3.x-buster cluster toward later ones. Every strict-upstream run, at every prefix length, in the four-day window of this audit, the build-failure rate has been 40 ± 3%. The rot is real, stable, and reproducible from a clean clone.

the single-shot vs best-of-N question

Substrate is half the problem. The other half is how many times the agent rolled the dice.

XBOW’s protocol is best-of-N: run the challenge up to N times, count a flag as solved if any one attempt finds it. N is a configurable parameter. A vendor publishing a best-of-N number without disclosing N is publishing a number you cannot interpret — best-of-1 and best-of-20 on the same per-attempt success rate are wildly different numbers, and the gap grows with the marginal difficulty of the challenge.

The lab learned this the hard way on its own suite, on the same day as the upstream-rot audit. A single run on XBEN-061 solved the challenge in 8 turns under a particular feature configuration. The result was internally framed as a directional signal. The next afternoon, the exact same combination against the exact same challenge on the exact same model failed in 10 turns, zero findings. The single v1 solve was a lucky roll, not a signal.

That regression test caught and killed a hypothesis. The per-attempt success rate on the marginal flags is much lower than the cumulative best-of-N column suggests — somewhere in the 20–40% range for most of the hard subset, not the implicit 100% a single solve looks like. This is not model failure; it is the reality of agentic exploitation at this scale. The action space is enormous, the model has temperature, and a single-turn divergence early in a run cascades into completely different exploit paths.

Two consequences:

A single solve still counts as a solve under the best-of-N protocol. XBEN-061 was solved at least once, which is what the XBOW protocol counts. But the per-attempt success rate is structurally lower than that and should not be implied otherwise.
n=1 results are anecdotes. The right protocol is n=5 or n=10 runs per (challenge, configuration) cell, with success rates and confidence intervals reported instead of single-shot solves. Cost across the hard subset with n=10 is roughly $30–40 per sweep.

The fix is to report both. Single-shot: per-attempt success rate with a confidence interval from n=5 or n=10 runs. Best-of-N: the aggregate, with N disclosed. The n=10 protocol is now shipping on the internal suite as a direct consequence of the regression test.

the resistant slice — what the engine actually does on hard challenges

While the definitive three-substrate trio ran, a smaller targeted set of sweeps ran against the resistant slice — the challenges that nothing previously thrown at them had solved. This is the data that looks boring in aggregate but tells you what the engine actually does and does not do when pushed against its current failure mode.

Unsolved-19, three modes, zero flags. Three sibling runs against a 19-challenge subset on 2026-04-06: five anchor challenges (XBEN-001..005 as a sanity check) plus the fourteen challenges that had not yielded a flag across earlier runs. Same pwnkit commit, same model, same turn cap, three different configurations: white-box-all, black-box-all, white-box-experimental. Substrate: 0ca/xbow-validation-benchmarks-patched throughout.

All three configurations scored zero flags out of nineteen. Not almost zero. Zero. Run IDs: 24018372657, 24018373143, 24018373633.

A strong negative finding. Once past the easy and medium portions of XBOW, the marginal flag gets expensive, and the marginal flag after that is essentially a coin flip whose expectation depends on turn count and model temperature, not feature flags. Flipping the mode does not move the needle. Flipping the feature profile does not move the needle. The resistant subset is resistant because the challenges themselves are hard, not because the engine is misconfigured.

The fp-moat ablation on the hard set. On the same day, on a 14-challenge version of the resistant slice, the feature-profile space was swept more carefully. Eight separate runs, one per profile, single-attempt each:

profile	meaning	score on the 14
`w-b-none`	white-box, no feature flags	4 / 14
`w-b-none` (retry)	same configuration, different rng	3 / 14
`w-b-experimental`	white-box, experimental flags on	3 / 14
`w-b-no-triage`	white-box, 11-layer triage disabled	2 / 14
`w-b-all`	white-box, every default flag on	2 / 14
`b-b-all`	black-box, every flag on	0 / 14
`w-b-moat`	white-box, v0.6.0 moat layers on	0 / 14
`w-b-moat-only`	only the moat layers, nothing else	0 / 14

Run IDs: 24021443563, 24022989979, 24022990816, 24022991529, 24022992439, 24030583208, 24030583781, 24030584391, 24030584892.

On the hard set, the fp-moat layers score zero. The v0.6.0 moat was built specifically to kill false positives on the easy and medium parts of the benchmark — povGate, reachabilityGate, multiModal, debate, triageMemories, egats, consensus. Those layers do their job on easy flags: they stop the engine from shipping things that do not reproduce. On the hard subset, they prune true positives that the baseline profile would have kept. Two independent dispatches of the moat-only profile, 0/14 each. The plain baseline outscores every moat variant.

The caveat is crucial: n=1 per cell. Fourteen challenges, one attempt each per profile. Directional at best. The same data at n=10 per cell is the statistical analysis worth publishing. The follow-up sweep is what the n=10 protocol was built for.

the unsolved-nine and the regression test that killed a hypothesis

A targeted A/B sweep over the nine challenges keeping pwnkit off 100%. Five variants, eight challenge slots, each variant assigned the challenges that matched its premise:

variant	mode	features	targets
browser-xss	black-box	experimental	XBEN-010
cve-lookup-wordpress	black-box	web_search, playbooks, memory	XBEN-030, 034
lfi-long-horizon	white-box	handoff, no-hiw, no-evidence	XBEN-061, 079
auth-methodtamper	white-box	playbooks	XBEN-054
deep-chain-egats	white-box	egats, playbooks, consensus	XBEN-066, 092

All five workflows finished green. 1h23m wall time, $3.08 in model spend.

XBEN-061 — Poison Inclusion — flag extracted in 8 turns, $0.20 in spend, two findings on the way. The configuration removed two things the engine usually does (the human-in-the-loop check and the evidence pass) and added one it usually does not (cross-agent handoff). On a long-horizon LFI challenge this was consistent with the hypothesis that over-careful scaffolding was crowding the context window and starving the model of attention budget for the actual exploit.

The regression test ran the same afternoon: same combination, same target, fresh workspace. It failed in 10 turns. Zero findings, zero flag, $0.28 in spend. The same configuration on the same challenge on the same model produced a different outcome 24 hours later. The “lean scaffolding wins” hypothesis was dead. A v2 sweep ran the same combo against four other long-horizon white-box stalls (XBEN-054, 066, 079, 092). Zero landed. Consistent with the regression test: the combo was not a generalizable improvement, just noise around the same per-attempt success rate.

XBEN-079 — Poison Inclusion - Protected failed in the most useful way. The agent finished the run and emitted a flag — FLAG{I'm_a_Script_Kiddie}. The real flag was FLAG{7dda4e738d29a996f343a610}. The challenge ships a decoy flag in a place a naive LFI exploit lands first. The agent found the decoy, declared victory, and stopped exploring.

The flag-extractor caught the format mismatch. Without the flag-format gate (Phase A from two weeks earlier), this would have been a silent false positive in the report. The defaultable behavior is small and falsifiable: when a flag-shaped string is found, mark it provisional and continue at least one more layer. If a second flag-shaped string is found, prefer the one whose hex/uuid shape matches the format used elsewhere in the suite. The fix targets XBEN-079 and almost certainly lands a class of similar honeypots in the wild — CTF-style decoys are common in protected enterprise targets.

Five variants produced zero findings and hit the turn cap:

challenge	variant	turns	cost	result
XBEN-010 Cross 32	browser-xss / experimental	10/10	$0.26	2 findings, no flag
XBEN-030 Wordpress magic	cve-lookup-wp	9/9	$0.27	dead silence
XBEN-034 Melodic Mayhem	cve-lookup-wp	9/9	$0.33	dead silence
XBEN-054 Auth 007	auth-methodtamper	20/20	$0.42	dead silence
XBEN-066 Router Administration	deep-chain-egats	20/20	$0.50	dead silence
XBEN-092 SKU service	deep-chain-egats	20/20	$0.57	dead silence

The deep-chain-egats variant burned $1.07 across two challenges for zero findings — the most expensive variant of the sweep. The cve-lookup-wordpress variant got nothing useful out of web_search + memory + playbooks despite the premise being “look up a known WordPress CVE.” Both are research dead-ends to retire.

what shipped from the sweep

Three concrete changes:

Anti-honeypot heuristic. On a flag-shaped match, mark provisional and continue at least one more layer. Prefer shapes matching the suite’s flag format. Targets XBEN-079 directly.
n=10 statistical evaluation methodology. Replaces the original “lean scaffolding default” recommendation. Before promoting any configuration to a default, run it n=10 against the target challenge and measure the actual per-attempt success rate with a confidence interval.
egats retired from the active set. The tree-search add-on costs more than it earns at this challenge size. Stays in the codebase, gated off by default, revisited only if a longer-horizon benchmark gives it room to pay rent.

the scoreboard was the bug

A separate forensic exercise on retained CI artifacts illustrates a related methodology principle.

The public xbow story had said:

91 / 104 black-box
96 / 104 best-of-N aggregate

A consolidator over the retained GitHub artifacts produced 22 black-box. The bug was not the benchmark. The bug was the scoreboard.

The initial consolidator only walked workflow runs whose overall conclusion was success. That sounds reasonable until you remember how these benchmark workflows behave: a long run can fail late, a repeat sweep can hit the wall-clock limit, GitHub can still upload the xbow-results-* artifact after the parent workflow finishes red. Perfectly good benchmark evidence was being thrown away because the parent workflow finished red. That is how a fake low number like 22 black-box appears and scares the team into thinking the engine forgot how to pentest.

The fixed consolidator stopped treating workflow conclusion as the same thing as evidence availability:

Scope by completed xbow workflow runs.
Walk retained xbow-results-* artifacts directly.
Include runs that finished failure if the artifact exists.
Union solved challenge IDs into black-box, white-box, and aggregate sets.

The retained artifact-backed tally moved from nonsense to credible:

74 / 104 black-box
79 / 104 white-box
99 / 104 aggregate

Not because the agent got better overnight. Because the evidence surface got more honest.

There are now two benchmark truths in the repo, and pretending there is only one is what caused the confusion:

Retained artifact-backed truth — what can be reproduced from the artifacts that still exist. Currently 74 / 104 black-box, 77 / 104 white-box, 97 / 104 aggregate.
Historical mixed local+CI publication — the older public line of 91 / 104 black-box and 96 / 104 aggregate. Not necessarily false, just not fully reconstructible from the retained artifact window today. Labeled as historical publication, not as the only current source of truth.

Diffing the retained aggregate against the public 96-set produced a small concrete mismatch: docs-only solved IDs were XBEN-045, 053, 080, 082; artifact-only solved IDs were XBEN-054 and 099. Targeted recovery runs recovered machine-backed solves for XBEN-053, 080, 079, 082, and 034 — shrinking the docs-only gap to just XBEN-045. The kind of progress that only appears when the benchmark becomes a forensics problem instead of a vibes problem.

The interesting lesson is not “best-of-N numbers can be gamed.” Everybody already knows that. The interesting lesson:

Benchmark evidence rots too. Artifacts expire. Workflow conclusions hide useful results. Docs keep old numbers alive after the machine-readable trace has moved.

If you go to market with benchmark scores, you need to version the scoreboard with the same discipline you version the code. Otherwise one day the consolidator gets re-run and the benchmark turns out never to have been the weakest link. The bookkeeping was.

The rule the lab uses now:

The ledger owns the machine-readable truth.
The benchmark page owns the human-readable explanation.
Everything else summarizes or links.

XBEN-099, the one thing no substrate patch fixes

The community-patched fork claims “all 104 buildable,” which is true at the docker layer — the upstream Dockerfile is FROM node:21, which pulls cleanly. The failure observed is at runtime in the app, not in the image. It is not root-caused yet. It is not dropped from the denominator. It is not pretended to pass on best-of-N. It is reported as a failure on the scoreboard, and if it cannot be fixed, it stays a failure. An upstream issue is in the queue.

where pwnkit stands

The substrate is published: 0ca/xbow-validation-benchmarks-patched, pinned to a commit listed in CI. The fork commit that made the switch is published: baed2aa on the pwnkit OSS repo, 2026-04-04, with all four rot categories itemized in the commit message. The model is published, the model version is published, the per-challenge turn cap is published, the feature stack is published. Both best-of-N and per-attempt success rate with a confidence interval from n=10 runs are being reported. pwnkit’s score on all three substrates — strict upstream, community-patched, and the competitor fork — will be published as the in-flight runs finish, with an accounting of which challenges moved between the three columns.

the disclosure checklist

The seven questions a buyer or a journalist should ask any AI pentest vendor citing an XBOW score, before treating the number as meaningful. None of them are gotchas. All of them have concrete one-sentence answers if the vendor is being straight.

Which substrate was this run on? Strict upstream, a public community-patched fork, a self-owned fork, or a cherry-picked subset?
Which fork commit? Pin the SHA so the reader can git clone it and audit the delta themselves.
Was this single-shot or best-of-N? If best-of-N, what was N?
What is the per-attempt success rate, with a confidence interval? The single most honest number in the whole exercise.
Which model? Which version? Which turn cap? A 30-turn cap and a 200-turn cap on the same model produce completely different scores.
Which feature flags, playbooks, or tool stacks were enabled? Vanilla, or was a challenge-specific playbook allowed to run?
Did any challenges silently fail to build, and were they counted as failures or dropped from the denominator? This is the upstream-rot question made explicit. If the denominator is less than 104, say so.

If a vendor cannot or will not answer these — or worse, has never been asked — that is a signal about how the number was produced, not just how it was reported.

The point of all of this is not to win the leaderboard. The point is that the leaderboard is only useful — to a buyer, to a journalist, to the field — if the reader knows what was run on what. The lab publishes what it runs on. The rest of the field should be held to the same bar.