We measured our own hash verdicts, found three defects, and fixed them

Ask a threat intelligence platform how often its verdicts are right and you rarely get a number — you get a confidence score, a methodology page, and a customer list. The accuracy question is almost never answered publicly by anyone in the category: when this platform says a hash is malicious, how often is it, and how often does it say so about software that isn't? So we measured ours. Here is the method, the results, three defects it exposed in our own scoring, and the claims we had to retract along the way.

The hard part isn't measuring. It's not cheating.

Feed an engine known-bad indicators and count the hits, and you learn almost nothing. If you label your test set from a public feed and your engine's evidence includes vendors that ingest that same feed, a "correct" verdict may only prove that two copies of one source agree with each other. The label and the evidence share a lineage — you've measured your own plumbing. The same trap exists on the benign side. So the first requirement was ground truth that never touches the evidence.

Ground truth that doesn't leak

Every benign hash came from publisher attestation — either the vendor's own signed checksum file (Mozilla's SHA256SUMS, Node's SHASUMS256.txt, Wireshark, PuTTY, Debian and Ubuntu ISO checksums) or Authenticode verification, downloading from source and verifying the signing chain to Microsoft Root CA 2011. We did not use NSRL: it confirmed only 3 of 39 current mainstream binaries, skewing heavily toward older Windows software. We also excluded offensive security tooling — flagging Mimikatz or a C2 framework is correct behavior, not a false positive.

Every malicious hash came from a public malware repository, pinned as a fixed corpus with each sample's condition recorded at sampling time — first-seen date, family, detection counts, sandbox verdict — so a future re-run can tell "our engine changed" apart from "the world changed." That pinning is the subject of a correction later in this post.

Why we report two régimes, never one number

Most malware hashes are already known to the major aggregator, so scoring those correctly mostly proves we can read an API response. What matters is the window where coverage hasn't caught up — fresh, targeted, not yet submitted. So every recall figure is reported twice: full (all evidence) and blind (the same case re-scored with the aggregator removed). Two thresholds, too: investigate is a score of 200+ (surfaced to an analyst); actionable is 500+ (sufficient to act on).

The numbers

Precision: across 61 known-good hashes — 39 mainstream binaries and 22 dual-use admin tools — zero false positives at either threshold. Seven mainstream hashes returned no aggregator data (mostly Linux ISOs); we count those as abstentions, not clean passes, so the honest true-negative count is 32. The heaviest real consensus we could source on legitimate software was PuTTY's GUI executable at 3 of 69 engines — a score of 43.7, nowhere near the investigate bar.

That 77% is mostly a fact about the corpus. Of the 27 misses, 21 were coverage — no source had anything yet, and ~15 of those were fresh Linux IoT variants only hours old. Just 6 were scoring calls: four suspicious-grade sandbox results (weighted below malicious-grade) and two under our three-engine floor. The more useful number: of the 98 samples where any source carried a positive signal, 92 reached investigate — 94%.

Sandbox detonation co-occurred on 41% of samples (49 of 119) — the exact same rate, and the exact same 49 hashes, as blind actionable recall. That identity has now held across three independently sampled corpora. With the aggregator removed, sandbox behavior is the only evidence class that reaches actionable on a hash alone. That is the honest scope of our independent contribution, and it points straight at where our next evidence investment belongs.

Mapping our own decision boundary, with zero labels

Sampling can only measure cases you can find — our benign set topped out at 4.3% detection consensus, a population incapable of producing a false positive. So we stopped sampling. We took a real known-good binary with its real evidence (ProcDump, genuinely 0 of 70), held every attribute fixed, and varied only the detection count through the same code path live enrichment uses. This maps the boundary directly, with no labeled data at all.

Three defects fell out of it — all invisible to sampling, because the populations that trigger them don't naturally occur.

Defect 1 — the threshold was not where we thought

Our meaningful-detection gate fires at ≥3 engines and ≥8% of panel; we had assumed the operative threshold was the strong band at 15%. It wasn't — and the boundary tracks ratio, not count (5 of 59 produced the identical cliff at 8.5%). Worse, investigate and actionable were crossed simultaneously: a marginal six-engine detection went straight to actionable, with no investigate-first tier for hashes at all. So we split it — a lone marginal detection (≥8% but not ≥15%, with no second dimension) now resolves to investigate. We priced the recall cost against all 120 malicious samples first: zero. Malware doesn't populate that band; the marginal cases that reached actionable did so on a second dimension the split leaves untouched.

Defect 2 — all resolution was lost above the bar

A marginal six-engine detection scored identically to unanimous agreement across all seventy — wrong under any policy. Six detections at 8% is often correlated generic-heuristic signatures, one or two opinions wearing six hats; seventy is genuine multi-vendor confirmation. A lone strong consensus now ramps by detection ratio instead of pinning flat, with a ceiling deliberately below critical so only a second independent dimension can reach critical.

Defect 3 — confidence was discarding information it already had

Confidence sat flat at 52% across 8.6%, 15.7%, and 100% — even though the underlying source confidence climbs from 0.47 to 0.95 across that range, computed and then thrown away by a fixed cap. Verdict confidence for a lone dimension is now pinned to that source confidence (still below what a two-dimension case can reach), lifting from 55% to 68% in the strong band. It stays flat within the band on purpose: source confidence genuinely saturates there. The score varies because consensus strength varies; confidence doesn't, because certainty about that single source doesn't. All three fixes shipped together against a green calibration gate.

What actually improved

None of this moved recall — that was the point. The fixes correct the shape of the verdict, not the count of hashes we catch: a lone strong consensus used to report the same score and the same confidence whether one band flagged it or all seventy did, and a marginal detection skipped the investigate tier entirely. After the fix, the score and confidence carry the information the evidence already contained, and recall on the pinned corpus is unchanged by design.

The corrections we had to publish about ourselves

Our first measurement runs weren't reproducible, and we didn't notice for a while. The malware corpus was pulled live from a rotating feed, so each run drew a different population — three readings of "the same" metric. The engine was unchanged across all three; only the corpus moved. This isn't a product getting worse, it's an instrument getting honest — and it's why the pinned corpus ships with this post.

We also retracted a claim: an earlier version said the 8–15% band was empty. It isn't — the pinned corpus holds four of 119 there, two of them lone-signal cases. Sparse, not empty. And we found an inconsistency in our own reporting axis: our engine bands on malicious plus suspicious, while an early script counted malicious only. They coincided exactly here (suspicious counts were zero across all 120), but they diverge on adware, PUPs, and packed installers — the population where our precision blind spot lives.

What we still don't know

Our precision figure is a floor, not a measurement — zero false positives across a population that couldn't have produced one. The FPs that worry us live in a population we couldn't source: packed legitimate installers, older dual-use tools before allowlisting, redistributed remote-access software, signed gaming and overclocking drivers. The band split cost nothing but hasn't been shown to catch anything. And our malicious corpus is Linux-IoT-heavy while our sandbox backend is Windows-oriented, so part of the 77% reflects samples our detonation path structurally can't reach. We have no lead-time claim on hashes, and likely never will — for a repository sample, everyone's evidence begins when someone submits the file. Lead time is a question for infrastructure — domains and IPs, where certificate transparency and passive DNS record activity before weaponization — and that's what we're measuring next.

Why publish this

A verdict you can't audit isn't worth much. We ask analysts to defend assessments to auditors, clients, and occasionally counsel; a platform that contributes to those should be able to state what its numbers mean, where its thresholds sit, and where it's blind. "Reproducible" and "explainable" are easy to put on a website — the test is whether the vendor will publish a figure that isn't flattering, and a correction when they get one wrong. The measurement also paid for itself: every defect above was invisible to a 500-case internal suite, and holding real evidence fixed while sweeping one input found all three in an afternoon. Any team can run it against any scoring system — including the one you're currently using.

Method notes: benign ground truth is publisher checksums and Authenticode chains — never a detection aggregator, never NSRL. Malicious ground truth is a public-repository listing, pinned with each sample's condition at sampling. Blind figures remove aggregator evidence and re-score the same case. Both corpora ship with this post. Scoring is deterministic evidence algebra — the same evidence produces the same verdict, with no language model in the judgment path.