Measuring Blind Spots in Intelligence Collection

An intelligence collection programme is defined as much by its absences as by its holdings. Yet almost every metric in routine use — sources onboarded, records ingested, indicators produced, uptime — measures the holdings. None of them measures the absences, and the absences are where the surprises live.

This is a note on how to measure the part you cannot see, which is harder than it sounds and more tractable than it is usually treated.

The circular denominator

Start with the failure that makes every other measurement untrustworthy.

Suppose a feed is expected to deliver hourly. You want to know your coverage, so you compute it: take the observations you received, find the span from first to last, divide the span by the cadence, and compare. The result is 100%.

It will be 100% forever. The denominator was derived from the numerator’s own source. If six hours are missing in the middle, the first and last observations still bound a span, and the arithmetic still divides evenly. If the feed dies entirely and delivers one record, the span is zero and the coverage is — by that formula — complete.

A denominator derived from the measured data’s own span cannot detect the data that never arrived.

The fix is structural rather than arithmetic. The expected set must be declared externally — by a schedule, a contract, a manifest — and the observed set compared against it. The declaration has to come from somewhere the observation cannot influence. That is the entire principle, and almost every coverage metric I have examined violates it in one form or another.

Two practical consequences follow immediately:

Slot the expectation, do not just count it. Divide the window into intervals and ask which intervals contain an observation. Counting observations against expected-count lets two observations in one hour silently cover for an hour with none. Use half-open intervals [start, end) — an observation at exactly the boundary belongs to one slot, not both and not neither. This detail is not pedantry; a closed interval will report a slot as missing when the observation landed one second into the next one.

Cap coverage at 100% and raise an anomaly instead. If a slot contains three observations, coverage is not 300%. It is one covered slot plus an anomaly worth a different name — slot overrun, say — which is not the same thing as a duplicate record and should not be aggregated with one.

Three shapes of blindness

Blind spots are not homogeneous. They fail differently and need different instruments.

Structural blindness — what the source population cannot contain

Every source systematically excludes something, and the exclusion is a property of the source rather than of the world.

Vendor reporting over-represents adversaries who attacked vendor customers. Sandbox corpora over-represent samples someone chose to submit. Leak sites show only victims who did not quietly pay. Certificate transparency shows only certificates from participating CAs. Each of these is obvious once stated, and each of them silently converts into a claim about reality the moment someone computes a frequency from it.

The instrument here is not a metric, it is a document: for each source, a written record of what it covers, what it structurally cannot cover, and its known biases. Its value is that it lets an analyst say we do not look there instead of it did not happen — which are different sentences that produce the same number.

Operational blindness — what the pipeline stopped collecting

A source that dies looks, downstream, exactly like a source with nothing to report. Both produce no new records. One is a failure and one is a fact about the world, and the pipeline as normally built cannot distinguish them.

The instrument is a positive liveness record: every collection run emits a statement that it ran, what it retrieved, and that zero may be a legitimate result. This inverts the default. Absence of a record then means the collector did not run, which is a detectable condition, rather than nothing happened, which is not.

Two properties make this work, and both are routinely got wrong:

• The heartbeat must come from the work, not from the scheduler. A timestamp written by the thing that launched the job proves the launcher ran. Only a record written at the point of completion proves the work did.
• The timestamp must belong to the same observation as the number beside it. A record whose event time is when it was written rather than when the work finished lets a system look fresh by back-filling. I have measured a gap of nearly twenty hours between those two clocks on a single record. If freshness is computed from the wrong one, a lane can improve its liveness score without doing any work at all.

Analytical blindness — what the instrument declined to judge

A classifier that labels what it recognises and skips what it does not will report excellent accuracy on a shrinking fraction of the problem. If it examines 38% of a corpus and is right about 91% of that, it is not 91% accurate. It is 91% accurate on roughly a third of the question and silent on the rest — and silence is read as success by every downstream consumer.

Worse, the metric contains a gradient: the score improves by examining less. Any process optimising against it will find that gradient, and nothing in the measurement will object.

The instrument is a three-part result, always reported together:

examined      what was actually looked at
findings      what was concluded, with confidence
not_examined  what could not be assessed, and why

The third field is what makes the first two trustworthy. Without it, coverage and accuracy are indistinguishable in the output. The assessment side of this is implemented in cti-confidence, where unable to assess is a first-class state that refuses to carry a confidence value.

Why the gauges stay green

Blind-spot measurement fails in a specific and recognisable way: the instrument reports a correct number that answers a different question from the one being asked. Three shapes account for most of it.

The filter that excludes broken rows from its own denominator. WHERE status = 'ok', IS NOT NULL, FILTER (WHERE valid). The query is correct. It reports the health of the records that arrived healthy, and says nothing about the rest — but it is displayed under a label like data quality, which is a claim about everything.

The tool default that hides the population. A search wrapper that respects .gitignore, a query with an implicit LIMIT, an API with a default page size. The command returns a number, the number is true of what the tool looked at, and what the tool looked at is not what the operator believes. I have seen the same search return 81 results through one wrapper and 527 through another — 84.6% of the population invisible, with no error, no warning, and a perfectly plausible number on screen.

The correct zero bound to the wrong question. Someone asks “how many mismatches?” and gets 0. The zero is accurate. It counted mismatches among records that were compared, and most records were never compared. This one is the most dangerous of the three, because a zero attracts praise rather than scrutiny, and it travels — into summaries, into reports, into decisions — with the caveat stripped somewhere around the second retelling.

The common structure: the counting rule was never stated next to the count. Which is the whole remedy, unglamorous as it is. State the rule with the number, every time, in the same sentence. A number whose counting rule cannot be stated is not yet a measurement.

A minimum programme

Six things, in the order I would add them:

1. An external schedule per source. Declared by the source owner, not inferred from the source’s own output.
2. Slot-based coverage against that schedule. Half-open intervals, capped at 100%, with overrun reported separately.
3. Positive liveness records. Emitted at work completion, carrying the completion time, not the write time.
4. A written source-limitation register. What each source structurally cannot see.
5. Three-part results on every analytical component. Examined, findings, not-examined.
6. A proven-red list. For each gate and validator, the date it was last observed to reject something. A control never seen to fail is an untested claim wearing the uniform of a passed test.

Item six is the one most likely to be skipped and the cheapest to do. Inject a known-bad input and confirm the gate goes red. It takes an afternoon, and until it is done, every green light in the system is of unknown value — not because the gates are probably broken, but because working and not running produce identical output, and you have no observation that separates them.

Status

This is a working framework, refined against practice rather than a finished publication. The confidence and attribution dimensions are developed further in Articles; the end-to-end pipeline this sits inside is described in From OSINT Collection to Actionable Cyber Threat Intelligence.