AI-Assisted Cyber Intelligence: Instrument, Not Oracle

There are two ways to introduce a language model into an intelligence workflow. You can treat it as an oracle — ask it a question, accept the answer, act. Or you can treat it as an instrument — a device with a known error profile, a calibration procedure, and a documented range outside which its readings are not trusted.

The first is what most deployments do. The second is the only one that produces intelligence, because intelligence is defined by traceable evidence and stated confidence, and an oracle supplies neither.

What follows is about the difference in practice.

The specific danger: fluency decoupled from grounding

A language model’s output quality, as perceived by a reader, is dominated by fluency. Its correctness is not correlated with its fluency in any reliable way. A confident, well-structured, correctly-formatted paragraph of fabrication reads exactly like a confident, well-structured, correctly-formatted paragraph of fact.

Human analysts have the opposite failure mode — when uncertain, we hedge, and our prose degrades in a way readers detect. We have spent our entire professional lives calibrated on that signal. Language models break the calibration, and they break it silently.

For intelligence work this is not a quality problem, it is a category problem. Intelligence requires that every assertion be traceable to a source and carry an assessed confidence. A model that produces assertions with neither, in a register indistinguishable from assertions with both, is not a weak intelligence producer. It is producing a different kind of object that happens to look like the right one.

Where LLMs genuinely help

This is not an argument against the technology. The gains are real and they cluster in recognisable places — specifically, in tasks where the output can be checked against the input.

Translation and multilingual triage. Threat reporting, forum content, and leak-site posts appear in languages no single team covers. Models make previously unreachable material triageable. The grounding property holds: the source text is right there, and a speaker can verify a sample.

Summarisation of documents you possess. Condensing a fifty-page report to its assessments and evidence is extractive. Every claim in the summary should be locatable in the source, which makes the output checkable — mechanically, if you require quoted spans.

Structured extraction. Pulling indicators, TTPs, timelines and named entities out of prose into schema. This is high-value because it is the most labour-intensive step in processing, and it is verifiable because the extracted value either appears in the source or does not.

Classification and routing. Is this report relevant to our PIRs? Which analyst queue? What ATT&CK techniques are implicated? These are judgements with ground truth available, which means they can be measured.

Drafting against an analyst’s outline. The analyst supplies the judgement and the evidence; the model supplies the prose. Authorship of the assessment stays with the human.

Notice what these share. In each, the model operates on material the user already has, and the output can be checked against that material. The moment you ask a model for facts not present in the input — “who is behind this campaign?”, “what is this IP known for?” — you have left the region where it is an instrument and entered the region where it is an oracle, and its training data is not a source you can cite.

Where they fail, predictably

Attribution judgement. Requires weighing evidence quality, considering deception, and assessing source independence. Models produce attribution-shaped text with a confident tone and no underlying evidential reasoning.

Confidence assessment. Ask a model how confident it is and you get a number that correlates with the phrasing of your prompt more than with the evidence. Self-reported confidence from a system with no model of its own error is not a measurement.

Negative findings. “We searched X and found nothing” requires knowing the search was performed and its coverage. A model asked whether something exists will often produce a confident negative from the absence of the thing in its own generation process — which is not a search result at all. This is the single most dangerous failure mode in security automation, because a false negative is silent by construction.

Anything requiring provenance. A model cannot tell you where it learned something. If your pipeline requires provenance — and intelligence pipelines do — the model must be constrained to work only over documents you supply and cite.

Retrieval does not solve grounding; it relocates it

The standard answer to hallucination is retrieval-augmented generation: fetch relevant documents, put them in context, require the answer to come from them. This is the right architecture. It is also routinely deployed in a form that quietly fails.

Retrieval failure is invisible. If the retriever returns nothing relevant, the model still answers. Fluently. Unless the system explicitly detects and surfaces “no relevant source retrieved”, a retrieval miss is indistinguishable from a retrieval hit in the output. Retrieval coverage must be measured separately from answer quality, or you are measuring a pipeline whose most common failure you never see.

Citation is not verification. A model asked to cite will produce citations. Whether the cited passage supports the claim is a separate question, and requires a separate check — ideally a mechanical one that confirms the quoted span exists verbatim in the cited document, and a semantic one that confirms it bears on the claim.

Chunking destroys context. An indicator split from its qualifying sentence — “observed in a sandbox”, “historically associated with” — retrieves as a bare fact and gets reported as a current one. The provenance survived; the meaning did not.

The measurement problem, stated plainly

The question to ask of any AI-assisted intelligence component is not “does it work?” It is: how would we know if it stopped working?

This is harder than it sounds, because AI systems fail gracefully. A degraded model keeps producing well-formatted, plausible output. There is no crash, no stack trace, no error rate spike. The pipeline stays green while the product decays.

A component that cannot fail loudly must be measured externally. Three requirements:

1. A deterministic baseline is mandatory

Before deploying a model for a task, build the dumbest thing that could work — regex, keyword match, rules, a classical classifier — and measure it on the same evaluation set.

Without this you cannot answer the only question that matters: does the model add value, or only cost? Teams routinely deploy an LLM extractor at 82% accuracy and celebrate, having never discovered that thirty lines of regex reached 79% for a fraction of the cost, with deterministic behaviour and complete auditability.

The baseline also serves as a regression detector. When the model is upgraded, re-swapped, or re-prompted, the baseline does not move. Divergence between them is a signal that something changed — and it is the only such signal you get for free.

2. Held-out ground truth, and it must actually be held out

An evaluation set of analyst-labelled examples, never used in prompt development, never shown to the model, re-measured on every change. If the prompt was tuned against the evaluation set, the number it produces is a measure of the tuning, not of the capability.

This contaminates more quietly than it sounds: an engineer who looks at failing examples while iterating on a prompt has, in effect, trained on the test set using themselves as the optimiser.

3. The instrument must report its own coverage

This is the requirement most often missing and the one I would place first.

A measurement that reports only what it examined, without reporting what it could not examine, systematically overstates. A classifier that labels 38% of a corpus and reports accuracy on those 38% is not 91% accurate — it is 91% accurate on a third of the problem and silent on the rest. The silence reads as success.

The correct output of any measurement component is a triple:

examined:      what the instrument actually looked at
findings:      what it concluded, with confidence
not_examined:  what it could not assess, and why

The third field is what makes the first two trustworthy. Without it, coverage and accuracy are indistinguishable, and a pipeline can improve its reported score by looking at less — a gradient every optimising system will find eventually.

The general form of the rule: a denominator derived from the measured data’s own span cannot detect the data that never arrived. If you compute coverage by dividing what you saw by what you saw, you will always report 100%.

Agentic systems: the audit problem gets structurally worse

Multi-agent systems — planning, delegation, tool use, several models in sequence — amplify every issue above, and add one.

Errors compound silently through delegation. Agent A extracts an indicator with a subtle error. Agent B enriches it. Agent C writes an assessment. Nothing in the chain re-examines A’s output against the original source. The error is now three steps from its origin and wearing the authority of an assessment.

Four properties are non-negotiable if such a system is to produce intelligence rather than plausible text:

Provenance survives every hop. Every derived fact carries a reference to the artefact it came from — not a summary of it, a reference to it. If a claim in the final product cannot be traced to a specific span in a specific source document, it must be marked as ungrounded rather than presented alongside grounded claims.

Unknown is a first-class value. Not null, not empty string, not a default that means “fine”. A field that could not be determined must be distinguishable, in the data itself, from a field determined to be absent. Systems that conflate these produce their most confident output precisely where they know least.

Every automated judgement is reversible and inspectable. An analyst must be able to ask why, and receive the inputs, the step, and the alternatives considered — not a post-hoc rationalisation generated by asking a model to explain its own earlier output, which is a fresh generation, not a record.

Gates must be proven capable of failing. A validation step that has never been observed to reject anything is indistinguishable from one that is not running. Before trusting a control, inject a known-bad input and confirm it goes red. This costs an afternoon and is the highest-yield test in the entire pipeline — because a gate that cannot fail provides exactly as much assurance as no gate at all, while consuming the confidence budget of a real one.

The division of labour

A defensible arrangement is narrower than most deployments assume:

The model may: translate, summarise material you hold, extract structure, classify against defined criteria, cluster, draft prose from an analyst’s outline, and surface candidates for human review.

The model may not: assign confidence, make attribution judgements, assert facts not present in provided sources, produce negative findings, or author an assessment.

The dividing line is simple. The model handles volume. The analyst handles judgement. Where a task is volume dressed as judgement, automate it. Where it is judgement dressed as volume — and attribution is exactly this — automating it produces confident nonsense at a rate no review process can absorb.

What this looks like when it works

An AI-assisted CTI pipeline worth trusting has a boring, checkable shape:

• Every extracted fact carries a source reference and a retrieval timestamp.
• Every model output has a deterministic baseline running alongside it, and divergence is logged.
• Coverage is reported separately from accuracy, always, including on dashboards.
• “Unable to assess” is a value the schema can hold and the report can display.
• Confidence comes from a published framework applied by a human, never from the model.
• Validation gates have a documented date on which each was last observed to reject something.
• The final product names its falsifier, and a human signed it.

None of this is exotic. It is ordinary measurement discipline applied to a component that is unusually good at concealing its own failures — and that concealment, not the error rate, is the actual reason the discipline has to be deliberate.

---

สรุปภาษาไทย

มีสองวิธีในการนำโมเดลภาษามาใช้ในงานข่าวกรอง — ใช้เป็น “คำพยากรณ์” (oracle) คือถามแล้วเชื่อแล้วลงมือ หรือใช้เป็น “เครื่องมือวัด” (instrument) คืออุปกรณ์ที่มี โปรไฟล์ความคลาดเคลื่อนที่รู้จัก · วิธีสอบเทียบ · และ ช่วงที่ค่าอ่านไม่น่าเชื่อถือ ซึ่งประกาศไว้ชัด แบบหลังเท่านั้นที่ผลิตข่าวกรองได้

• อันตรายเฉพาะตัว: ความคล่องแคล่วแยกขาดจากการมีหลักฐานรองรับ — ย่อหน้าที่แต่งขึ้น กับย่อหน้าที่เป็นความจริง อ่านแล้วเหมือนกันทุกประการ ⛔ นักวิเคราะห์มนุษย์เวลาไม่มั่นใจ สำนวนจะแย่ลงให้ผู้อ่านจับสัญญาณได้ โมเดลทำลายการสอบเทียบนั้น และทำลายอย่างเงียบ ๆ
• จุดที่ได้ประโยชน์จริง — แปลภาษา · ย่อเอกสารที่เรามีอยู่แล้ว · สกัดข้อมูลเข้าโครงสร้าง · จัดหมวด/คัดแยกคิว · ร่างเนื้อความตามโครงที่นักวิเคราะห์วางไว้ 🔑 ทั้งหมดมีสมบัติร่วมกันคือ ตรวจผลลัพธ์ย้อนกลับกับวัตถุดิบได้
• จุดที่ล้มเหลวอย่างคาดการณ์ได้ — การระบุผู้กระทำ · การให้ระดับความเชื่อมั่น · ข้อค้นพบเชิงลบ (“ค้นแล้วไม่พบ”) · และทุกอย่างที่ต้องการที่มาของหลักฐาน
• RAG ไม่ได้แก้ปัญหา แต่ย้ายที่ตั้งของปัญหา — เมื่อการค้นคืนล้มเหลว โมเดล ยังตอบอยู่ดี และตอบอย่างคล่องแคล่ว ⇒ ต้องวัด ความครอบคลุมของการค้นคืน แยกต่างหากจาก คุณภาพคำตอบ มิฉะนั้นคือการวัดระบบที่เราไม่เคยเห็นความล้มเหลวที่พบบ่อยที่สุดของมัน
• ต้องมีเส้นฐานที่กำหนดผลได้แน่นอน (deterministic baseline) — สร้างของที่โง่ที่สุด ที่พอใช้ได้ (regex/กฎ) แล้ววัดบนชุดเดียวกัน ⛔ ถ้าไม่มี จะตอบคำถามเดียวที่สำคัญไม่ได้เลย — โมเดลเพิ่มคุณค่า หรือเพิ่มแค่ต้นทุน
• 🔑 เครื่องวัดต้องรายงานความครอบคลุมของตัวเองด้วย — ตัวจัดชั้นที่ติดป้ายได้ 38% ของคลังแล้วรายงานความแม่น 91% ⛔ ไม่ใช่แม่น 91% แต่คือแม่น 91% บนหนึ่งในสามของปัญหา และ เงียบ บนส่วนที่เหลือ — ความเงียบนั้นถูกอ่านว่าสำเร็จ ผลลัพธ์ที่ถูกต้องต้องเป็นสามช่อง: ตรวจอะไรไป · สรุปว่าอย่างไร · ตรวจอะไรไม่ได้ และเพราะอะไร
• 🔑 ตัวหารที่ได้มาจากช่วงของข้อมูลที่วัดเอง ตรวจจับข้อมูลที่ไม่เคยมาถึงไม่ได้ — ถ้าคำนวณความครอบคลุมด้วยการเอาสิ่งที่เห็นหารด้วยสิ่งที่เห็น จะได้ 100% ทุกครั้ง
• ระบบหลายตัวแทน (agentic) ทำให้ปัญหาการตรวจสอบย้อนกลับแย่ลงเชิงโครงสร้าง — ความคลาดเคลื่อนสะสมเงียบ ๆ ผ่านการมอบหมายงานต่อกัน ต้องมีสี่อย่าง: ที่มาของหลักฐาน รอดทุกทอด · “ไม่ทราบ” ต้องเป็นค่าชั้นหนึ่ง ⛔ ไม่ใช่ null หรือค่าปริยายที่แปลว่าผ่าน · ทุกคำวินิจฉัยอัตโนมัติต้องย้อนกลับและตรวจสอบได้ · และ ด่านตรวจต้องพิสูจน์ได้ว่าแดงเป็น ⛔ ด่านที่ไม่เคยเห็นว่าปฏิเสธอะไรเลย แยกไม่ออกจากด่านที่ไม่ได้ทำงาน

การแบ่งงาน — โมเดลรับผิดชอบ ปริมาณ · นักวิเคราะห์รับผิดชอบ คำวินิจฉัย งานใดที่เป็นปริมาณซึ่งแต่งตัวมาเหมือนคำวินิจฉัย ให้ทำอัตโนมัติ ส่วนงานใดที่เป็นคำวินิจฉัยซึ่งแต่งตัวมาเหมือนปริมาณ — และการระบุผู้กระทำคือข้อนี้พอดี — การทำอัตโนมัติจะผลิต ความไร้สาระที่ฟังดูมั่นใจ ในอัตราที่ไม่มีกระบวนการทบทวนใดรับไหว