How we rate our sources: the Admiralty Code explained

Why a catalogue needs a grading system

Every entry in this catalogue rests on evidence: a press release, a vendor case study, a trade-press article, a regulator's filing, or sometimes just a marketing landing page. Not all of that evidence is equal. A claim that an insurer deployed a particular AI system carries very different weight depending on who says it and how well it is corroborated. To make that judgement explicit rather than leaving it buried in an editor's head, we grade every source with the Admiralty Code — the same two-axis system that intelligence analysts have used for the better part of a century.

Origin and history

The method is often called the Admiralty Code, the NATO System, or the 6×6 system. Its lineage runs through British naval intelligence and into NATO doctrine. Today it is codified for allied forces in NATO STANAG 2511 (which superseded the older STANAG 2022) and expressed in the allied joint doctrine publication AJP-2.1 [1]. The United States Army carries an almost identical matrix in Field Manual FM 2-22.3, Human Intelligence Collector Operations (2006), Appendix B, where it appears as the Source and Information Reliability Matrix [2].

The core idea is elegantly simple. Two separate questions are asked about every piece of information, and each gets its own grade: how reliable is the source (a letter, A to F), and how credible is this specific piece of information (a number, 1 to 6). A report is then tagged with a combined code such as B2 or C3.

The two scales

Reliability of the source (A–F):

Credibility of the information (1–6):

The crucial point: the two axes are independent

The letter describes the track record of the source; the number describes this particular claim. In principle these vary independently. A completely reliable source (A) can pass along a single, uncorroborated rumour (so the item might be A3 or even A6). Conversely, a source you have never used before and cannot vet (F) might report something that three other independent outlets already confirm — yielding F1. The grade F1 is the textbook illustration that the two dimensions really are separate: we cannot judge the messenger, but the message is independently confirmed.

How to assign each grade in practice

In insurance-AI catalogue work the mapping looks roughly like this.

Reliability (the letter) — who is speaking? An official press release or annual report from the insurer itself is primary and accountable; we rarely award a clean A because even first-party communications are promotional, so usually reliable (B) is the realistic ceiling. Established trade press and recognised analysts are B/C. A vendor case study describing its own deployment has an obvious incentive to flatter the result, so it is C/D unless the named insurer co-signs it. Anonymous, promotional or thin material is D/E, and genuinely unjudgeable origins are F.

Credibility (the number) — is this specific claim true? We reserve 1 for claims corroborated by independent domains: the insurer's press release plus trade-press coverage plus, ideally, a third party — not three pages that all trace back to the same announcement. A credible-but-single-sourced claim is 2 or 3. The key rule for credibility 1 is independence: five copies of the same syndicated announcement are one source, not five.

The documented weaknesses

We adopt the Admiralty Code with open eyes, because its limitations are well documented. The earliest empirical critique, Baker, McKendry and Mace (1968), found that the great majority of ratings fell on the diagonal — A1, B2, C3 — implying raters were not separating the two axes in practice [3]. Samet (1975) similarly found that most intelligence officers could not treat the scales independently, and that judgements were driven far more by the credibility rating than the reliability rating [4]. On subjectivity, Joseph and Corkill (2011) observed that evaluation was often cursory and inconsistently applied [5]; Irwin and Mandel (2019) argued that systems like the Admiralty Code tend to mask rather than discipline subjectivity, and recommended numeric probability estimates [6][7]. Recent experimental work confirms that source-reliability cues measurably bias judgements of information quality even when they logically should not [8]. Finally, because "cannot be judged" (F/6) is always defensible, raters reach for it too readily.

How this catalogue applies it

With those caveats in mind, we use the Admiralty Code as a disciplined disclosure tool, not a false-precision generator. Every catalogue source carries an A–F + 1–6 code, visible to readers. Independent corroboration across separate domains moves the credibility number toward 1. And we never auto-assign the top human-verified confidence: a machine pipeline can detect corroboration and propose a grade, but the highest "verified" tier is a human-only decision — no algorithm is allowed to award itself an A1. The result is honest rather than flattering: a vendor case study with no outside confirmation might read C3; an insurer announcement echoed across independent trade press becomes B1; a slick but unattributed landing page might be E4 or F6.

References

• [1] NATO STANAG 2511 / Allied Joint Doctrine for Intelligence Procedures (AJP-2.1) — source reliability (A–F) and information credibility (1–6) scales. ETURWG, George Mason University.
• [2] U.S. Department of the Army (2006). FM 2-22.3, Human Intelligence Collector Operations, Appendix B. PDF.
• [3] Baker, J. D., McKendry, J. M., & Mace, D. J. (1968). Certitude Judgments in an Operational Environment. U.S. Army. DTIC.
• [4] Samet, M. G. (1975). Subjective Interpretation of Reliability and Accuracy Scales for Evaluating Military Intelligence. U.S. Army Research Institute. PDF.
• [5] Joseph, J., & Corkill, J. (2011). Information evaluation: how one group of intelligence analysts go about the task. DOI.
• [6] Irwin, D., & Mandel, D. R. (2019). Improving information evaluation for intelligence production. Intelligence and National Security, 34(4), 503–525. DOI.
• [7] Irwin, D., & Mandel, D. R. (2019). Standards for Evaluating Source Reliability and Information Credibility in Intelligence Production. SSRN.
• [8] Kelly, M. O., Budescu, D. V., Dhami, M. K., & Mandel, D. R. (2025). The effect of source reliability and information credibility on judgments of information quality. Judgment and Decision Making, 20, e7. DOI.