This study pairs a failure analysis of a managed legal RAG deployment with a control, its cost model and its build. §3 and §4.1 are reconstructed from public documentation and framework defaults. §6.1 to §6.6 are the specification as written in May 2026. §6.7 records what was built in October 2026, legal-rag-verifier, and what it measured, including one prediction of §15.4 that came true: the small model built to check each sentence (the auditor) failed.

The central argument (§12.1): behind a managed API, a contaminated answer cannot be partly repaired, so each hallucination event costs a full regeneration.

---

1. Summary

A legal-tech platform runs conversational RAG over 6M+ Italian and EU legal sources. Its generation path, reconstructed from public documentation, uses the common generate-then-validate pattern: the full answer is checked after it is written. A fabricated citation written mid-answer stays in the generation state, the model's working memory of the text so far, and conditions every token after it. The check sees it only at the end.

Published research (Stanford RegLab, Magesh et al., 2025) measured hallucination at 17 to 33% in specialised legal RAG systems. (Confirmed: peer-reviewed, domain-specific, other systems.) Under SRA Principles 2 and 7, the solicitor who relies on the output carries the liability. The Italian Codice Deontologico Forense, Articles 9 and 12, places the same duty on the avvocato.

The control checks each sentence during generation and cuts a rejected sentence out of the generation state before later tokens build on it. That is possible only where the generation state is accessible, which managed APIs do not allow (§12.1).

---

2. Regulatory Context

Warning

The exposure falls on two parties. The platform operator under the EU AI Act, Article 15. The lawyer under their professional rules. One fabricated passage reaching a tribunal exposes both.

---

3. The Failure

3.1 What the User Sees

A lawyer asks about an Italian statutory provision. The answer contains a fabricated holding, a misattributed citation or unsupported commentary, formatted and stated with the same confidence as the rest. Nothing marks it. Relied on in a filing, it is fabricated law presented as authority.

On roughly one query in five, at the Stanford rate, the check catches the error after generation and regenerates the whole answer. The user waits 12 to 25 seconds (By design, modeled) while the system discards the answer, re-processes every input token and starts again. In a chat interface that wait looks like a frozen application, and the discarded tokens are billed.

3.2 Why It Happens

1. The error stays in the generation state. A language model writes each token conditioned on every token before it, held in the key-value (KV) cache. An unsupported sentence written mid-answer stays in that cache and shapes everything after it.

2. Nothing checks the answer while it is written. The gate reads the finished answer. Between the first wrong token and the check, every new token is built on the error.

3. The platform has two choices, both costly:

graph TD
    A["Post-Hoc Verification Gate<br/>fires after full generation"] --> B{Hallucination<br/>Detected?}
    C["Option A: Catch & Regenerate<br/>━━━━━━━━━━━━━━━━━━━━<br/>Discard all tokens.<br/>Re-process all input tokens.<br/>Re-generate entire response.<br/>━━━━━━━━━━━━━━━━━━━━<br/><b>Latency: 12–25 seconds</b><br/><b>User assumes app is frozen</b><br/><b>Session abandonment</b>"]
    D["Option B: Ship Contamination<br/>━━━━━━━━━━━━━━━━━━━━<br/>Hallucinated tokens pass gate.<br/>Fabricated citations reach user.<br/>Lawyer relies on fake holdings.<br/>━━━━━━━━━━━━━━━━━━━━<br/><b>Professional liability</b><br/><b>Sanctionable event</b><br/><b>TPRM disqualification</b>"]
    B -->|Yes, strict threshold| C
    B -->|No, lowered threshold| D
    B -->|Not detected| D

    style C fill:#593e13,stroke:#000,stroke-width:2px
    style D fill:#4d1414,stroke:#000,stroke-width:2px

Option A costs users through latency. Option B costs through liability. Moving the threshold trades one for the other. Removing both needs a check during generation.

3.3 The Pipeline

flowchart LR
    A[User Query] --> B[Retrieve Chunks]
    B --> C[Generate Full Response]
    C --> D[Post-Hoc Gate]

    D --> E[User]
    D -->|Re-run<br>If Fails Gate| C

The gate passes or discards the whole answer. It cannot repair one sentence.

The constraint comes from the API boundary (By design). The platform generates through managed frontier APIs (OpenAI GPT-4o, Anthropic Claude). These do not expose the KV cache. The application cannot inspect, edit or cut the generation state. When a check fails, the only remedy is full regeneration. This holds for any application on any managed endpoint, and the cost argument rests on it.

The verification model adds its own limit. Natural language inference (NLI) models judge whether a source supports a sentence. Those of the DeBERTa class accept 512 tokens. Italian legal clauses, with nested qualifications, cross-references and exceptions, often run longer. The checker either drops the qualifying words or splits the clause and loses the cross-references.

---

4. The Hallucination Tax

The Hallucination Tax is the per-query cost of post-hoc verification on a managed API: regenerated answers plus discarded tokens.

4.1 Data Classification

Each input to the cost model is classified by source.

4.2 Pricing Anchors (Confirmed, Published List Prices)

List prices exclude enterprise and batch discounts, so the model is an upper bound on API cost.

4.3 Per-Query Cost

Let $C_{\text{gen}}$ be the cost of one generation pass, $r$ the hallucination rate:

$$C_{\text{gen}} = \underbrace{T_{\text{in}} \cdot P_{\text{in}}}_{\text{input cost}} + \underbrace{T_{\text{out}} \cdot P_{\text{out}}}_{\text{output cost}}$$

For GPT-4o ($T_{\text{in}} = 5{,}000$, $T_{\text{out}} = 800$):

$$C_{\text{gen}} = (5{,}000 \times \$2.50/\text{1M}) + (800 \times \$10.00/\text{1M}) = \$0.0125 + \$0.0080 = \$0.0205$$

The tax is the expected cost of regeneration plus the discarded output:

$$C_{\text{tax}} = r \cdot C_{\text{gen}} + r \cdot T_{\text{out}} \cdot P_{\text{out}}$$

$$C_{\text{tax}} = 0.20 \times \$0.0205 + 0.20 \times 800 \times \$10.00/\text{1M}$$

$$\boxed{C_{\text{tax}} = \$0.0041 + \$0.0016 = \$0.0057 \text{ per query}}$$

Two ratios describe the same $0.0057:

$$\frac{C_{\text{tax}}}{C_{\text{gen}}} = \frac{\$0.0057}{\$0.0205} = 27.8\% \quad \text{(overhead on the \textit{generation pass})}$$

$$\frac{C_{\text{tax}}}{C_{\text{total}}} = \frac{\$0.0057}{\$0.0263} = 21.7\% \approx 22\% \quad \text{(share of \textit{total per-query cost})}$$

$C_{\text{total}} = \$0.0263$ includes generation, local NLI verification and the tax (§10.1). "22%" in this document always means the second ratio.

Important

The tax follows from two constraints: managed APIs do not expose the generation state (By design), and the check runs after all tokens are generated and billed (Inferred). Fine-tuning lowers the rate $r$. It does not lower the cost per event, which the API boundary fixes. At 10% instead of 20%, each remaining event still costs a full regeneration.

4.4 At Scale

The tax is linear in daily query volume $Q$:

$$C_{\text{daily}} = Q \cdot C_{\text{tax}}$$

• Mid-tier = GPT-4o generation + local cross-encoder NLI → $C_{\text{tax}} = \$0.0057$/query
• High-tier = Claude Opus 4.8 generation → $C_{\text{tax}} = \$0.0130$/query (same formula, Opus pricing: $C_{\text{gen}} = \$0.045$)

(30-day month. By design, modeled. Volumes are illustrative; I have no view of the vendor's traffic. The figure to rely on is the per-query tax.)

4.5 Three Scenarios

† Compliance overhead = verification call + Hallucination Tax, a wider quantity than the tax in §4.3. A = $0.0000 judge + $0.0057 tax. B = $0.0165 judge + $0.0057 tax. C = $0.0340 judge + $0.0130 tax. In A the local NLI cost (~$0.0001) is negligible.

Warning

In Scenario B the judge call alone costs $0.0165 a query, about 80% of generation, on every query. With the tax, over half of generation spend is verification overhead.

---

5. Compliance Gaps

Derived from the reconstruction in §3 and §4.1, not from an audit of the vendor's systems.

---

6. The Control

Important

§6.1 to §6.6 are the specification as written in May 2026, before anything was built. Their component figures (rollback latency, auditor throughput, GPU allocation) were modeled from published benchmarks. §6.7 records what was built in October 2026, what was measured, and where the build departs.

6.1 Principle

The vector database, retrieval and model training stay as they are. The generation path changes from checking after generation to checking each sentence during it.

This needs one prerequisite: replace the managed API with a self-hosted open model (Llama-3-70B) on dedicated GPUs behind vLLM or SGLang. That gives access to the KV cache, so a rejected sentence can be cut.

Warning

The prerequisite is the largest cost, and most of it is operational. Self-hosting moves capacity planning, GPU supply, model upgrades, inference-engine updates and on-call for a stateful serving tier onto the platform team. The open model's Italian legal quality must be qualified against the frontier model it replaces (§15.2). The cost model in §10 counts GPU savings, not this.

6.2 The In-Flight Loop

As the model streams tokens, a separate 8B sentence-level NLI auditor checks each sentence:

sequenceDiagram
    autonumber
    actor User as User Query
    participant Router as Query Router
    participant Retriever as Retrieval Tier
    participant LLM as Llama-3-70B<br/>(KV Cache Exposed)
    participant Aud as 8B NLI Compliance<br/>Auditor
    participant Log as Audit Trail<br/>(Structured JSON)

    User->>Router: Submit legal query
    Router->>Retriever: Route to retrieval pipeline
    Retriever->>LLM: Deliver evidentiary chunks + query

    loop Sentence-Level Generation & Audit
        LLM->>Aud: Stream sentence buffer
        Note over Aud: Sentence compared against<br/>retrieved evidentiary sub-spans<br/>(ColBERT coordinate extraction)

        alt Entailment (Claim verified)
            Aud->>LLM: Lock KV cache state & continue
            Aud->>Log: Log: ENTAILMENT, sentence verified,<br/>source sub-span hash, confidence score
        else Contradiction (Hallucination detected)
            Aud->>LLM: TRUNCATE KV cache to last<br/>verified token index
            Aud->>LLM: Inject corrective context prefix
            Aud->>Log: Log: CONTRADICTION, sentence rejected,<br/>rollback depth, corrective prefix
            Note over LLM: Generation resumes from<br/>clean state, contaminated<br/>tokens removed
        else Neutral (No factual claim)
            Aud->>LLM: Allow passage without lock
            Aud->>Log: Log: NEUTRAL, conversational<br/>sentence, no verification required
        end
    end

    LLM->>User: Deliver verified output
    Log->>Log: Finalise query audit record<br/>with SHA-256 hash

6.3 Cutting a Rejected Sentence

When the auditor rejects a sentence, generation stops and the KV cache is cut back to the end of the last verified sentence:

1. The model writes $S_1$. The auditor compares it with the matching sub-span of the retrieved text and returns ENTAILMENT. The cache state for $S_1$ is kept as a branch point in the inference engine's prefix tree, the structure in which it stores cached text so that later requests can reuse it.

2. The model writes $S_2$, with a fabricated citation. The auditor returns CONTRADICTION.

3. The engine discards the cache for $S_2$. Nothing after this point is conditioned on it. Managed APIs do not offer this operation.

4. The engine adds a correction (e.g., "Correction: The retrieved sources do not support the existence of the cited holding. Resuming with verified evidentiary basis...") and resumes from the clean state.

By design, modeled, cost against full regeneration:

$$\boxed{t_{\text{rollback}} + t_{\text{corrective}} \approx 120\text{ms} \quad \text{vs.} \quad t_{\text{full\_regeneration}} \approx 8{,}000\text{–}12{,}000\text{ms}}$$

Tokens discarded per event fall by 95%:

$$\text{Tokens wasted}_{\text{rollback}} \approx 40 \quad \text{vs.} \quad \text{Tokens wasted}_{\text{full\_regen}} \approx 800$$

Tip

The 95% is close to arithmetic: a sentence-level catch discards one sentence (~40 tokens), a post-hoc catch discards the answer (~800). The 120 ms figure is softer: modeled, and it assumes the auditor keeps up with generation.

Measured, reference implementation, since (§6.7, 6 October 2026): one retry costs 117 ms on Qwen 2.5 7B and one NVIDIA L4 (69 ms with 8-bit weights), against 270 ms (170 ms) to re-process the prompt. That is a 7B model, not the 70B specified, one request at a time, and compared with re-processing the prompt, not with the 8 to 12 s regeneration modeled above.

6.4 Three Verdicts

The auditor returns one of three verdicts, with a cap on retries:

sequenceDiagram
    participant M as Model
    participant S as System
    participant A as Auditor
    participant UI as User Interface

    M->>S: Produces sentence buffer
    S->>A: Request verification

    alt Verdict: ENTAILMENT
        A-->>S: ENTAILMENT
        Note over S: Lock KV cache state
        S->>UI: Stream verified text
        S->>M: Continue generation

    else Verdict: CONTRADICTION
        A-->>S: CONTRADICTION
        Note over S: Check rollback count for sentence

        alt Count ≤ 3
            Note over S: Truncate KV Cache
            S->>M: Inject corrective context prefix
            M-->>S: Resume generation from clean state

        else Count > 3
            S->>A: Escalate to frontier model re-audit
            S->>UI: Mark sentence as "Unverified"
        end

    else Verdict: NEUTRAL
        A-->>S: NEUTRAL
        Note over S: Allow passage (no factual claim)
        S->>M: Continue generation
    end

Note

The cap of 3 is a design parameter, not a tuned value. It bounds the worst case at four attempts per sentence. The right value trades "Unverified" labels against tail latency and needs a running system to set.

6.5 Sub-Span Extraction

The auditor does not receive the whole 500-word chunk. Full chunks exceed the NLI model's useful context and add noise. ColBERT's token-level scores pick the ~50-word sub-span that best matches the sentence:

$$\text{SubSpan}(S_i, C_j) = \arg\max_{\text{span} \subseteq C_j} \sum_{k \in \text{span}} \max_{l \in S_i} \cos(\mathbf{c}_k, \mathbf{s}_l)$$

Two effects:

1. A ~40-token sentence against a ~50-token sub-span fits the auditor's range, which keeps the check precise.
2. The record shows which sub-span of which source verified which sentence, which makes each verdict traceable.

6.6 Current and Target

graph LR
    subgraph PRE ["Current State (Uncontrolled)"]
        direction LR
        Q1([User Query]) --> R1[Retrieve<br/>Chunks]
        R1 --> G1["Generate Full<br/>Response<br/>(Proprietary API)"]
        G1 --> PH["Post-Hoc<br/>Verification Gate"]
        PH -->|Pass| U1([User])
        PH -->|Fail| G1
    end

    subgraph POST ["Target State (Specified)"]
        direction LR
        Q2([User Query]) --> R2[Retrieve<br/>Chunks]
        R2 --> G2["Generate with<br/>In-Flight Audit<br/>(Self-Hosted)"]
        G2 --> CG["Per-Sentence<br/>Compliance Gate"]
        CG -->|Verified| U2([User])
        CG -->|Contradiction| TB["Truncate &<br/>Resume"]
        TB --> G2
    end

6.7 As Built: legal-rag-verifier, October 2026

Built as legal-rag-verifier, listed on /tools. A proof of concept. Every figure below is Measured, reference implementation, on English questions about UK statutes. Methods were fixed before each run. Results from 14 to 16 October are from the repository at commit c044b84; the LLM judge they use is a script there, not part of release 0.2.0. Statistical tests are in the repository.

Where the Build Departs From the Specification

The Mechanism

The Auditor: the Small Model Failed and an LLM Judge Replaced It

On a sealed set of 346 planted errors and correct sentences, fixed and hashed before the run, the rule check plus DeBERTa caught 80.0% of planted errors (124 of 155) and rejected 7.9% of correct sentences (15 of 191), 6 October 2026. On live output it did not hold. On statutes, 18 of its 46 rejections were real errors (labelled by frontier models). On 40 software questions graded by hand, 1 of its 12 rejections was a real error, and the loop's final answers were good 29 and 30 times against 32 for one plain answer from the same model, 8 October 2026. That is the falsifier in §15.4, row 2.

Hand-graded, 9 to 12 October 2026. The judge scored 354 of the 358 sentences in the first test.

Smaller and Hosted Judges

The judge takes about a third of each answer's time (§15.3). Cheaper judges were tested on the same two hand-graded tests, each asked for a label and the deciding passage only, 12 and 13 October 2026. The rule, fixed before the runs, was to come within 3 points of Qwen 27B on both tests.

None met the rule. The trained 1B judge takes a ninth of the 27B's time per call and is 4.8 and 8.0 points less accurate. The hosted APIs cost $0.07 and $0.13 per 1,000 calls and send each sentence and its source to a third party, which a confidential deployment may not allow (confidential AI case study).

The Loop With the LLM Judge

Qwen 27B can judge its own answers. On 40 questions answered with and without the source, it caught 35 of the 36 wrong sentences in its own text, all of them in answers written from memory, and Gemini 3.1 Pro found no error it passed, 11 October 2026. It also rolled back 18 right sentences, 16 of them honest refusals.

Against Frontier Models

The same 240 questions with the source in the prompt, 16 October 2026:

The 27B with the loop is within a 4-point margin of both frontier models, fixed before the run; the 7B with the loop is outside it. The 27B states less per answer than either frontier model.

Where the Loop Pays

The loop reduces errors when the generator answers from memory or is small. From memory, it cut answers holding a wrong sentence from 132 to 23 for Qwen 27B and from 198 to 28 for Qwen 2.5 7B. With the source in the prompt, it cut them from 25 to 13 for the 7B (p = 0.045).

With the source in the prompt and the stronger generator, the loop changed nothing: Qwen 27B made 4 errors with it and 4 without, at 2.0 s against 1.1 s per answer. On answers written with the source, the judge rolled back more right sentences than it caught wrong ones. It was right on 88.9% of their sentences, against 94.4% with no auditor (11 October 2026).

A platform that answers with the source in the prompt, as the analysed one does, gains accuracy from the loop with a small, cheap generator and none with a strong one. The cost case in §10 holds in both: a sentence caught in flight costs a retry, not a regeneration.

The runs used one server per run, greedy decoding and one run per arm. Frontier models graded most sentences; disagreements were graded by hand.

---

7. Compliance Outcomes

Caution

The Target State column is the specification. The Basis column says which rows have since been measured on the reference implementation (§6.7), on English text of UK statutes, and which are still to build. Nothing here was measured on the analysed platform or on Italian law.

Secondary: Projected Performance and Cost

All rows By design, modeled. §10 has the derivation and §15 what would invalidate it. The cost rows price verification from the LLM judge's measured share of time (§10.1).

Measured, reference implementation (§6.7), for the first row: a retry costs 117 ms on Qwen 2.5 7B on one L4, and the first verified sentence is released after 3.8 s. The 70B model specified and the cost rows were not run.

---

8. Audit Record Specification

The control writes one JSON record per query, for traceability, anomaly detection, incident review and monitoring.

8.1 Specimen Record

Note

A specimen of the specified schema with representative values. Not an extract from a running system; the Italian text and hashes are illustrative. auditor_model names the 8B NLI auditor of the specification. In the build the auditor is an LLM judge (§6.7), which returns a label and the deciding passage, not an entailment confidence.

{
  "query_id": "q-20251215-004712",
  "timestamp": "2025-12-15T14:47:12.331Z",
  "session_id": "sess-a3f8c921",
  "query_hash": "sha256:e4b2c1...",
  "generation_model": "llama-3-70b",
  "auditor_model": "nli-8b-legal-v2",
  "retrieval_context": {
    "chunks_retrieved": 4,
    "total_tokens": 3200,
    "source_ids": ["it-cc-art42", "it-cc-art2359", "eu-gdpr-art82", "it-dlgs-231-art6"]
  },
  "sentence_audit_trail": [
    {
      "sentence_index": 0,
      "text": "L'articolo 42 del Codice Civile stabilisce i requisiti per la costituzione...",
      "verdict": "ENTAILMENT",
      "confidence": 0.94,
      "source_subspan": {
        "chunk_id": "it-cc-art42",
        "token_range": [12, 67],
        "subspan_text": "requisiti per la costituzione delle persone giuridiche..."
      },
      "kv_cache_action": "LOCKED",
      "latency_ms": 18.4
    },
    {
      "sentence_index": 1,
      "text": "La responsabilità solidale si applica ai sensi del decreto...",
      "verdict": "CONTRADICTION",
      "confidence": 0.89,
      "source_subspan": {
        "chunk_id": "it-cc-art42",
        "token_range": [68, 112],
        "subspan_text": "nessuna disposizione relativa alla responsabilità solidale..."
      },
      "kv_cache_action": "TRUNCATED_TO_INDEX_0",
      "corrective_prefix_injected": true,
      "rollback_count": 1,
      "latency_ms": 22.1
    },
    {
      "sentence_index": 2,
      "text": "Pertanto, i requisiti formali includono l'atto costitutivo...",
      "verdict": "ENTAILMENT",
      "confidence": 0.91,
      "source_subspan": {
        "chunk_id": "it-cc-art42",
        "token_range": [14, 58],
        "subspan_text": "l'atto costitutivo e lo statuto devono contenere..."
      },
      "kv_cache_action": "LOCKED",
      "latency_ms": 16.7
    }
  ],
  "summary": {
    "total_sentences": 8,
    "entailment_count": 6,
    "contradiction_count": 1,
    "neutral_count": 1,
    "rollback_events": 1,
    "escalation_events": 0,
    "total_generation_latency_ms": 2340,
    "total_audit_latency_ms": 147
  },
  "human_intervention_required": false,
  "audit_hash": "sha256:7f3a91c2d8e4b..."
}

8.2 What the Record Covers

Tip

audit_hash is a SHA-256 hash of the record, so later edits are detectable. It is not a trusted timestamp and does not prove when the record was made. That needs an external notary or an append-only log.

---

9. Regulatory Applicability Matrix

---

10. Unit Economics

10.1 Per-Query Cost

Self-hosting replaces per-token billing and removes most of the tax.

The specification priced verification at $0.000135 a query, for an 8B auditor on an L4 GPU. That auditor was not built, and the small model built in its place failed (§6.7). An LLM judge replaced it. In the live loop with Qwen 27B as generator and judge on one L40S, the judge took 35 to 38% of the time per answer (Measured, reference implementation, 12 October 2026). Verification is priced here at the ratio of judge time to generation time, 0.54 to 0.61 times the cost of generation.

$$\boxed{\text{Modeled per-query cost reduction} = \frac{\$0.0263 - \$0.00299}{\$0.0263} = 88.6\%}$$

At the lower verification cost the reduction is 89.1%. With the 8B auditor of the specification it was 92.2%. The judge's share was measured with a 27B generator on answers of one or two sentences. A 70B generator takes longer per sentence, so the same judge would take a smaller share (By design).

Warning

This table leaves out two costs. GPU amortisation assumes steady utilisation; at low or bursty traffic the cost per query rises and can exceed the API (§15.3). Self-hosting adds engineering and on-call cost (§6.1). The 88.6% is a compute-cost figure, not total cost of ownership.

10.2 The Tax

Modeled, the tax falls by over 99%:

$$\frac{C_{\text{tax,before}}}{C_{\text{tax,after}}} = \frac{\$0.0057}{\$0.00002} = 285\times$$

Each event discards ~40 tokens instead of ~800. The model hallucinates at the same rate; the cost per event changes.

10.3 At Scale

(30-day month, modeled at the higher verification cost. At 10K/day, check the utilisation assumption in §15.3 before relying on the delta.)

---

11. Deployment Specification

A shadow migration, with production unchanged until the gates pass.

Phase 1: Shadow Auditing

Production keeps running on the managed API. Traffic is mirrored asynchronously to a staging cluster with the self-hosted model and the auditor. No user sees staging output.

graph TD
    User([Production Traffic]) --> Prod["Production Pipeline<br/>(Proprietary API +<br/>Post-Hoc Gate)"]
    User -.->|"Async mirror<br/>(100% traffic)"| Shadow["Shadow Pipeline<br/>(Self-Hosted +<br/>In-Flight Auditor)"]

    Prod --> ProdOut([Production Output<br/>to User])
    Shadow --> Telemetry["Telemetry Comparison<br/>━━━━━━━━━━━━━━━━<br/>• Accuracy delta<br/>• Latency delta<br/>• Rollback rate<br/>• Contamination caught"]

    style Prod fill:#000,stroke:#f57c00,stroke-width:2px
    style Shadow fill:#000,stroke:#1a73e8,stroke-width:2px
    style Telemetry fill:#000,stroke:#16a34a,stroke-width:2px

Phase 2: Acceptance Gates

At least 72 hours, with cutover gated on:

Note

Gate 4 measures what the migration gives up. A system that halves compute cost and reasons worse on Italian law is a worse system.

Phase 3: Rollout

A feature flag moves a rising share of traffic to the new pipeline. The managed-API path stays through a bake period, so rollback is a flag change. Target downtime: zero.

---

12. Technical Appendix

12.1 Why Managed APIs Lock in the Cost

This argument carries the document. It is By design: it follows from the API interface, with no benchmark or vendor detail needed.

Three properties combine:

1. Generation is autoregressive. Each token depends on all before it. An error at $t_k$ reaches $t_{k+1}, t_{k+2}, \dots, t_n$.

2. Managed endpoints are closed. The APIs of OpenAI, Anthropic and Google take a prompt and return an answer. The KV cache is not accessible, so the application cannot inspect or cut it during generation.

3. Every output token is billed, used or not. A rejected answer is billed in full, and so is its replacement.

So the application cannot stop the error spreading (1), cannot remove it mid-answer (2), and pays for all of it (3).

This is why the control needs self-hosting. Better engineering on a managed endpoint does not remove property 2. This claim stands apart from every modeled number in the document.

Note

Property 2 describes managed endpoints as they are today. A provider could add a partial-rollback or resume primitive and close the gap. The point would remain: the control has to sit where the generation state is.

12.2 Prefix-Tree Cache Management

The cut uses a prefix tree over the KV cache (RadixAttention, as in SGLang). Each verified sentence is a branch point:

$$\text{RadixTree}: \text{Token Sequence} \to \text{KV Cache Coordinates}$$

On rollback the engine returns to the last kept branch point and resumes. The discarded branch is freed, and the model's state no longer holds the rejected sentence.

12.3 Fine-Tuning and In-Flight Checking Are Complements

Fine-tuning reduces how often errors occur. In-flight checking reduces what each one costs. A system can use both.

---

13. Lessons

1. A check after generation runs after the error has spread. Between the first wrong token and the check, every token builds on the error. Where accuracy carries professional liability, the check belongs inside generation.

2. A managed API limits which controls are possible. Hiding the inference engine behind request and response suits general applications. It rules out any control that needs the generation state. An audit of a legal AI system needs to ask what the hosting model allows.

3. The tax is set by billing and architecture, not model quality. Fine-tuning lowers how often the tax applies, not its size. Cost per event belongs in the unit economics next to the error rate.

4. The generation-tier check depends on retrieval. The auditor is only as good as the evidence it compares with. The build used the supplied provisions directly (§6.7).

5. Standard practice produces this gap. Generate-then-validate is the default in the major RAG frameworks. The tax looks like normal API spend, and the gate eventually catches most errors, so dashboards show neither. Finding it needs tests that measure the verification loop's cost and latency, not only output quality.

6. An argument and a benchmark are different evidence. The strongest claim here, that a managed API cannot partly repair an answer, has no measurement and needs none. The largest numbers, 88.6%, 285× and 120 ms, are the weakest, because they rest on assumptions. The build has since measured the mechanism and the judge's share of time (§6.7, §10.1); the cost figures remain modeled.

---

14. What This Specification Does Not Establish

• It is a proof of concept, not a deployment. The loop was built and run (§6.7) on English questions about UK statutes, one server per run, greedy decoding, one run per arm. It was not run on Italian law, on the analysed platform, or with the 70B model specified. No shadow migration was run.
• The 0% target covers detected contamination only. Rollback removes what the auditor catches. Measured, reference implementation: with the loop, 4 of 240 answers from Qwen 27B and 13 of 240 from Qwen 2.5 7B still held a wrong sentence with the source in the prompt. Auditor recall sets the residual rate.
• The diagnosis is a reconstruction from public sources. I had no access to the platform. If its architecture differs from §4.1, the diagnosis does not apply to it; the failure class remains for systems on framework defaults.
• The Stanford 17 to 33% rate was measured on other systems, applied here by analogy.
• No regulatory conformity is claimed. §9 states design intent. Conformity is assessed by qualified assessors on running systems.
• The economics are compute-only. Engineering time, on-call, GPU procurement and model qualification are excluded (§6.1, §10.1).

---

15. Residual Risks and Falsifiers

The argument in §12.1 holds. The engineering around it has weak points, and they are not where the cost model points.

15.1 Auditor Recall Is the Ceiling

Every guarantee in §7 covers detected contamination only, so auditor recall limits the whole design.

• Legal entailment is harder than general entailment. A misattributed holding can read as consistent with the source and still be legally wrong.
• The judge is the costliest part of the check. Every cheaper judge tested fell more than 3 points below Qwen 27B on at least one of the two tests (§6.7).
• False positives cost too. An over-eager auditor cuts valid text, loops, hurts fluency and adds latency. Gate 2 in §11 covers this.

Measured, reference implementation (§6.7): the small auditor built, DeBERTa-v3-base with a rule check, was right on 58.3% of unseen sentences against 55.6% with no auditor. On the sealed set it caught 6 of 18 dropped qualifiers. Qwen 27B as the judge was right on 90.1%. On answers written with the source, the judge rolled back more right sentences than it caught wrong ones (88.9% against 94.4% with no auditor).

15.2 Changing the Model Changes Capability

§10 treats Llama-3-70B as a cost line. It is also a change in Italian legal reasoning, possibly for the worse. If quality drops, the control protects a weaker product. This is Gate 4 in §11, the gate most likely to fail.

15.3 GPU Cost Depends on Utilisation

$0.00175 a query holds only under steady high load. Legal research traffic follows business hours. With idle capacity overnight the cost per query can rise several-fold, and at low volume self-hosting can cost more than the API. The break-even volume is the first number a buyer needs, and it requires the vendor's real traffic shape.

The judge also uses the GPU. In the live loop with Qwen 27B as generator and judge on one L40S, the judge took 35 to 38% of the time per answer when it returned a label and the deciding passage only, and 58 to 61% when it also wrote a reason (legal-rag-verifier, 12 October 2026). §10 prices verification from the first figure.

15.4 What Would Falsify the Core Claims

---

16. Summary

A post-hoc gate checks an answer after every token is generated and billed. On a failure it regenerates the whole answer: 12 to 25 seconds and the discarded tokens paid for. Modeled at list prices, that is about 22% of the cost per query, linear in volume.

A managed API does not expose the generation state, so model size, prompting or fine-tuning cannot lower the cost of each event.

The specification checks each sentence during generation, cuts a rejected sentence from the generation state and regenerates only that sentence. Modeled, waste per event falls about 95%, from ~800 tokens to ~40.

It was built as a proof of concept, legal-rag-verifier, and measured on English questions about UK statutes (§6.7). Rollback works on a production inference server: a retry costs 117 ms against 270 ms to re-process the prompt on one L4. The first auditor, a small entailment model, failed on live output, as §15.4 allowed. With Qwen 27B as the judge, closed-book answers holding a wrong sentence fell from 198 to 28 of 240 for Qwen 2.5 7B. With the source in the prompt, Qwen 27B with the loop was within a fixed margin of two frontier models, and made the same 4 errors without it.

The loop pays in accuracy where the generator answers from memory or is small. In cost it pays wherever an error is caught, because a retry replaces a regeneration. Re-modeled with the judge's measured share of time, the per-query compute cost falls 88.6 to 89.1%, against 92.2% with the auditor first specified.