MEMON SYSTEMS

journal

Measurements, failure modes, and notes on retrieval systems.

RSS: /feed.xml
Benchmarks

Can You Build Confidential Legal AI Without a TEE? Empirical Lessons on PIR and Keyed Models

Two studies on fictional UK employment matters, October 2026, with the cloud provider treated as an attacker with full access. A model hidden behind a secret key was read in full: 100% of the secret content recovered from public statistics of its weights. Keeping client work on a laptop and fetching public law by private information retrieval sent 0 of 17 client secrets to the cloud, where a standard cloud pipeline sent 16 private tokens. With dates and amounts computed in code, 30 of 30 conclusions on 8 unseen matters were correct. Work that needs cloud compute on client data needs a hardware trusted execution environment, such as Prem Enclave.

Benchmarks

In-Flight Rollback on SGLang: A Sentence-Level Check Cut Wrong Legal Answers From Memory by 83 to 86%

legal-rag-verifier checks each sentence while the model writes it, cuts a rejected one and rewrites it with the statutory text added. 240 UK statute questions answered from memory, 14 October 2026: answers holding a wrong sentence fell from 132 to 23 for Qwen 27B and from 198 to 28 for Qwen 2.5 7B. After a real error, the default retry was wrong again 71 times in 95, and a retry with the provision added at the cut 16 times. With the source already in the prompt, Qwen 27B had 4 such answers with the loop and 4 without, against 3 for Gemini 3.8 Flash and 8 for Claude Opus 4.6.