Statute Research Agent · tutorial

How the agent is built, and how to show it in 5 minutes

An agent that answers questions about five Canadian federal Acts, where every claim carries a verbatim quote that code checks before anyone reads it. Watch the recorded runs on the demo page.

Architecture: one research run

The agent answers questions about five Canadian federal Acts (PIPEDA, the Privacy Act, the Canada Labour Code, the Copyright Act and the Canadian Human Rights Act). The corpus is public Justice Laws XML parsed into section records (ingest/parse.py, committed as data/corpus.jsonl) and indexed in SQLite FTS5 with bm25 ranking (app/corpus.py). Not legal advice.

POST /v1/research ──► run stored, 202 + run_id          (app/main.py, app/runs.py)
        │
        ▼
 Claude tool-use loop ── search_statutes ─┐             (app/agent.py, app/tools.py)
        │             ── get_section ─────┼──► Corpus   (app/corpus.py, SQLite FTS5)
        │             ── list_acts ───────┘
        ▼
 submit_answer: claims with {section_id, quote}
        │
        ▼
 verifier, no model call ── fail ──► one repair turn    (app/verify.py)
        │                               │
        ▼                               ▼ still failing
 gate: pass                     claim moved to "blocked", gate: blocked
        │
        ▼
 usage: tokens, cache, latency, USD per call and per run (app/settings.py prices)
        │
        ▼
 GET /v1/research/{id}  and  /v1/research/{id}/events (SSE trace)

POST /v1/research stores a run and returns 202 with a run_id. Sending the same Idempotency-Key again returns the same run.
The model gets three read-only tools that read the same corpus as the REST API, and must answer through a strict submit_answer tool: claims, each with {section_id, quote} citations.
app/verify.py checks every citation without a model: the section exists, the quote is verbatim (after whitespace normalisation), every claim is cited, nothing cited is repealed.
A failing answer gets one repair turn with the findings. A claim that still fails is moved to blocked, never silently dropped, and the run's gate reads blocked.
Every model call records input, output and cache tokens, latency and USD. The run sums them.
The same retrieval and verifier are exposed over MCP (mcp_server.py) for Claude Code or Claude Desktop.

The 7 harness layers

Each layer of a long-task agent harness, and where it lives in the repo.

1. Workspace facts (SPEC.md, README.md, docs/architecture.md)
No CLAUDE.md in this repo. The spec, README status table and architecture page are the facts every agent session reads first.
2. Reusable skill (Not in this repo)
The repeatable steps live as Makefile targets (make ci, make eval, make run) that any agent or person runs the same way.
3. Source access via MCP (mcp_server.py)
search_statutes, get_section and verify_citations over stdio or streamable HTTP, using the same corpus and verifier as the REST API.
4. Permission and hook rules (app/tools.py, app/settings.py)
No hooks in this repo. The agent can only call three read-only tools plus submit_answer; an unknown tool is an error. Tool-call and time limits stop a run.
5. Evidence reviewer (app/verify.py)
A deterministic reviewer, not a model: every quote is checked against the section text, and failing claims are blocked after one repair turn.
6. Model and effort configuration (app/settings.py)
Model (default claude-sonnet-5-5, overridable by STATUTE_MODEL), max tool calls, time limit, max tokens and the price table in one frozen dataclass.
7. Completion condition (app/agent.py, app/tools.py)
A run is done when submit_answer is called and verified, or a limit is hit and the answer is marked partial. The eval gates in evals/thresholds.yaml define done for a change.

How the gates stop bad answers

Three gates and one score, from the cheapest to the most expensive. None of the first two calls a model.

1. The citation gate, on every run. app/verify.py is plain code. Its findings are no-citation, unknown-section, quote-mismatch, repealed and malformed. After the one repair turn, app/agent.py moves each claim that still fails into answer.blocked with its findings, and gate becomes blocked. The answer the user reads can only contain quotes that exist in the law. Tests: tests/test_verify.py, tests/test_agent.py.

2. The retrieval gate, on every change. evals/golden.yaml holds answerable questions with their expected section ids and out-of-scope questions the agent must decline. evals/retrieval.py runs them through search alone and fails when recall@5 drops below 0.85, the threshold in evals/thresholds.yaml. The last export, docs/evals.json, recorded recall@5 = 1.000 and MRR = 0.617 on 12 questions. make ci (Makefile) runs it with lint, types and tests.

3. The answer eval, by hand. evals/run.py runs the agent on the golden set and checks the answer thresholds in evals/thresholds.yaml: quote fidelity 1.0 and correct decline rate 1.0. --client fake checks the harness with no model and no key; --client live costs money and is run by the owner.

4. DeepEval faithfulness, merged. Statute PR #10 added a DeepEval faithfulness suite (evals/faithfulness/suite.py) over 12 golden questions (evals/faithfulness/set.yaml). It scores each answer's kept claims from 0 to 1 against the retrieved and cited sections and reports the mean and the lowest score. A gate block fails the suite; a score under 0.80 (faithfulness.review_below in evals/thresholds.yaml) only flags the answer for review. The default run uses a fake judge with no key; the live run costs money and is the owner's. Walkthrough: docs/eval-tutorial.md. On /demo the faithfulness field shows each recorded run's score, or "not scored" where the run has none.

Cost control

Cost is a number on every run, not an estimate after the fact.

Price table. app/settings.py holds USD per million tokens. The default model, claude-sonnet-5-5: input 2.00, output 10.00, cache read 0.20, cache write 2.50. A model without a price refuses to start.
Limits. At most 8 research tool calls, 90 seconds of wall clock and 16,000 output tokens per call (app/settings.py). Past a limit the answer is marked partial instead of running on.
Per call and per run. app/agent.py records input, output, cache-read and cache-write tokens, latency and USD for every model call and sums them for the run, with the share of prompt tokens served from cache.
Caching. Tools and the system prompt go first and are cached, and a breakpoint on the newest tool result lets each next call read the earlier conversation at the cache price (app/agent.py).
Free paths. Search, section reads and the retrieval eval make no model call.
Not built yet. The spend guard (USD 3.00 a day, 15 runs per IP a day) is designed in SPEC.md section 8 and not on main, so live research stays private. The demo replays recorded runs and needs no key.

Arithmetic to say out loud: 30,000 input tokens and 2,000 output tokens cost 30,000 × 2.00 / 1,000,000 + 2,000 × 10.00 / 1,000,000 = USD 0.08, before cache savings.

Team process: how it was built

The repo was built by an AI team under one human, with the same loop for every change:

Spec. SPEC.md is the design; each feature cites its section.
Work order. A director agent turns the spec into a bounded task: goal, start ref, steps, the checks that prove it, what is out of scope.
Worker PR. A worker agent builds it test-first in its own branch and opens one PR with the check output.
Reviewer. A separate reviewer agent reruns the checks in its own session and posts a verdict: APPROVE or CHANGES with findings.
Human merge gate. Anything that spends money (cloud resources, paid API use) waits for a human decision.

docs/build-log.json, generated on 26 Oct, counted 7 PRs, 21 reviewer verdicts, 7 change requests, 7 findings from a second code-review model and 3 human gates. Main now carries 14 merged PRs (#1 to #14), from the CDK stack to the citation gate, request caching, the DeepEval faithfulness suite, the Promptfoo suite, Langfuse traces and the demo-run export.

The 5-minute script

0:00 Open /demo. Point at the run picker: runs recorded live against the model, replayed, no live key.
Say: Legal research where every sentence must be backed by the statute. Replayed runs, so nothing can break while we talk.
0:45 Pick the PIPEDA collection-without-consent run. Walk the trace: search, then read, then submit.
Say: The model can only search and read the law. It must answer through one structured tool: claims with section ids and quotes.
1:45 Open the claims. Show the verbatim quote next to its section id and the gate badge reading pass.
Say: This check is plain code, not another model. The quote either exists in the section or it does not.
2:30 Switch to the California malpractice run: out of scope, so the agent declines with a scope note and no claim. Then open /demo/evals and show the failing golden questions and their reasons.
Say: No recorded run had a claim blocked: a failing answer gets one repair turn, and what still fails is shown as blocked with the reason, never quietly dropped. The evals page is honest about precision: those answers also cited a neighbouring section.
3:15 Show the faithfulness panel and the usage block: tokens, cache share, seconds, USD, as recorded.
Say: Faithfulness is the model-graded layer: a DeepEval suite, merged, that reports the mean and the lowest score. A run shows its score, or "not scored" where it has none. Cost is per call and per run, from a price table in code.
4:00 Open this tutorial at the 7 layers and the process section.
Say: Each layer maps to a file. The repo was built by agents through spec, work order, PR, independent reviewer and a human money gate.

Likely questions

What stops the model from inventing a citation?: The verifier in app/verify.py checks every quote against the section text. A claim that still fails after one repair is blocked and shown with its finding.; Engineering manager point: Put the hard guarantee in deterministic code and keep models for the judgment calls.
Why not use a model as the judge for citations?: Verbatim matching is a solved problem in code: cheaper, faster and the same result for the same input. The DeepEval faithfulness suite (statute PR #10, merged) covers what code cannot: whether the claim follows from the quote.; Engineering manager point: Layer gates by cost: free checks on every run, paid ones on the eval set.
How do you know a change did not make retrieval worse?: The golden set runs through search on every change and fails below recall@5 = 0.85. The last export shows 1.000.; Engineering manager point: Quality is a merge gate with a number, not a demo-day impression.
What does one run cost?: Each call records tokens and USD from the price table; the run sums them. Tool calls, time and tokens are capped, and repeated prompt parts are cached.; Engineering manager point: Cost visible per unit of work is what lets a team set budgets and spot regressions.
What happens when the question is outside the five Acts?: The agent is told to decline through submit_answer, and the golden set has out-of-scope questions; the answer eval requires a correct decline rate of 1.0.; Engineering manager point: Define refusals as a tested behaviour, not a prompt hope.
How was this built so fast with a small team?: Agents in fixed roles: a director writes bounded work orders, workers open one PR each, an independent reviewer reruns the checks, and a human decides anything that spends money.; Engineering manager point: Roles, written acceptance checks and an independent reviewer scale better than heroics.
How do you review AI-written code?: A separate reviewer session reruns lint, types, tests and evals itself and posts APPROVE or CHANGES; a second review model adds findings. The build log counts 21 verdicts and 7 change requests.; Engineering manager point: Trust the checks the reviewer ran, not the author's claim that they passed.
Can other tools use the same research?: Yes. mcp_server.py exposes search, section reads and citation verification over MCP, on the same code as the REST API.; Engineering manager point: One source of truth behind every interface keeps behaviour consistent.
What is not production-ready yet?: The Lambda handler, the daily spend guard and live streaming behind API Gateway are designed but not on main, so live research stays private.; Engineering manager point: Say what is missing plainly; a known gap with a plan beats a hidden one.