About the project

How it was built,
and why.

An experiment in retrieval and tool use over a notoriously dense rulebook. The goal: answers accurate enough to trust at the table, for a cent or less per question.

01
Motivation

A 1M-token rulebook.
A one-cent budget.

Advanced Squad Leader's rulebook runs over a million tokens, densely cross-referenced and full of exceptions. Looking up rules mid-game is slow even for veterans.

A frontier model answers most rules questions well — at 25–30 cents a question. The experiment: how much tool scaffolding does it take before a model nearly a hundred times cheaper is trustworthy at the table?

So far: on easy questions, an open-source model with agentic tools reaches 100% recall and 86% calculation accuracy at about 0.4¢ per query. On a harder medium set, a real gap to the frontier remains. Results, including the failures, are on the evals page.

02
Architecture

Retrieval, tools, and an
agentic loop.

FastAPI on Digital Ocean; answers stream over a WebSocket. The core is the loop below: the model asks for exactly the rules it needs, code fetches them, and the loop repeats until it can answer. On the evals, this loop is worth about 30 points on calculation questions compared to plain retrieval.

BROWSER ask a question answer streams back SERVER retrieves rules executes tools locally MODEL reads rules · calls tools writes the answer RULEBOOK & TOOLS 2,781 exact sections · ~2,400 official Q&A search_rules · get_section ift_attack · resolve_attack · resolve_cc QUESTION ANSWER · STREAMED 1 · RULES + TOOL DEFS 2 · TOOL CALLS 3 · FETCH · COMPUTE AGENTIC LOOP 1 → 2 → 3, repeating until the model stops calling tools and answers
The model asks for exactly the rules it needs; code fetches or computes them; repeat until it answers.

Here is one real run from the eval set — twelve tool calls to settle a one-line stacking question, because the answer is spread across three chapters of the rulebook:

ABRIDGED TRACE · EASY EVAL Q1 · DEEPSEEK-V4-FLASH
Q: Can I set up a half squad, MMG and leader in a steeple?
search_rules ×2 steeple stacking lives in B31
get_section("B31.21", …) ×4 limit: one HS-equivalent — and no Guns
search_rules ×3 is an MMG a Gun or a Support Weapon?
get_section("C2.1", "A5.4", …) ×3 ½″ counter = SW; SW have no stacking limit
A: Yes — the half squad fills the limit, the leader counts as zero (A5.5), and an MMG is a SW, not a Gun.
12 tool calls156k tokens in63 sjudged pass
  1. i
    Retrieve & generate
    The question plus retrieved rulebook chunks go to the model, which cites section numbers; tokens stream back with timing and per-query cost.
  2. ii
    Ground in the source
    Every citation is a link that opens the rulebook PDF at the exact page, so each claim can be checked against the source.
  3. iii
    Evaluate
    200+ eval questions score every change; an LLM judge flags rows for manual review. Results are on the evals page.
  4. iv
    Curate & retrain
    Every chat is logged; strong Q&A pairs become evals or training data, and mistakes become regression tests.
03
Deterministic

Where rules end and
arithmetic begins.

Rules questions split into retrieval ("which rule applies?") and arithmetic ("what does it compute to?"). LLMs are good at the first and unreliable at the second — the same model that cites the right rules will quietly drop a +1 DRM. So the math lives in code, in four tools the model must call instead of deriving by hand:

  1. i
    ift_odds · IFT lookup
    Column + DRM in, break / pin / casualty odds out. Also the /ift page.
  2. ii
    ift_attack · Full attack builder
    The full A7.2–.36 pipeline from a described situation: per-unit FP, range effects, itemized DRM, cowering, final odds.
  3. iii
    resolve_attack · Save-driven fire
    A whole fire attack computed from an attached VASL .vsav save — every modifier derived from the file, returned as an itemized ledger.
  4. iv
    resolve_cc · Save-driven Close Combat
    Both sides of a Close Combat from the save: FP ledgers, CCT odds, kill / reduction thresholds, Melee state.

The model picks the tool and writes the explanation; the tool supplies auditable numbers, surfaced line by line.

04
Verified

Never cite a rule
you haven't read.

Vector search finds the rule; the failures come from the rule next to it — a qualifier in a cross-referenced section that never entered the context. So the rulebook was made addressable: 2,781 numbered sections plus ~2,400 official Q&A entries, exposed two ways:

  1. i
    get_section · exact lookup, in the loop
    With Tools on, the model fetches the verbatim text — and every official ruling — of any section before citing it.
  2. ii
    Cite-check · code-driven verification
    After a draft answer, code — not the model — extracts every cited section, fetches its exact text, and forces one revision turn against it.