A human-reviewed evaluation of whether today's best LLMs can reliably — and cheaply — answer Advanced Squad Leader rules questions.
1. Open-source models with agentic tools can answer easy rules questions for less than a cent — deepseek-v4-flash scored 100% recall / 86% calculation on the easy eval at about 0.4¢ per query.
2. On a medium-hard eval a real gap opens between open-source and frontier models — in both directions. Fable holds 91% recall / 100% calculation against deepseek's 82% / 80%, but at roughly 50× the cost (~30¢ vs 0.57¢ per query).
TABLE 01 — 85-question easy + 60-question medium evals, human-reviewed · each model with and without agentic tools · cost & timing from live production usage
| Model | Agentic | Eval | Recall | Calc | Cost / q | Avg time | Date run | |
|---|---|---|---|---|---|---|---|---|
| deepseek-v4-flash | yes | Easy | 100% | 86% | 0.36¢ / — | 30s / — | Jul 12 | VIEW → |
| deepseek-v4-flash | no | Easy | 88% | 54% | 0.26¢ / — | 24s / — | Jul 14 | VIEW → |
| Fable | no | Easy | 98% | 100% | ~30¢ / — | ~20s / — | Jul 4 | VIEW → |
| deepseek-v4-flash | yes | Medium | 82% | 80% | 0.57¢ / — | 46s / — | Jul 14 | VIEW → |
| Fable | no | Medium | 91% | 100% | ~30¢ / — | ~20s / — | Jul 7 | VIEW → |
Retrieval — no tools rows get ~20k tokens of rulebook context per question (20 chunks from a vector store of the 700-page rulebook). with tools rows start from a 5-chunk baseline and the model fetches more sections itself via lookup tools.
Cost / q & Avg time — each cell shows two numbers: eval run / live production. The first is measured from the eval itself (the run's token volume at the model's API pricing; wall-clock response time). The second is the average across live chat turns in that row's mode — agentic rows draw only from tool-using turns, non-agentic rows only from plain turns; hover it for the sample size. An em dash means that measurement doesn't exist yet. Fable's eval-side numbers are approximations (its runner doesn't record token usage).
Daily averages from live chat · text-only baselines, image variants dashed, agentic (tool-using) variants dotted