60 coach questions: why niche AI needs its own benchmark
Built the first narrow slice of LII Sport Bench: 60 questions for a basketball coach, grounded in public methodology materials from the RFB Academy.
This is not a joint release with the federation and not an endorsement claim. I used public sources, credited them in the methodology, and ran my own assistant through the kind of questions where a general benchmark usually tells me nothing.
In a niche, the question gets harder: can I trust this assistant here — in basketball methodology, children’s age groups, tactics, regulations, and safe refusals.
What is in the benchmark
v0.1 is small on purpose:
→ 60 questions
→ 9 public RFB Academy sources
→ 3,448 corpus fragments after re-chunking; the first chunking pass was inflated by a chunker bug
→ 6 modules: school / PE teacher, age-group methodology, technique, tactics, attestation, boundaries and refusals
→ 0–4 rubric
→ automatic cite-match and refusal-recall checks
→ the model under test is always the examinee, never the answer key
That last line carries the method. If the AI writes the question, writes the gold answer, and grades itself, that is not an evaluation. It is a neat loop around the room. The answer key comes from the corpus; where the corpus is not enough, the item gets a methodologist slot. In the first run those questions are marked separately instead of pretending to be settled knowledge.
Frontier models hallucinate with confidence too
Before the run, I asked a frontier model to review the question set. It suggested ten confident edits.
After checking them against the sources:
→ 7 edits held
→ 2 had to be downgraded to “needs methodologist”
→ 1 was simply invented
The clearest case: the model confidently introduced a rule that zone defense is forbidden before age 12. That rule was not in the sources. The sentence sounded plausible, basketball-shaped, and professionally written.
That is why source verification is mandatory. Models can be wrong without looking stupid: confidence in prose and grounding in a document are different things.
First numbers
I ran three lanes. Same judge for all lanes: gemini-3.1-pro, pass threshold 7/10.
| Lane | Composite | Cite-match A–E | Refusal F |
|---|---|---|---|
| RAG assistant over the RFB Academy corpus | 69.2% | 50/50 | 0/10 |
| Raw qwen3.6-35b, no RAG | 36.8% | 2/50 | 9/10 |
| Closed RU cloud LLM, no RAG | 32.5% | 0/50 | 10/10 |
I am not naming the closed cloud model in the public comparison table; those APIs have their own legal rail. The internal protocol keeps the exact name. Publicly, the category, score, and conclusion are enough.
The important result is not the top-line rank. The RAG assistant is the only lane that consistently ties answers back to documents: 50 out of 50 on answerable modules. Raw models can often produce good coaching prose, but they almost never prove where it came from.
For Trusted AI, that is not a footnote. A source-less answer can read well. It becomes operational only when it can be checked.
The weak part is ours
On boundary questions, my assistant failed: 0/10 refusal-recall.
The raw models were more cautious here: 9/10 and 10/10. They more often say “I cannot answer” when the question touches medicine, dangerous practice, or an out-of-corpus regulation. My assistant sometimes tries to be useful and attaches corpus-shaped citations where it should stop.
Bad result. Good benchmark result.
Bad, because that boundary is not ready to show as a finished safety layer. Good, because the weakness showed up in measurement — not on a call, not in a public demo. Now I know exactly what to fix: refusal without fabricated citations, explicit redirect to a doctor or methodologist, zero citations when the corpus does not support the answer.
That same night I fixed the boundary gate and the chunker. The fix is live; the formal benchmark re-run is still pending, so the table keeps the first honest result.
The domain-benchmark family
LII Sport Bench is not a one-off table. It is part of the domain-benchmark family on bench.csylabs.com: sport → basketball now, medicine → next.
The idea is simple: every niche needs its own exam. General model “smartness” is secondary here; the practical question is whether we can trust it next to a human who is responsible for children, patients, documents, or money.
The basketball slice showed both sides at once: corpus-grounded RAG already wins where source grounding matters; refusal boundaries are still weak and need work. That is the point of the benchmark.
Trust is measured, not promised.