Every graded answer, public

On the unseen set, all 90 answers across six runs were right

Messy set 2 is fifteen requests that were never used while we fixed the tool. Six runs answered them (Qwen3 8B on the Mac mini and on the Dell test box, plus gpt-oss-20b on the Dell), so 90 answers, every one graded right. Fifteen distinct requests is a small sample, so change the filters below to see every other set, misses included.

Tap or press Enter on any answer to see the request, the expected answer and the raw model output. The filters are kept in the page address, so a filtered view can be shared as a link. Long outputs are cut at 600 characters with "…"; the full text is in the dataset.

Notes plus the quote action took Qwen3 8B from 1 of 40 to 40 of 40

The same 40 tests, four ways. Alone, the model got 1 right. With the company notes it got 33; with notes and a calculator, 31, below notes alone (a general calculator didn't help); with notes and the quote action (prices in code), which picks the item and quantity with the model and does the arithmetic in code, 40. The notes do most of the lift and the quote action closes the rest. Set the Setup filter to "Model alone" and the result to "Wrong" to read what a bare model says about a price list it has never been shown: across every run, 14 of 300 such answers were right.

Code grades every answer, so read the misses, not just the score

Each test has one expected answer: a dollar total for a quote, one of a few accepted phrasings for a lookup, a queue name for an email to route, or "not in the notes" for a question the notes cannot answer. A script compares the worker's final answer line with it. No person edits a grade. The grader was changed twice (accept the stated total whether it comes first or last; ignore spacing such as "25 %"), and every affected answer was re-graded, as the dataset README records.

The split into right, declined and wrong is a second rule, written for this page. A miss whose answer refuses or asks the customer instead of answering, with no dollar figure in it, counts as declined. A routing miss, a timeout or any other miss counts as wrong. A decline is still a miss, so a worker cannot score well by declining everything. On the 6 October runs the rule gives 10 declined and 1 wrong out of 475 answers, which matches the published results.

There are five kinds of test set. Standard: 40 fixed tests per fictional company. Generated: 360 messy requests written by a local model, with the expected answers computed by code; useful for volume, not proof on their own. Seen while fixing: messy set 1, which we studied while fixing the tool. Unseen: messy set 2, held out. New company: the two Ponce Electric Supply sets (40 standard, 15 messy), a company the tool had never seen, scored once on 14 October; kept apart so the unseen result above stays about messy set 2. The companies are fictional, and no customer has run Staffbox yet. The Gemini Pro runs used the notes only, without the quote action, so compare them with care. Times are seconds per answer, one request at a time; the gpt-oss-20b times on the Dell are not comparable, because it was sharing the GPU with another job.

Download the data

• Dataset page and licence (MIT): huggingface.co/datasets/staffbox/staffbox-scorecards
• One row per answer: answers.jsonl · one row per run and setup: runs.csv, runs.jsonl · README
• The requests and the fictional company notes: staffbox/staffbox-evals
• This page's compact copy, built 14 Oct 2026: answers.json, runs.json
• The result files the dataset is built from: evals/results on GitHub

How the numbers are summarised, and the pass bar a real pilot uses, are on the scorecard method page. Every kind of miss is explained on what the AI got wrong.