What the AI got wrong, in public

Five kinds of miss exist, and only a wrong total can slip past a tired reviewer

One fix took wrong totals from 6 to 0 on the same 360 requests

On the 360 generated messy requests the 5 Oct version of the quote action (16 GB Mac mini M4, Qwen3 8B) got 345 right and 15 wrong:

All 6 wrong totals were on requests that also asked about an item the price list does not carry. Fieldstone scored 198 of 204 and Peachtree 147 of 156. Our reading is that the arithmetic was fine and the weak point was saying no to one line while pricing the others.

Ten of the 11 remaining misses were declines a person sees

After the 6 Oct fix the quote action flags unstocked items instead of substituting (by design, and tested rather than guaranteed). Same hardware and model. Across 475 answers (the standard sets, two messy sets and the 360 generated requests) there were 0 wrong totals and 11 misses:

Ten of the 11 were declines. One was the routing miss. Declining is the right direction for a failure to go, because a person sees it, but it still counts as a miss in our own score. We do not hide it by calling it "safe".

On the earlier 2 October runs (before the fix, 115 answers across four test sets) the count was 2 misses and no draft quoted a wrong price. One was the same Peachtree routing miss, the other a request where the worker asked to confirm quantities instead of pricing. Browse every graded answer, with the raw output beside each miss.

The notes and the tool keep misses small; without the notes an 8B model scored 1 of 40

The misses above are small because two things are in place: the company brain and the quote action that does the arithmetic. Take them away and the picture changes. On Fieldstone IT (Dell RTX 3090, 30 September 2026), Qwen3 8B scored 1 of 40 alone, 33 of 40 with your notes, and 40 of 40 with the notes plus the quote action. Qwen3 27B went from 2 of 40 alone to 39 of 40 with notes. The notes do most of the lift; the quote action closes the rest.

Our offer notes also record a comparison on 20 messy held-out requests: local with the quote action against cloud with notes only. The cloud model (Gemini Pro, same notes, no quote tool) got 14 of 20 and refused six it could have priced, while our local 27B model with the quote action priced all 17 priceable requests, got 14 exactly right, and declined 2 of the 3 it should not price. That means one of those three was priced when it should not have been. A single comparison on 20 requests is thin evidence either way.

An unseen trade, electrical on 14 Oct: 23 misses in 55 answers, 8 of them wrong totals

The 6 Oct numbers above come from two companies the worker was built and fixed on. On 14 Oct we wrote a third, Ponce Electric Supply, a fictional electrical distributor, and scored the worker once with no changes: 30 of 40 on the standard set and 2 of 15 on messy contractor-style requests, notes plus quote action, on our Dell RTX 3090 test box. Its 8 wrong prices in 55 answers compare with 0 in 475 on 6 Oct. Trade units, price levels and stock flags it had never met broke it.

The largest wrong total, verbatim. Request: "300 yellow wire nuts". The list sells them in boxes of 100 at $9.80, so the right total is $29.40. Raw output: Quote action: 300 x WN-YL-100 at $9.80 = $2,940.00, less 10% = $2,646.00; line total $2,646.00. In our view a reviewer would likely catch a wire-nut line a hundred times too high; a contractor quoted list price instead of their level is harder to spot.

Nothing was changed for this run, and its timings are not valid because another job shared the GPU. Then the blind trade. A fictional plumbing & HVAC set, written by someone who did not see our fix, scored once: 26 of 40 and 3 of 15 messy as it ships, with 11 wrong prices in 55. The trade-units fix lifted it to 31 of 40 and 5 of 15 but left the same 11 wrong prices, so it failed the bar set beforehand (no wrong prices) and is not shipped. Account levels, the refrigerant surcharge and quantity breaks are not handled reliably yet when they interact. Then a third blind trade. On a fictional lumber and building supply yard (15 Oct), the worker as it ships made 15 wrong prices in 55, and a second fix that had reached 0 on the two trades it was built against made 16, including list prices where a market adjustment or quantity break applied. Neither fix is shipped. This is why we build and score your own list before anything is installed. The third blind trade · The blind result · The full write-up · The electrical page · standard results · messy results.

Every miss becomes a test, and what we cannot measure yet needs a live site

• The reviewer sees it: a declined request carries a note, and a priced one carries its row and list date. See human approval.
• A correction is added as a new test with staffbox addtest, which never edits an existing test.
• The fix is tried in a fixed order: the brain first, then a tool, then a bigger model. The substitution fix on 6 October was a change to the tool.
• The monthly re-test runs the whole set again before an update goes live and compares it with last month.

What we do not know yet

• How the worker behaves on a real customer's price list, with its own odd rows and exceptions. The one new trade we tried, electrical, went badly.
• How often reviewers pass a wrong draft over months. That needs a live site to measure.
• Whether the declined cases are the same ones each month, or drift.
• Speed on Apple hardware for the gpt-oss model option. Not measured yet.
• Anything about attachments, PDFs or drawings. We make no claim about them.

The way to find out for your business is the free test on your own past quotes. The method for scoring it is on the scorecard page.

Sources

• Counts, 5 and 6 Oct 2026: github.com/Staffbox-ai/staffbox, evals/results.
• Every graded answer: huggingface.co/spaces/staffbox/scorecard.
• Cloud model comparison and 20-request test: our offer notes, 2 Oct 2026 (not re-run for this page).