Workflow guide
The scorecard: how we grade an AI quoting tool on your past quotes
Short answer: On a messy set of requests it had never seen, our worker scored 15 of 15 (6 Oct 2026); yours is scored on your own past quotes. On three new trades it scored 30, 26 and 22 of 40, with wrong prices on all three.
Accuracy claims are cheap, and a demo on the vendor's own price list proves little about yours. A scorecard on your own past requests is harder to fake. Send up to three requests your team already answered; drafts come back within 2 business days, misses included.
Updated 14 October 2026
Only your own past quotes test the way your customers write
A vendor demo uses the vendor's price list. A benchmark uses someone else's questions. Neither tells you whether a tool can handle the way your customers actually write: part numbers typed without the dash, quantities in words, a forwarded thread with last year's order in it. The only fair test is a sample of your own past requests, with the answers your team actually sent.
That is how Staffbox is scored before anything is installed. Everything below is the method, then the numbers from our own fictional test companies, misses included.
Six steps take a test set to a monthly re-test
| Step | What happens | Tool |
|---|---|---|
| 1. Test set | 30 of your real past requests with the answers you sent. 10 are held back. | A test file, one line per request |
| 2. Brain | Your price list, discount rules and procedures become notes the worker reads. | Notes in a folder you own |
| 3. Score | Run the set. A refusal counts as a miss. The 10 held-back answers are graded blind by your team. | staffbox eval |
| 4. Fix | Fix in a fixed order: the brain first, then a tool, then a bigger model. Re-run after each fix. | staffbox eval again |
| 5. Gate | A workflow goes live only if it meets the pass bar, agreed before day 0. | staffbox review --bar |
| 6. Monthly re-test | Re-run the set, compare with last month, turn every correction into a new test. | staffbox review, staffbox addtest |
The scorecard reports the share priced, the share right with no edits, the minutes your team spends editing, and the 90th-percentile time to a draft (nine answers in ten arrive at least that fast). The commands come from the open repository README.
Unseen requests: 15 of 15 right on 6 Oct, but wrong prices on all three new trades on 14 and 15 Oct
Hardware: a 16 GB Mac mini M4. Model: Qwen3 8B reading the company notes, with the quote action. Companies: Fieldstone IT (an IT provider) and Peachtree Cabinet Works (a cabinet shop), both fictional. Times are one request at a time with the model warm; concurrent requests queue.
| Test set | Right | Average time | 90th percentile |
|---|---|---|---|
| Fieldstone IT, standard test | 40 of 40 | 2.3 s | 5.3 s |
| Peachtree Cabinet Works, standard test | 39 of 40 | 2.5 s | 4.4 s |
| Messy requests, set 1 (used while fixing) | 19 of 20 | 1.2 s | 2.3 s |
| Messy requests, set 2 (never used while fixing) | 15 of 15 | 1.1 s | 1.7 s |
| Generated regression set, Fieldstone | 201 of 204 | 1.6 s | 2.1 s |
| Generated regression set, Peachtree | 150 of 156 | 3.2 s | 3.4 s |
| Ponce Electric Supply, standard test (unseen trade, Dell RTX 3090, 14 Oct) | 30 of 40 | not valid | not valid |
| Ponce Electric Supply, messy requests (unseen, 14 Oct) | 2 of 15 | not valid | not valid |
Overall on 6 Oct: 0 wrong totals in 475 answers. Of the 11 misses, 10 declined and asked the customer, and 1 was a routing miss.
Then a new trade. On 14 Oct we scored a fictional electrical distributor it had never seen, once, with no tuning: 30 of 40 on the standard set (notes alone 27, model alone 2) and 2 of 15 on messy requests, with 8 wrong prices in 55 answers. Per-foot wire, contractor price levels, trade shorthand and special-order flags are not handled yet. Times from that run are not valid, because another job shared the GPU. A second new trade, plumbing & HVAC, written blind and scored once: 26 of 40 and 3 of 15 messy, with 11 wrong prices in 55; a fix for trade units lifted it to 31 of 40 but left the 11 wrong prices, so it failed our no-wrong-prices bar (the blind result). A third blind trade, lumber and building supply (15 Oct): 22 of 40 and 15 wrong prices as it ships; a second fix that reached 0 wrong prices on the two trades it was built against made 16 here (the third blind trade). Account levels, surcharges and quantity breaks are not handled reliably on a new trade. That gap is why every site gets its own build and its own score before anything is installed. What broke, and what it needs · standard results · messy results.
Why two messy sets matter: set 1 was seen while we were fixing the tool, so 19 of 20 is partly a score on questions we had studied. Set 2 was never used for fixing, so 15 of 15 is the better guide, and it is still only fifteen requests. The 360 generated requests were written by a local model on a separate machine on 3 and 4 October, and the expected answers were computed by code. Fifteen requests is a small sample, and we say so.
See every graded answer
All of it is public. Browse every graded answer on this site, filtered by test set, model and result. The Staffbox scorecard on Hugging Face shows each graded answer. The Hugging Face organisation also holds the test sets with their fictional brains and the scorecards (2,825 graded answers across 42 runs, 30 September to 14 October 2026; browse them all). The raw result files sit in the github.com/Staffbox-ai/staffbox repository under evals/results.
Read these scores with four caveats. Notes + calculator scored 31/40, below notes alone (33/40): a general calculator didn't help; the quote action (prices in code) did. "Unseen" = 15 distinct requests re-run 6 times (90 answers). Fieldstone's 40 standard tests are the set the quote action was built on. All Mac mini scores are on a 16 GB unit; customer pilots may run on 32 GB (the install guide: "32 GB memory recommended; 16 GB runs the 8B model"), not yet scored on 32 GB. Browse every graded answer.
Most of the score comes from the notes and the tool, and every fix stays public
The same 40 Fieldstone tests were run on 30 September 2026 on a Dell RTX 3090 test box. Qwen3 8B: 1 of 40 alone, 33 of 40 with your notes, 40 of 40 with notes plus the quote action. Qwen3 27B: 2 of 40 alone, 39 of 40 with notes, 40 of 40 with notes plus the quote action (39 of 40 on Peachtree). The notes do most of the lift; the quote action, which does the prices in code, closes the rest. Model size matters far less (our view).
An alternative model for buyers who cannot use Alibaba models, OpenAI's gpt-oss-20b, scored 39 of 40 on Fieldstone, 40 of 40 on Peachtree and 15 of 15 on the unseen messy set, on the same GPU box. It needs a 32 GB Mac, and its speed on Apple hardware is not measured yet. See models and their scores.
From 5 October to 6 October
On 5 October the 5 Oct version scored 345 of 360 on the generated regression set (requests written by a model, answers checked by code): 198 of 204 for Fieldstone and 147 of 156 for Peachtree. Its 15 misses were 9 refusals of requests it could have priced and 6 wrong totals. All 6 wrong totals were on requests that also asked about an item the price list does not carry. The 6 Oct version (after the fix) flags unstocked items instead of substituting (by design; 0 wrong totals in 475 test answers), and scored 351 of 360 on the same set. We keep both results public because a scorecard that only moves up would be hard to believe. Every kind of miss is on its own page.
A pass bar agreed before day 0 decides whether anything goes live
The bar is a number agreed before any test is run, by the person who owns quoting. For example, the repository README shows the gate command with --bar quote=85 as its example. We do not pick the number for you. A workflow that misses the bar does not go live, and a refusal counts against it, so a tool cannot pass by declining everything.
The return on top of the score is simple: quotes per week times minutes saved per quote times the loaded hourly rate. It is a projection until the pilot runs, then it is measured from editing time. It is shown on the scorecard next to $595 a month at founding sites (first 100) for as long as the box is live (draft pricing).
Questions
How do you grade an AI quoting tool?
On your own past requests. You send 30 or more past quotes with the answers you sent; 10 are held back and your team grades them blind. The scorecard shows share priced, share right with no edits, minutes spent editing and the 90th-percentile time to a draft.
What counts as a miss?
A wrong total, a wrong queue, or a refusal of something that could have been priced. A refusal counts as a miss so a tool cannot score well by declining everything.
Is 15 requests enough?
No. Fifteen is a small sample and we say so. The 360 generated requests add volume but come from a model, with expected answers computed by code. The 30-quote test on your own history is the better evidence.
Sources
- Scores and times: evals/results on GitHub: the 6 Oct and 5 Oct 2026 Mac mini runs, and the 30 Sep 2026 runs on the Dell RTX 3090 test box (including gpt-oss-20b).
- Every graded answer: huggingface.co/spaces/staffbox/scorecard; organisation: huggingface.co/staffbox.
- Method and commands: README section "How a site goes live" in the repository, read 13 Oct 2026.
Or call us: 404 493 6248 · How the free test works