Three trades it had never seen: all three failed on wrong prices

We built a fictional electrical distributor and ran it once, with no tuning

The two earlier test companies, an IT provider and a cabinet shop, shaped how the worker reads notes and prices lines. A fair test needs a trade nobody tuned for. Electrical supply is a good one: wire sold by the foot from reels, contractor price levels, copper surcharges and counter shorthand.

• The company: Ponce Electric Supply, invented for this test. About 90 parts with house codes across wire, conduit, breakers and panels, boxes and devices, and lighting, plus notes on a 12% contractor level, an 8% copper surcharge, cut charges and special-order items. The full company and tests are public.
• The tests: 40 standard (16 quotes, 10 lookups, 8 emails to route, 6 questions the notes cannot answer) and 15 messy requests written the way contractors type.
• The answers: computed by a separate script from the price book, not by the model and not by hand. "Right" means the exact line total.
• The run: Qwen3 8B on our Dell RTX 3090 test box, 14 Oct 2026, each set run once. Nothing in the worker was changed for this run, and every answer is published as it came out.
• Not valid: answer times. Another job shared the GPU during the run, so we do not quote speed from it.

It scored 30 of 40 and 2 of 15, and 8 of 55 answers had a wrong price

On the two earlier companies the same worker made 0 wrong totals in 475 answers. Here, notes did most of the lift again, from 2 to 27 of 40, and the quote action added three more. The messy set is where it fell apart: 11 of its 15 requests needed a price, and none came back right.

Every graded answer is in the public scorecard, filtered to this company, and in the raw result files linked under each chart.

Units, price levels, shorthand and stock flags broke it, not arithmetic

In all eight wrong prices the multiplication itself was right. The misses came earlier: deciding which row a request means, in which unit, at which price level. Fourteen were declines, which a person sees and answers; eight were wrong prices, which a person has to catch.

Each cause, with one real answer

Requests and raw output are copied from the result files; only long stock lists are cut, marked with an ellipsis.

Two of these worry us more than the rest. A contractor quoted list price looks like a normal quote, so a reviewer may not notice it. And the panel line shows the model reading "40 space" as a quantity of 40 and pricing a part the notes mark special order. A wire-nut line a hundred times too high is easier to catch.

The blind test: the fix helped but failed our bar

We scored the trade-units fix once, on 14 Oct 2026, on a second trade it had never seen: a fictional plumbing and HVAC supply house. It turned many declines into right answers, but it made 11 wrong prices in 55 answers, the same as the worker as it ships. The pass bar, set in writing before scoring, was no wrong prices, so the fix failed and is not shipped.

The blind set was written by a separate author working only from the public main branch, without access to the fix. Its test files were published and fixed by hash before scoring, and they still match: tests.jsonl sha256 166807f8…, tests-messy.jsonl sha256 800b1fd3… (the blind company and tests).

Three blind misses with the fix, verbatim

Requests and raw output are copied from the result files.

Of the 11 wrong prices with the fix, 9 priced a stocked item but got the total wrong. Our reading of those answers (not a measured breakdown): a quantity break taken on pipe or refrigerant sold by the foot or pound, the surcharge left off, an account level missed or applied in the wrong order, or a unit misread. The other 2 priced requests it should have declined and passed to a person.

The regression check, which is not proof

The fix kept the two earlier companies where they were: 40 of 40 on the IT provider and 39 of 40 on the cabinet shop. On the electrical set it went from 30 to 39 of 40, and from 2 to 13 of 15 on messy requests. We built the fix while reading those electrical answers, so that set is now a development set: the jump shows the fix does what it was written to do, not that it carries to a new trade. The blind set above is the test of that (the trade-units fix on GitHub, open as a draft, not merged).

What it means

The fix turns many "I don't know" answers into right ones on a new trade. But it still commits wrong prices when price-level, surcharge and quantity-break rules interact, and a wrong price is the miss a reviewer has to catch. So the fix is not shipped, and the numbers above for the worker as it ships are the current state for these trades. The next round needed another blind set, written the same way and held to the same bar, because this one is now seen: that third trade is below. Every miss we have counted is on our misses and refusals page.

Reproduce it

The worker, the test sets and the results are public. Electrical set, from a clone of the repo at commit f362db5, with Ollama serving qwen3:8b at a 65,536-token context:

bin/staffbox eval examples/ponce-electric-supply/vault \
  examples/ponce-electric-supply/tests.jsonl \
  --model qwen3:8b --modes none,brain,brain+quote

Our result files: standard set sha256 3203682c44f7e2d6…, messy set sha256 2c1c2c49e5a0624a….

Blind set, the worker as it ships, from main at commit c3c50a3 (check out the fix branch from the fix on GitHub to score the fix); run it again with tests-messy.jsonl for the messy set:

bin/staffbox eval examples/chattahoochee-plumbing-hvac/vault \
  examples/chattahoochee-plumbing-hvac/tests.jsonl \
  --model qwen3:8b --modes brain+quote

Your model run will differ in wording; the grading is by code.

The third blind trade: the second fix did not generalise

A second fix spelled out the order in which pricing rules apply: quantity break, then account level, then surcharge, then cut charge. It reached 0 wrong prices on the electrical and plumbing and HVAC sets, the two trades it was built against. Then we scored it once, on 15 Oct 2026, on a third trade written blind: a fictional lumber and building supply yard with 98 items, board-foot pricing and a lumber market adjustment. It scored 24 of 40 and 4 of 15 messy, and made 16 wrong prices in 55 answers, against 15 for the worker as it ships. It fit the trades it was built on and did not carry to a new one.

Each cell: standard set of 40, messy set of 15, wrong prices in those 55 answers. Pass bar, set before every blind scoring: no wrong prices. Every blind run failed it.

The lumber set was written blind from the public main branch, and its tests were fixed by hash before scoring: tests.jsonl sha256 3d7c032d…, tests-messy.jsonl sha256 0dbfafe8… (the blind company and tests; the second fix on GitHub, a draft, not merged).

Two lumber misses with the second fix, verbatim

Requests and raw output are copied from the result file; the right answers come from the price book by script.

The lesson

Out of the box, the worker does not price a new trade reliably, and a fix tuned on two trades did not carry to a third. What did reach 0 wrong prices was work done on a trade's own price list and checked against that trade's own quotes. That is how a site goes live with us: we build the notes on your price list, score 30 of your past quotes against a pass bar agreed first, and go live only if it passes. Neither fix is shipped. How the free score works.

Your trade's units and price levels decide fit, and the free score shows it first

• Parts sold each at list price fared well: all 8 of those standard quotes were right. Wire by the foot (0 of 5) and contractor pricing (0 of 3) did not.
• If your counter sells by the foot, prices off levels, or gets shorthand, today's worker will decline or misprice those lines. We would rather you see that on your own list than after an install.
• The free score runs 30 of your past quotes on our Mac mini in Atlanta before anything is installed; your team grades 10 blind, a refusal counts as a miss, and it goes live only on a pass bar agreed first. How the free test works.
• For the trade-specific detail, see AI quoting for electrical supply houses.

Sources

• Results, 14 Oct 2026: standard set scorecard and raw answers; messy set scorecard and raw answers.
• Test company, tests and the script that computed expected answers: examples/ponce-electric-supply.
• Blind plumbing & HVAC set, scored once on 14 Oct 2026, as it ships and with the trade-units fix: standard, as it ships, standard, with the fix, messy, as it ships, messy, with the fix (raw answers in the matching .jsonl files); company and tests in examples/chattahoochee-plumbing-hvac.
• Earlier results (two fictional companies, 30 Sep to 6 Oct 2026): how we score and the scorecard on Hugging Face. No customer has run Staffbox yet.
• Third blind trade, lumber and building supply, scored once on 15 Oct 2026, as it ships and with the pricing-rule-order fix: standard, as it ships, standard, with the fix, messy, as it ships, messy, with the fix (raw answers in the matching .jsonl files); company and tests in examples/brookhaven-lumber-building-supply.
• More notes: all engineering notes.