Guide · pilots
How to run an AI pilot you can actually judge
Short answer: Agree the pass bar before day 0, grade 10 of 30 past quotes blind, and go live only if the scorecard passes.
Most AI pilots fail quietly: nobody wrote down what success meant, the vendor graded its own work, and a month later someone says it felt useful. A pilot you can judge fixes the bar first and lets your own team grade blind. This is the method we use for a quote desk, and it works for any tool that produces drafts. Send up to three requests your team already answered; drafts come back within 2 business days.
Updated 14 October 2026
Before day 0, write and date a pass bar that counts refusals as misses
The whole pilot fits on one line of dates, and the first entry comes before the tool sees a single request.
Writing the bar first prevents the three mistakes that leave a pilot unjudgeable:
- Moving the goalposts. The bar gets set after the results. Fix: write it first and date it.
- Grading your own homework. The vendor decides what counts as right. Fix: your team grades, and part of the grading is blind.
- Rewarding silence. A tool that declines everything never gives a wrong answer. Fix: a refusal counts as a miss.
Staffbox's scorecard reports four numbers: the share of requests priced, the share right with no edits, the minutes spent editing, and the 90th-percentile time to a draft. You choose the thresholds. The ones below are placeholders to argue about. They are not our standard.
| Measure | What it tells you | Placeholder bar (edit it) |
|---|---|---|
| Share priced | How often the tool tried instead of declining | At least 80% of priceable requests |
| Share right with no edits | Whether a person can send it as is | At least 70% |
| Wrong totals | The expensive failure | Zero in the 10 held back |
| Minutes editing per draft | The real saving | Less than half your current time to write the quote |
| 90th-percentile time to a draft | The slow case, not the average | Under 5 minutes |
Two rules sit above every threshold. A refusal counts as a miss. A wrong total fails the pilot, whatever the other numbers say.
During the run, 30 past quotes go in and 10 come back graded blind
- Pick one workflow. One kind of request with a source of truth. For a quote desk: requests that come in by email and are answered from a price list.
- Collect 30 past cases. Real requests you already answered, with your answers. Include the awkward ones. Leave out passwords and personal data.
- Hold back 10. The vendor does not see your answers for these until grading is done.
- Write the pass bar, then date it. See the template below.
- Run the tool on the 30. Without touching your price list or your instructions in between.
- Grade the 10 blind. Our suggestion (our view): mix the tool's answers with your own originals in a random order. The grader marks each right, fixable or wrong without knowing the source.
- Decide against the bar. Pass means go live on that workflow only. Fail means fix the cause or stop.
Our free score follows exactly these steps on our Mac mini in Atlanta, and we delete your files afterwards.
A worked example: the grading sheet
Fieldstone IT is a fictional IT provider we use for demos. This shows the format of a grading sheet, not real grades. In this format (our suggestion), the 10 answers are mixed with originals so the grader cannot tell them apart.
| Case | Source (hidden from grader) | Grade | Edit minutes | Note |
|---|---|---|---|---|
| H-01 | Tool | Right | 0 | |
| H-02 | Original | Right | 0 | |
| H-03 | Tool | Fixable | 3 | Wrong unit |
| H-04 | Tool | Wrong | n/a | Total off. Fails the bar. |
| H-05 | Original | Right | 0 |
After grading, the sheet is unblinded. A single wrong total fails this pilot under the rules above, and the fix ladder starts: first the brain (the notes and price list the tool reads), then the tool, then the model. Corrections become new tests so the same miss cannot return unnoticed.
What our own runs looked like
As a reference for what machine-graded runs on fictional companies produce: on 6 October 2026, on a 16 GB Mac mini M4 with Qwen3 8B, the unseen messy set scored 15 of 15, Fieldstone IT 40 of 40 and Peachtree Cabinet Works 39 of 40; a generated regression set (requests written by a model, answers checked by code) scored 351 of 360, with 0 wrong totals across 475 answers. Of 11 misses, 10 declined and 1 was a routing miss. These are not customer results. No customer has run Staffbox. On a new trade it had never seen, an electrical distributor (14 Oct, Dell test box), the default Qwen3 8B setup scored 30 of 40 and 2 of 15 on messy requests, with 8 wrong prices in 55, because per-foot units, price levels and trade shorthand are not handled yet; that is why we score your own list first (the electrical result).
At the example inputs, break-even is about 7 minutes saved per quote
The return is quotes per week times minutes saved per quote times a loaded hourly rate. Until the pilot runs it is a projection. After, minutes saved come from measured editing time. The break-even for the draft price of $595 a month is a short sum:
| Step | Value |
|---|---|
| Monthly price, converted to a week ($595 x 12 / 52) | $137.31 a week |
| Loaded hourly rate (example input) | $40 |
| Hours of saving needed a week ($137.31 / $40) | 3.4 hours, about 206 minutes |
| Quotes per week (example input) | 30 |
| Minutes saved per quote to break even (206 / 30) | about 7 minutes |
The $40 rate and the 30 quotes a week are example inputs, not typical values. If you send 8 quotes a week at the same rate, break-even needs about 26 minutes saved on each. That is why the entry bar is at least 8 quotes a week and why we say the return is yours to compute. guess for the saving per quote until a pilot measures it.
After a pass, go live on that one workflow and re-test every month
- Go live only on workflows that passed.
- Re-test monthly on your own newest quotes. Re-run the test set before any model update.
- Keep the pass bar file. When the bar changes, date the change.
- Keep a person on the send button.
Price if you go ahead: $595 a month at founding sites (first 100) for as long as the box is live (draft pricing), free 30-day pilot, no onboarding fee; $695 a month plus $1,800 onboarding from unit 101.
Related: Prepare the input with the price-list guide. Understand misses on the misses page. The first free step is the free test. Terms are in the glossary. Back to all guides, or see how an IT provider packages this.
Questions
Why grade blind?
Because a grader who knows which answer came from the tool tends to be kinder or harsher than the answer deserves. Mixing the tool's answers with your own originals is one way to remove that (our view).
Why 30 past quotes and 10 held back?
Thirty is enough to see the shapes of your requests, and ten held back gives a blind check on cases the tool was not tuned on. Fifteen requests is a small sample, so treat any score as a start.
Who sets the pass bar?
You do, and you write it down before day 0. Staffbox does not set it for you.
Sources
- How we measured: two fictional companies (Fieldstone IT, Peachtree Cabinet Works), graded by code, Qwen3 8B on a 16 GB Mac mini M4 (Dell RTX 3090 for the 30 Sep runs), 30 Sep to 6 Oct 2026. No customer has run Staffbox yet.
- Score method: the Staffbox offer of 2 Oct 2026. Measured results of 6 Oct 2026: evals/results on GitHub.
- Break-even arithmetic is ours; inputs are examples.
Or call us: 404 493 6248 · How the free test works