Which local model for Hermes Agent: scores on quoting, not chat

Notes took the 8B from 1/40 to 33/40 on Fieldstone IT (Dell test box, 30 Sep) and the quote action to 40/40; model size barely moved it

Without the brain (the folder of company notes Hermes reads), the 8B model scored 1 out of 40 on the fictional IT provider Fieldstone IT, and the 27B model scored 2. Neither knew the firm's prices, products or rules, and a model three times larger could not guess them either. With the brain alone, the 8B reached 33 and the 27B 39 on Fieldstone (Dell test box, 30 Sep). Adding a quote action that looks prices up and does the arithmetic in code took both to 40 out of 40.

A cloud model showed the same pattern with notes only (it had no quote action). On the clean set a leading cloud model (Gemini Pro) scored 5/40 without the notes and 39/40 with them. On 20 messy held-out requests the comparison was local with the quote action against cloud with notes only: the local 27B priced all 17 priceable requests, got 14 exactly right and declined 2 of the 3 it should not price; the cloud model got 14 of 20 and refused six it could have priced. fact One comparison, small sample, from our own offer notes.

With notes, the 8B model on a 16 GB Mac mini matched the 27B on both clean sets

The design, not the parameter count, carried the result on this task. That does not make the 8B as good as the 27B on every task; we tested one. Companies are fictional: Fieldstone IT is an IT provider, Peachtree Cabinet Works is a cabinet shop.

Sources are listed below. The 27B model class is what a Mac Studio would run, and Studio speed is not measured. The Mac mini figures are on the Mac mini page, including timings per test set.

gpt-oss-20b is the non-Qwen option: one test short on Fieldstone, but it needs 32 GB

Some buyers cannot run Alibaba models. For them, gpt-oss-20b scored 39/40 on Fieldstone IT, 40/40 on Peachtree Cabinet Works and 15/15 on a messy held-out set, all on the Dell test machine. Two costs come with it: it needs 32 GB of memory, twice the demo Mac mini, and its speed on Apple hardware is not measured. The results files date those runs 2 October 2026. Rules on which models some public buyers may use are on the non-Chinese model page and the government page. On a new trade it had never seen, an electrical distributor (14 Oct, Dell test box), the default Qwen3 8B setup scored 30 of 40 and 2 of 15 on messy requests, with 8 wrong prices in 55, because per-foot units, price levels and trade shorthand are not handled yet; that is why we score your own list first (the electrical result).

Models we have not graded, including the one the Hermes docs favour, need your own 30-request test

The Hermes local Ollama guide (checked 13 October 2026) lists gemma4:31b as the one model on its table with reliable tool calling, and says it needs 24 GB or more of RAM. We have not run gemma4 on a quoting task, so we neither recommend nor rule it out. A third-party write-up reports that qwen3, with a 40K native window, trips Hermes's context check while gpt-oss at 128K passes (betterclaw.io, as reported, seen 13 October 2026). Our installer gets Qwen3 8B past that check by setting the window to 65,536; the details, and the prompt lengths we have not tested, are on the 64K page.

To choose for your own job, grade it the way we did:

1. Write 30 real requests and the right answers. Hold 10 back.
2. Run the smallest model that fits your hardware, with your notes.
3. If it misses, fix in this order: the notes, then the tool, then the model.
4. Count a refusal as a miss. A model that declines everything scores well on "no wrong prices" and badly on usefulness.
5. Grade the held-out 10 blind. Fifteen requests is a small sample; thirty is better.

Sources

• Hermes docs: local Ollama setup, checked 13 Oct 2026
• betterclaw.io: Hermes context length fix, as reported, seen 13 Oct 2026
• github.com/Staffbox-ai/staffbox (Staffbox code and evals/results, MIT), evals/results 2026-09-30, 2026-10-02 (gpt-oss-20b) and 2026-10-06
• huggingface.co/spaces/staffbox/scorecard (every graded answer)