Hermes Agent updates: a monthly update and a test set first

Hermes changes faster than a quote desk should: one release rolled up about 460 pull requests

On 13 October 2026 the project's release page listed v0.21.7 as the latest of dozens of releases, and the v0.21.5 notes, dated 24 September 2026, rolled up about 460 pull requests. As reported on GitHub. Security researchers are watching it too: the Cloud Security Alliance counts three CVEs disclosed against Hermes components (CVE-2026-7396, CVE-2026-7397 and CVE-2026-6829; Cloud Security Alliance note, 4 May 2026); the note's nine-in-four-days figure refers to OpenClaw, a different agent platform.

Hermes is only one layer. The Ollama version, the model files and macOS can each shift an answer, and a small shift can move a price. Not updating is a risk and updating without a check is a risk, so updates pass through a gate. Staffbox is not affiliated with Nous Research.

Each month the test runs first, and the update goes on only if the score holds

Updates are manual today: the IT provider applies them monthly, and we re-run your tests before and after each monthly update. A scripted update and rollback are in review. The client joins a monthly re-test on their own quotes. The test runs on the same Mac with the same settings, against the pass bar agreed before day 0. If a result drops, nothing is applied until the cause is found.

1. Freeze. Note the current versions of macOS, Ollama, Hermes Agent, the Staffbox CLI and the model.
2. Baseline. Run the client's test set (the CLI has an eval command) and record the score and the time per answer.
3. Stage. Staffbox reviews the release notes for the changes that matter to a quote desk and runs the new versions against the same test set.
4. Compare. Same pass bar, same set. A refusal still counts as a miss.
5. Apply. The IT provider applies the update on the client unit if, and only if, the staged run passed.
6. Check settings. Confirm the settings in the next section survived the update.
7. Re-test. Re-run the set on the unit, then with the client on their own recent quotes.
8. Log it. One line per update: date, versions, score, who applied it.

Step 3 is the part of the process that is Staffbox's to run, and the update itself is applied by the IT provider only after step 4 passes. This split follows our partner materials; no customer unit has been through a cycle yet. Each correction becomes a test in next month's set.

Three Ollama settings must be confirmed after every update

From our demo unit's notes (1 Oct 2026), these Ollama settings are the ones we check after anything changes.

OLLAMA_KEEP_ALIVE=24h
OLLAMA_CONTEXT_LENGTH=65536
OLLAMA_MAX_LOADED_MODELS=1

• Changing the context size (num_ctx) costs about 100 seconds on a 16 GB machine, so keep one context size per run. A test that mixes sizes will look slower than it is.
• Ollama can hang when the disk is busy or models unload between steps. The fix on our demo unit is to restart its launch service: launchctl kickstart -k gui/$UID/ai.staffbox.ollama. The label is specific to our unit; yours may differ.
• A 14B model timed out on 16 GB and is not used in demos.
• Hermes's reported minimum context is 64K. See the 64K page for what that means for each model.

The test set has already caught a pricing fault, and each correction becomes a new test

The 6 Oct fix changed how the quote action handles items that are not on the price list. Same 16 GB Mac mini M4, same Qwen3 8B model, same generated regression set of 360 requests (written by a model, answers checked by code), fictional companies:

All 6 wrong totals in the 5 Oct run were on requests that also asked about an unstocked item. The 6 Oct fix makes the quote action flag such an item instead of substituting the nearest one (by design; 0 wrong totals in 475 test answers). Of the 11 misses that remained on 6 Oct across all 475 answers, 10 were declines that asked the customer, and 1 was a routing miss. Without the test set we would not have seen the problem, and without a re-run we could not have shown the fix. On a new trade it had never seen, an electrical distributor (14 Oct, Dell test box), the default Qwen3 8B setup scored 30 of 40 and 2 of 15 on messy requests, with 8 wrong prices in 55, because per-foot units, price levels and trade shorthand are not handled yet; that is why we score your own list first (the electrical result).

The same loop runs on client work. When a client finds a wrong draft, the failing request is added to the test set (the CLI has an addtest command), and the next monthly re-test includes it. Over time the set reflects what the client actually gets asked. The fix ladder is brain first, then tool, then model.

Urgent fixes, rollback and version pinning are not decided yet

• An out-of-cycle process for an urgent security fix. Our public record only defines the monthly cycle (not decided yet).
• A rollback procedure for a failed update. A scripted update and rollback are in review, not shipped, so we make no promise here.
• Whether Hermes updates will ever be pinned for a quarter. We do not know. Today the installer pulls the current Hermes release on install day; a change to pin it is in review.
• Any real customer update history. No customer unit exists yet.

Sources

• Hermes Agent repository and releases (Nous Research, MIT), seen 13 Oct 2026 (as reported).
• Cloud Security Alliance research note on Hermes Agent CVEs, 4 May 2026: three CVEs against Hermes components (CVE-2026-7396, -7397, -6829); its "9 CVEs in 4 days" headline refers to OpenClaw (as reported).
• Staffbox: security sheet (Updates), demo environment notes of 1 Oct 2026, scorecards before and after the 6 Oct fix (evals/results 2026-10-05 and 2026-10-06) in github.com/Staffbox-ai/staffbox (MIT).