One agent, one claim, and the difference between a shallow test and a real one.
Hermes is the most-used AI agent of 2026. Nous Research shipped it in February, and it crossed 100,000 GitHub stars in two months. Its headline feature is self-evolution: after a task, it writes a SKILL.md, files it away, and claims to get faster and cheaper on the next similar task. The docs put a number on it: skills cut token consumption 30 to 85 percent.
We tested that claim. Properly, which turned out to matter.
What we tested
Hermes, run on DeepSeek Flash. Four probes.
- Eight diverse checkable tasks.
- Cross-session memory.
- The skill-writing loop itself, in a full session.
- The token claim, using the skill it actually wrote.
Task completion: 8 for 8.
File creation, arithmetic, CSV to JSON, string reversal, sum of squares, prime check, file copy, multi-step. Each effect verified on the filesystem, not taken from the agent’s word. Sum of squares of 1 through 10 came back 385. The reverse of ‘hello’ came back olleh. Whether 97 is prime came back true. Eight correct, zero wrong.
Memory: held.
We planted a preference in one session: name Prakarsh, preferred language Rust. A fresh session recalled both, correctly.
Skill writing: works.
This was the part a scripted one-shot mode can’t reach, because the skill-writing loop is a background process that only runs in a full session. So we ran a full session.
Hermes did the task, then wrote a skill called script-artifact-delivery. The procedure it captured was correct: check the path, write the script, run it for real, save the output, read it back to confirm. Not a stub. A working, reusable procedure, written from the task it had just done.
Token claim: held. But only when tested fairly.
Here is the part worth reading closely, because it is where the shallow version of this test would have lied to you.
Our first pass loaded generic, hand-written skills onto non-matching tasks. Tokens went up, not down: plus 32 percent, plus 102 percent, plus 2 percent. By that test, the claim fails.
Then we did it properly. We took the skill Hermes itself wrote, from a create-script-run-save task, and replayed it on a similar task.
Cold: 87,034 tokens, 6 API calls. With Hermes’s own skill: 59,150 tokens, 4 API calls. A 32 percent cut, inside the claimed 30 to 85 percent band.
The claim is specific. Experience-derived skills, on matching tasks, cut tokens. Generic skills on unrelated tasks do not. That is exactly what the documentation says, and only the deep test showed it.
We hunted for a catch. Didn’t find one.
A verifier that only tests the easy path isn’t a verifier. So we went hunting for a genuine claim-vs-effect gap in a strong agent, the way a real adversary would.
Five traps. Second-largest in a list with duplicates. Case-insensitive word counting. Palindrome checks that ignore punctuation. Sums over inclusive ranges, where off-by-one is the classic failure. Line counts with exact semantics.
Hermes handled all five. Second-largest in [1, 5, 5, 3, 2] came back 3, the duplicate handled. Sum of 1 through 10 inclusive came back 55, no off-by-one. The edge cases were called out and guarded.
That is the honest result. We widened the test, recorded what happened, and found no falsification. We do not manufacture one. We sign not_falsified and move on.
What we sign
Verdict: not_falsified. Task completion held. Memory held. Skill writing works. And the token claim held, at 32 percent, when measured the way the claim is stated.
The honest finding is not that Hermes failed. It is that the claim holds when you actually measure it, and a shallow test would have missed it. We nearly did. The deep test is the whole point.
You can check the receipt yourself. The raw JSON is in the public record.