Last week a paper from ServiceNOW's AI lab was published on arxiv about some things they did with an evolving harness they wrote. This is kind of like what OpenClaw did with a learning loop but more formalized, reproducible and less yolo, as they target what they call "Enterprise Agents".
The evolved framework is much more successful, faster and cheaper:
| Metric | Baseline | Evolved | Δ |
| Domain objective score | 57.1% | 83.2% | +26.1 pp |
| Mean partial credit | 67.3% | 86.1% | +18.8 pp |
| Turns per task | 16.35 | 11.98 | −4.37 |
| Cost per task | $0.14 | $0.10 | −29% |
| Tasks with guardrail violations | 20 | 4 | −16 |
| Total guardrail violations | 33 | 4 | −29 |
| Zero-score tasks | 24 | 6 | −18 |
But they are claiming something much more interesting than just saving time and money: it's model agnostic. They evolved the harness once and then benched the single evolved harness instance with different models:
| Benchmark | Model | Baseline | StarHarness | Δ |
| ITBench | Qwen3.5-27B | 25.6% | 70.0% | +44.4 pp |
| ITBench | GPT-5.4-mini (medium) | 33.1% | 79.4% | +46.3 pp |
| ITBench | GPT-5.4 (medium) | 40.0% | 75.0% | +35.0 pp |
| ITBench | GPT-5.5 (medium) | 50.8% | 78.7% | +27.9 pp |
| EnterpriseOps-Gym | Qwen3.6-27B | 18.2% | 38.8% | +20.6 pp |
| EnterpriseOps-Gym | GPT-5.4-mini (medium) | 13.6% | 31.1% | +17.5 pp |
| EnterpriseOps-Gym | GPT-5.4 (medium) | 23.3% | 43.7% | +20.4 pp |
| EnterpriseOps-Gym | GPT-5.5 (high) | 37.8% | 48.5% | +10.7 pp |
| AutomationBench | Qwen3.6-27B | 48.2% | 75.5% | +27.3 pp |
| AutomationBench | GPT-5.4-mini (medium) | 29.6% | 70.0% | +40.4 pp |
| AutomationBench | GPT-5.4 (medium) | 57.1% | 83.2% | +26.1 pp |
| AutomationBench | GPT-5.5 (medium) | 59.6% | 84.9% | +25.3 pp |
Being able to boost performance almost 3x with Qwen3.5 with just a better harness is a lot better than anything I've seen from generic task-oriented LoRA. It's similar to what I saw on GLM-5.2 with prompts engineered for Claude though, so this is an interesting thing.
Instead of needing 20 H200s for RL, you only need 1, and an evolving harness? Looking forward to more insights into this.
PS: don't trust verify will have to wait because the repo literally says "coming soon", but let's see.
What does evolving harness mean? Like the AI rewrites its own harness?
As they use the harness, they feed the traces and output to what they call a
proposerwhich proposes patches to the modifiable surface of the harness (the stuff that @optimism lists), they then evaluate the results produced by each patch relative to the pre-patch harness using a "hidden selection set" of problems, accept or reject each patch based on whether it improves the harness performance or not, then at the end of that loop make sure it's not overfitting to the hidden selection set by testing it against a final "holdout set."They have a cool picture of the process.
You read that as ongoing metrics against a the last pre-patch run?nvm the caveat that I missed is the fixed task.
Somewhat - but I understand it to be not as freeform as what Claw does. From the paper:
It's a bit hard to say how they do it exactly with the source still under
soon™, but I'd imagine they A/B test prompts/skills/mcp and tool source code, and then optimize and generalize what works into skills and subagent templates.I recall a presentation from the Pi developer who basically said less harness is more in terms of performance, that as the models get better the harnesses built for earlier models get worse.
Seems recursive improvement of the harness is the next logical conclusion.
Trying to overcome the urge to build a clanker news experiment of sorts for distributed recursive self-improvement. Just let anyone point an agent at it, clankers self-moderate with sats in effort to collaborate on self-improvement.
Wonder how long it would take for Dario to show up crying at my door
I'm on my 4th iteration of my local forge skill and on the 3rd of my set of dependency analysis templates (I basically have specific ones for each framework / language and some get tuned for things that I review often). The only thing that stays relatively static are my standard prompt templates for "do this", "test that", "examine this", "answer the effen question". It would be better if I had something more structural to this than reminding myself that "hey maybe I can make some time this week to orchestrate a bot retro".
I need deeper traces if I'm to improve things with all these new models in the mix though. Sometimes it's obvious and I can just add a line that prevents specific behavior. But based on the paper, the real gains ought come from looking at where tool calls fail and then (a) improve the tool and (b) improve the instruction.
I saw something that framed this as inner-loop / outer-loop and how subagents are increasingly a better way to prevent context pollution
Unless it's super straight-forward, it generally works better for me to have the option to edit the instruction so I pass that into an issue that I can edit before assigning. I do often do subtask decomposition instructions though - it doesn't all have to be hitl within a consistent task. Context matters then though, so I try to define my tasks so that context stays under 100k tokens.