The Harness Is Becoming Its Own Layer
The word harness did not exist in this conversation eighteen months ago.
The scaffolding around an LLM was called a wrapper, a runtime, an orchestration layer, an agent framework, a system prompt, a tool loop…
Anything but a harness.
The harness became the distinction between what the model can do versus what the runtime makes the model do.
I wrote previously on Medium about how much of Claude Code’s actual behaviour lives in the harness rather than the model.
The next eighteen months of agent work will be argued on harness terms, not model terms and the teams that recognise this early will be the ones building agents that survive the next model release.
98% of Claude Code Is Not AI
98% of Claude Code Is Not AI A new 46-page study reverse-engineers Claude Code’s architecture from its open-source…cobusgreyling.medium.com
Overview
A new study manually labelled 393 failed agent episodes across seven deterministic environments.
The result is the cleanest map I have seen of where AI agents actually break and where in the lifecycle to fix them.
AI Agents fail in four places.
Action realisation; the AI agent fails to format the tool call. The JSON is broken. A required field is missing. The tool name is misspelled or invented.
Environment contract: the AI agent calls the wrong tool, or calls the right tool the wrong way. It uses search when the rules say lookup. It passes a value the system rejects.
Trajectory degeneration: the AI agent gets stuck. It repeats the same step. It loops between two states. It runs out of allowed steps without finishing the task.
General reasoning: the AI agent simply thinks wrong. The format is fine, the tool is right, the trajectory makes sense and the final answer is still incorrect. Only 9.9% of failures sit here.
The mix is not the same everywhere.
In Telecom, 90% of failures are contract and action-realisation problems.
In ALFWorld (text-based household simulator), trajectory degeneration dominates.
In WebShop (is a simulated e-commerce site), the four buckets split roughly evenly.
There is no single dominant failure mode. So there is no single fix.
The standard response is to fine-tune the model on the failures. It works, but the fix is locked to that checkpoint. New model, new fine-tune. You lose model-agnosticism the moment you start training.
LIFE-Harness took the opposite route.
The four failure categories were each mapped to a lifecycle layer in a wrapper around the model. Contract layer before interaction. Skill retrieval at task conditioning.
Action validation before execution. Trajectory monitoring after execution.
The harness was evolved from the failures of a single 4B model and then reused, unchanged, across 17 other model backbones.
The result: 116 out of 126 model–environment settings improved.
Average relative gain of 88.5%.
A frozen Qwen2.5–32B with the harness beats xLAM-2–32B, the fine-tuned tool-use derivative of the same base model.
Same failure signal. The harness fixes the wrapper, not the weights. And it keeps working when the model underneath changes.
Some background
Most agent failure analysis stops at the symptom.
The agent hallucinated a tool name.
The agent looped.
The agent stopped.
The authors ran a frozen Qwen3–4B-Instruct across seven deterministic agent environments.
This covers household interaction, web shopping, operating-system control, database tasks, and policy-guided business workflows.
They then manually inspected 393 failed episodes.
Each failure was tagged with its earliest dominant bottleneck in the agent–environment loop.
What came out is a four-category taxonomy that is general enough to map onto almost any production agent log.
The four failure categories
1. Action Realisation Failures
The model knew what to do. It just got the output format wrong.
It wrote a sentence describing the tool call instead of emitting the tool call.
It left out a required argument. It produced JSON the parser cannot read. It invoked a tool name that does not exist.
The reasoning was there. The format was wrong.
2. Environment contract mismatches
The model calls the wrong tool.
It calls search when the contract specifies lookup. It supplies a value the environment rejects. It picks the wrong tool for the job.
NVIDIA tackles this exact problem the opposite way, with a data flywheel.
The loop is familiar. Collect tool-call traces from production. Find the failures. Generate synthetic training data from the corrected examples.
Fine-tune the model. Redeploy. Repeat.
The flywheel fixes the model. LIFE-Harness fixes the wrapper around the model.
The trade-off is plain. Fine-tuning gives you a better model, but only that model, only for that training distribution.
The next time you swap in a stronger base model, you re-run the flywheel from scratch.
A harness sits outside the model. Swap the model underneath and the harness still works.
The paper runs this comparison head-to-head. A frozen Qwen2.5–32B with LIFE-Harness beats a fine-tuned version of the same base model, trained on tool-call data via a flywheel-style pipeline.
Same source model. Opposite fix. The wrapper wins, in this case.
Neither approach is wrong. But the harness is cheaper to run, easier to audit, and does not need to be redone every time a new model lands.
3. Trajectory Degeneration
Individual actions are fine.
The episode as a whole is not.
The agent repeats the same invalid command. It loops between equivalent states. It exhausts its step budget without making progress. It submits a final answer prematurely.
These failures are self-reinforcing. They are visible at the trajectory level, not the action level.
4. General Reasoning Failures
Incorrect inference. Bad decision-making. Wrong computation.
The agent largely follows the protocol and still gets the answer wrong.
This is the residual bucket. The category most agent-research work has historically focused on. In this study it accounts for only 9.9% of all failures.
The environment matters
Action realisation = turning the model’s intent into something the environment can actually execute.
The headline is not the taxonomy itself.
It is that the failure mix varies dramatically per environment.
In Telecom, 90% of failures are concentrated in contract and action-realisation problems.
There is no single dominant failure pattern.
Which means there is no single fix.
A team that ships agents across multiple domains will face different failure mixes in each domain.
A fine-tune that improves Retail will not necessarily improve Telecom. An action validator that helps OS may not help WebShop.
This is the practical insight buried in the chart.
The result
LIFE-Harness was evolved from training-trajectory failures of a single 4B model.
The resulting harness was reused unchanged across 17 other model backbones.
It improved 116 out of 126 model–environment settings, with an average relative improvement of 88.5%.
A frozen Qwen2.5–32B with LIFE-Harness outperforms its own tool-use-fine-tuned derivative, xLAM-2–32B. The harness then still adds gains on top of xLAM.
The taxonomy is portable.
The harness built on top of it is portable.
The model underneath is the swappable part.
Finally
Most agent-research conversations are still framed around model capability.
This study reframes them around interface failures and provides the diagnostic vocabulary to triage them.
Four buckets. Four lifecycle insertion points. One map.
Chief AI Evangelist @ Kore.ai | I’m passionate about exploring the intersection of AI and language. From Language Models, AI Agents to Agentic Applications, Development Frameworks & Data-Centric Productivity Tools, I share insights and ideas on how these technologies are shaping the future.
Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents
LLM agents are shaped not only by their language models, but also by the runtime harness that mediates observation…arxiv.org
GitHub - Tianshi-Xu/Life-Harness: Offical implementation of “Life-Harness”
Offical implementation of “Life-Harness”. Contribute to Tianshi-Xu/Life-Harness development by creating an account on…github.com
COBUS GREYLING
Where AI Meets Language | Language Models, AI Agents, Agentic Applications, Development Frameworks & Data-Centric…www.cobusgreyling.com
Only about 10% of the labelled failures were "the model thought wrong." Most broke on action format, environment contract, or trajectory loops. Fix those in the harness and the same wrapper still helps when you swap the model.