When Better Models Break Your Tools: The Claude Opus 4.8 Tool-Calling Regression Nobody Warned You About

The finding: Claude Opus 4.8 and Sonnet 5, Anthropic’s most capable models, are worse at calling third-party tool schemas than their predecessor Opus 4.5 was. Not worse at reasoning. Not worse at writing code. Specifically, measurably worse at conforming to tool schemas that differ from what Claude Code uses internally. And Ronacher’s hypothesis for why this happens is the kind of thing that should make every developer building on top of LLM tool APIs stop and think carefully about what they’re actually relying on.

This article unpacks the full story: what tool calls actually are under the hood, how constrained decoding works, why reinforcement learning against a closed harness creates hidden distribution biases, what the OpenAI Harmony format is doing differently, and what this means for anyone building agentic systems in 2026.

The Bug That Started It All

Pi’s edit tool supports multiple exact-string replacements in a single call. The schema uses a nested structure: a top-level edits array where each entry contains an oldText and newText field. This is a reasonable design — batching multiple edits into one round-trip is efficient and reduces latency in agentic workflows.

In Ronacher’s testing, Opus 4.8 started producing malformed edit calls. Not wrong edits — the oldText and newText payloads were byte-correct. The model had correctly identified what text to replace and what to replace it with. But it then appended extra, invented fields at the end of each entry object:

{
  "oldText": "...",
  "newText": "...",
  "requireUnique": true
}

Or this:

{
  "oldText": "...",
  "newText": "...",
  "oldText2": "",
  "newText2": ""
}

Across repeated trials, the model produced a full zoo of invented trailing keys: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, oldText2, newText2, oldText_2, newText_2, and even an event.0.additionalProperties key. Every trial produced a different made-up name. The model wasn’t consistently hallucinating one specific alias — it was freshly hallucinating a plausible-sounding field name each time.

The failure didn’t reproduce on simple prompts. A fresh single-turn “edit this file” call worked fine. The bug surfaced in agentic contexts: long histories where the model had read files, diagnosed a problem, and was composing a complex multi-step edit. Context length and thinking blocks both mattered — stripping extended thinking from history reduced the failure rate by roughly half. Turning on strict tool invocation eliminated it entirely in Ronacher’s tests.

The most important observation: Opus 4.5 did not show this behavior. When Opus 4.5 launched, it adapted cleanly to Pi’s edit schema. Opus 4.8, Anthropic’s most capable model as of this writing, does not.

Tool Calls Are Text, Not Magic

To understand why this happens, you need to understand what tool calls actually are at the implementation level.

When you send a request to an LLM API with a tools parameter, the server doesn’t maintain a separate structured interface between the model and your tool definitions. The model receives a single, large input — a prompt — that has been assembled from your messages, your system prompt, and your tool definitions, serialized into a format the model understands. The model then produces output text, and the server parses that output to determine if it constitutes a tool invocation.

Anthropic’s models use what’s become known as the ANTML format. The name comes from “Anthropic Markup Language” and uses pseudo-XML tags. A tool call emitted by the model looks approximately like this:

<antml:function_calls>
  <antml:invoke name="edit">
    <antml:parameter name="path">some/file.py</antml:parameter>
    <antml:parameter name="edits">[{"oldText": "replace this", "newText": "with this"}]</antml:parameter>
  </antml:invoke>
</antml:function_calls>

Note what this is not: it’s not real XML, and it’s not a strongly-typed structured output. It’s a pattern the model was trained to emit, which the server then parses with its own logic. Simple top-level string parameters appear inline between tags. But nested structures — arrays of objects, for instance — are embedded as serialized JSON inside a single parameter tag.

This distinction matters enormously, because it identifies two fundamentally different failure modes. A simple string parameter that the model writes inline is protected by the tag structure — the model can’t accidentally add extra fields because there are no fields, just a string value. But a JSON blob embedded in a parameter? The model writes that JSON from scratch, character by character, guided entirely by its learned priors about what valid JSON in that context looks like.

Constrained Decoding: The Two Approaches

When you want a model to produce structured output that conforms to a schema, there are two fundamentally different strategies:

Strategy 1: Generate and validate. Let the model produce whatever text it wants, then parse the output and check if it conforms to the schema. If it doesn’t, either retry or report an error. This is simple to implement and doesn’t interfere with the model’s sampling process. The downside is that the model may produce invalid output, and you’re relying entirely on learned behavior to get it right.

Strategy 2: Constrained decoding (grammar-aware sampling). Integrate schema enforcement directly into the sampling process. At each token position, compute which tokens would keep the output conformant with the target grammar, and mask out all other tokens before sampling. The model can never produce an invalid key name inside a JSON object if the grammar says only specific keys are allowed — because those tokens are masked to zero probability before the sample is drawn.

Constrained decoding can operate at different levels. You can constrain to syntactically valid JSON (preventing malformed brackets, unescaped strings, etc.) without constraining the key names. You can also enforce the full schema structure, preventing any key that isn’t in the declared properties list. The latter is what Anthropic’s strict mode appears to do — and it’s what fixes the bug Ronacher observed.

The catch is that constrained decoding has costs. It adds latency to token generation (computing the valid token mask takes time). It can interfere with quality if the model has a strong prior that a different token should come next — the forced constraint essentially overrides the model’s learned reasoning. Anthropic reportedly applies complexity limits to tool definitions when strict mode is enabled, which is why Claude Code itself doesn’t use strict mode: the tool schemas hit those limits, so the client instead relies on its own error handling and retry logic.

Without constraints, the model is purely following convention. If its trained convention for “edit tool call” includes trailing fields that weren’t in your schema, you’ll see them.

The RL Training Artifact Hypothesis

Ronacher’s strongest hypothesis — and the one that should worry every developer building tool-using agents — is that the regression isn’t random deterioration. It’s a training artifact.

Here’s the chain of reasoning:

Older Anthropic models were trained on tool call examples involving a variety of tool shapes. No single harness dominated their post-training distribution. When they encountered a novel schema like Pi’s nested edits[] array, they had to generalize from first principles. They were reasonably good at this.

Modern Anthropic models — Opus 4.8 and Sonnet 5 — were almost certainly post-trained with Claude Code as a primary or dominant environment. Claude Code is a real, deployed product with millions of users. It generates enormous amounts of tool call interaction data. If any of that data ends up in post-training (directly or through RLHF feedback loops), the model will learn not just “how to call tools in general” but “how Claude Code specifically expects tool calls to look.”

Claude Code’s own edit tool has a flat schema: file_path, old_string, new_string, and an optional replace_all flag. Not a nested array. Not oldText and newText. A simple flat structure.

Now imagine the model is generating an edit call in Pi’s nested schema format. It correctly identifies the oldText and newText values. Then, at the closing brace of the object, it faces a decision: end the object with }, or add another field. The model’s prior — trained extensively on Claude Code interactions — says that an edit operation often has one extra optional field (the replace_all flag). But it doesn’t have a trained name for an extra field in Pi’s schema. So it samples a plausible-sounding name from its general distribution, and that distribution produces a different random name each time.

Ronacher identifies exactly where in the generation process this happens: “after closing a several-hundred-token escaped newText string, where the model must decide } vs , "...".” This is the highest-entropy point of the generation. The model has just finished writing a long, complex string value. It’s uncertain. Its prior for “edit operations can have optional trailing fields” wins over “this specific schema doesn’t have one.”

The particularly uncomfortable implication: the better the model gets at using Claude Code, the worse it gets at other harnesses. These two goals are in tension, and Claude Code’s post-training is apparently winning.

The Slop Harness: Why Claude Code Trains for Forgiveness

Why does Claude Code’s behavior matter for training dynamics? Because Claude Code is extraordinarily forgiving of malformed tool calls. Ronacher’s analysis of Claude Code’s (minified) client code reveals a system built to tolerate the exact kinds of errors that strict schemas would reject.

Claude Code explicitly:

• Scans the model’s visible text for leaked <antml:invoke markup (ANTML tags that accidentally appear in the assistant’s visible response instead of as tool calls), and retries when it finds them
• Performs Unicode escape repair, fixing broken \uXXXX sequences and lone surrogates in string values
• Maintains per-tool parameter aliases: old_str and new_str are accepted alongside old_string and new_string; path is an alias for file_path
• Silently filters out unexpected keys in tool call arguments
• Does not use strict schema validation mode
• Has retry paths for malformed tool use in its state machine

From a software engineering perspective, this is a robust, defensive client. From a training dynamics perspective, it creates a serious problem.

If the model is trained (or fine-tuned through RLHF) against a harness that silently absorbs and corrects errors, there is little or no gradient against producing those errors. The model invents a trailing key, the harness strips it, the task completes successfully, the model receives positive reward. Over many iterations, the model learns that inventing trailing keys is acceptable behavior — because it always was, in the training environment.

This is a specific instance of a general problem in RL: when your reward signal doesn’t distinguish between “did the task correctly” and “did the task through a forgiving error-correction path,” you train a model that relies on the error-correction path. The model isn’t rewarded for schema precision; it’s rewarded for task completion. In Claude Code’s ecology, those two things can diverge significantly, and the model will follow the reward.

OpenAI’s Different Approach: The Harmony Format

Ronacher draws a revealing contrast with OpenAI’s approach in their gpt-oss and hosted models: the Harmony response format.

Harmony makes tool-call semantics explicit at the token level, as part of the format itself:

<|start|>assistant<|channel|>commentary to=functions.get_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|>

The critical token here is <|constrain|>json. The model, as it generates this output, explicitly marks the boundary where JSON-constrained sampling should activate. The inference stack can detect this marker and switch sampling mode on the fly. The constraint is in-band — it’s part of the token stream — which means it’s straightforward for the serving infrastructure to apply.

For hosted GPT models, OpenAI also exposes a LARK grammar option for custom tools, letting developers specify exactly what tokens are valid at each position in the output. This is grammar-constrained decoding as a first-class API feature, not just a backend implementation detail.

The contrast with Anthropic’s approach is stark. Anthropic’s models are fully closed — the training process, the RLHF data, the serving infrastructure, and the prompt format are all proprietary. Claude Code, the dominant harness, is also closed-source (though readable in minified form). Developers building on the Anthropic API cannot know what tool shapes the model was trained to prefer, what the hidden encoding of their prompt looks like, or whether their schema is close to or far from the training distribution.

Strict mode exists as a partial mitigation, but it has limitations (complexity caps on tool definitions) and it’s not the default. The model’s schema adherence, in the absence of strict mode, depends on whether your schema happens to resemble what the model was reinforced on — and you have no way to know what that was.

The High-Entropy Failure Point

To understand the specific failure mechanically, consider what the model is generating at the problematic moment.

The ANTML format embeds array-of-objects parameters as JSON strings inside parameter tags. So a multi-edit call looks approximately like:

<antml:parameter name="edits">[
  {
    "oldText": "a long, multi-line string\nwith escaped characters\nand newlines",
    "newText": "the replacement text\nalso potentially long"
  }
]</antml:parameter>

After generating the closing " of the newText value — a token that arrives after potentially hundreds of tokens of escaped string content — the model must decide: is this the end of the object, or is there more? The correct answer is }, end of object. But the model’s uncertainty at this point is high. It has just processed a long, complex string. Its attention is distributed across a large context window. Its prior for “edit objects can have extra fields” (learned from Claude Code interactions) is strong.

The wrong token, ,, opens a path where the model must now invent a field name. And since no specific name is in the training distribution for this position in this schema, the model samples from its general distribution of “plausible JSON key names for tool call arguments,” producing a different random name each time.

This explains two otherwise puzzling observations. First, why the actual edit content is correct even when the schema is violated — the model got the hard part right (identifying what to change) but failed at the structural bookkeeping. Second, why the failure is inconsistent — the exact token probabilities at the failure point depend on the full context, including the length of the preceding strings, the thinking block content, and the conversational history. Different contexts produce different probability distributions at that critical decision point.

What Strict Mode Actually Fixes

Enabling strict tool invocation fixed the problem in Ronacher’s testing. Understanding why illuminates how constrained decoding works in practice.

With strict mode enabled, Anthropic’s servers presumably apply grammar-constrained sampling to the JSON content inside array parameters. At the point where the model would decide between } and , after a complete object, the constrained sampler checks the schema: does the schema for this object type allow additional properties? If not, the , token is masked out. The model can only produce }. The invented trailing key becomes impossible.

This is grammar-constrained decoding working exactly as intended. The model’s learned prior is overridden by the enforced grammar. The quality of the generated content (the actual text of oldText and newText) is unaffected; only the structural boundary tokens are constrained.

The catch: Anthropic imposes complexity limits on tool definitions when strict mode is enabled. Presumably, more complex schemas require more computation for the constrained sampler at each token position, and above a certain complexity threshold the overhead becomes prohibitive. Claude Code’s tool definitions exceed these thresholds (they include things like the full file path types, optional parameters, and schema for various code operations), which is why Claude Code doesn’t use strict mode and instead relies on its forgiving client-side error handling.

This creates a perverse situation: the harness that dominates the model’s post-training is also the harness that explicitly avoids strict mode. The model learns that strict schema adherence is not required. Other harnesses that want strict adherence must pay the complexity-limit cost.

The Broader Implications for Agentic Development

Ronacher’s conclusion deserves to be quoted directly: “Tool schemas are not neutral, at least not on Anthropic models. We like to pretend that a schema is an abstract contract and the model is a general reasoner that will follow it, but that might no longer be the case for some of the tools.”

This is a significant claim. The assumption underlying most tool-using LLM integrations is that the schema is the contract: define your tool correctly, document the parameters, and the model will call it correctly. This assumption treats the model as a general-purpose schema-following reasoner. Ronacher’s evidence suggests that this is no longer a safe assumption for Anthropic’s latest models — and possibly for other models that undergo extensive post-training in a single dominant harness environment.

What are the practical implications?

Schema distance from training distribution matters. If your tool schema closely resembles what the model was post-trained on, it will work reliably. If it diverges — even in logically equivalent ways — reliability will degrade in ways that are hard to predict and hard to debug. The failure mode (extra invented keys) can look like occasional noise rather than a systematic problem until you look at it statistically.

Strict mode is now effectively a correctness requirement for complex nested schemas on Anthropic models. If your tool uses nested arrays of objects with defined properties, you should enable strict mode and design your schema to fit within complexity limits. Alternatively, flatten your schema to match the shapes the model is most familiar with.

Retry logic is not sufficient mitigation. The typical response to occasional tool call failures is to add retry logic. But if the failure rate is 20% per attempt (as Ronacher observed in some sessions), retries are expensive. And if the model’s prior is strongly biased toward the wrong schema shape, retries will keep producing similar failures until you exhaust your retry budget.

Context length and thinking blocks affect failure probability. The bug is context-dependent. Long agentic histories with extended thinking blocks increase the failure rate. This makes the problem harder to reproduce in testing (short prompts don’t show it) and more likely to appear in production (where agents do long multi-step tasks).

Thinking block stripping is a partial mitigation. Removing extended thinking content from the conversation history before calling the model reduced the failure rate by about half in Ronacher’s tests. This is a non-obvious optimization that could be added to harnesses that experience this issue.

The Monopoly Harness Problem

There’s a broader structural issue embedded in Ronacher’s analysis that deserves explicit attention.

Claude Code has become, by any measure, one of the most widely used AI coding assistants in the world. Its popularity means that a significant fraction of Claude’s usage happens through Claude Code. If Anthropic uses production data or simulated Claude Code interactions in post-training, the model’s behavior increasingly reflects the Claude Code interaction pattern — not the general API pattern.

This creates a feedback loop: Claude Code becomes dominant → model is post-trained on Claude Code interactions → model becomes better at Claude Code but worse at other harnesses → other harness developers have worse experiences → they either switch to Claude Code or build workarounds → Claude Code becomes more dominant.

It’s not a conspiracy. It’s just optimization. Anthropic is optimizing for their best product’s user experience, which is a reasonable thing to do. But the consequence is that the API, which was supposed to be a neutral interface that any developer could use to build any tool-using application, is now quietly biased toward one specific tool ecology. And because Claude Code is closed-source and the post-training process is opaque, developers outside that ecology have no way to know how their schemas compare to the training distribution.

Ronacher frames this diplomatically: “The more post-training happens inside one dominant harness, the more every other harness will have to inherit its quirks.” The longer-term implication is that building a successful tool-using application on top of a closed model platform means understanding and conforming to the hidden conventions of the dominant harness, whether or not those conventions are documented.

What You Should Actually Do

If you’re building tool-using agents on top of Anthropic models today, here’s what Ronacher’s findings suggest you should do:

Enable strict mode for nested schemas. If your tool definition uses additionalProperties: false at nested array element level, this is now a correctness requirement, not a nicety. Test whether your schema fits within Anthropic’s complexity limits for strict mode. If it doesn’t, you’ll need to restructure.

Flatten schemas where possible. Pi’s nested edits[] array is elegant, but it’s off-distribution for a model post-trained on Claude Code’s flat old_string/new_string interface. Where you have design flexibility, prefer flat schemas over nested ones for edit-type operations.

Strip thinking blocks from long histories. If you’re running extended agentic sessions and seeing tool call failures, try stripping thinking content blocks from the conversation history before each new call. This won’t eliminate the problem but may significantly reduce failure frequency.

Log and measure schema conformance, not just task success. If your harness silently filters unexpected keys (as Claude Code does), you won’t see schema violations in your success metrics. Add explicit logging for unexpected keys in tool call arguments. This will tell you whether you’re experiencing this failure mode at some background rate.

Test with long agentic histories, not short prompts. Unit tests that make single-turn tool calls won’t catch this bug. Your integration tests need to simulate realistic agentic sessions with multi-step histories, because that’s where the failure mode emerges.

Consider schema-aliasing as a harness feature. Claude Code’s parameter aliasing — accepting both old_str and old_string, both path and file_path — is a practical accommodation to the model’s training distribution. If you’re seeing systematic errors in specific parameter names, consider whether adding aliases to your harness would reduce failure rates.

Why This Happened to the Smartest Models

One thing that makes this story counterintuitive is that it affects the best models, not the worst ones. Haiku, Anthropic’s smallest model, doesn’t show this regression. Older Opus didn’t either. It’s Opus 4.8 — the most capable, most expensive, most “intelligent” model in the family — that has the worst schema adherence for off-distribution tool shapes.

This makes perfect sense once you understand the mechanism. Intelligence, in this context, means having stronger priors. A more capable model has been trained more extensively and has developed more confident internal representations of “what an edit tool call looks like.” When it encounters a schema that differs from its training distribution, its confidence in its prior is high — high enough to override the formal schema definition. It knows what an edit call should look like, and it produces that, even when the given schema says otherwise.

A less capable model, with weaker priors, is more likely to follow the schema literally because it doesn’t have a strong enough prior to override it. The regression is, paradoxically, a consequence of getting smarter in the wrong direction.

This pattern — where increased capability in one domain causes regression in another — is a known challenge in RLHF. The model is being optimized for a specific task performance metric, and that optimization can come at the cost of other behaviors that weren’t explicitly part of the reward function. Tool schema fidelity was apparently not part of the reward function for Opus 4.8’s post-training, at least not for schemas that differ significantly from Claude Code’s.

The State of the Art in Mid-2026

Ronacher notes that the Codex models he tested (GPT-5.5 and earlier versions) do not show this type of regression. This is consistent with OpenAI’s different approach: the Harmony format, with its explicit in-band constrain markers, makes grammar-constrained sampling more tractable at serving time. If the model can signal “this is a constrained zone” as part of its output, the inference stack can apply constraints without the complexity-limit problem that Anthropic faces with strict mode.

There’s also a cultural difference. OpenAI has kept Codex’s harness open-source and has published documentation about the Harmony format and how tool calls are structured. Developers can examine what the model is trained on and design their schemas accordingly. Anthropic has kept both the model internals and the harness fully closed.

Neither approach is necessarily superior in all dimensions. But for developers building tool-using applications on the API, transparency about the training distribution is a meaningful practical advantage. If you know that the model was reinforced on tool calls in format X, you can make your tool definitions look like format X. Without that information, you’re flying blind.

The Takeaway

Ronacher’s investigation surfaces a failure mode that is likely affecting many production applications right now, silently, in the form of occasional retries that users never see. The 20% per-attempt failure rate he observed in affected sessions is high enough to cause real latency and cost problems. And the context-dependence means it gets worse exactly when users are doing the most complex, most valuable tasks — long agentic sessions, not simple single-turn requests.

The deeper lesson is about the relationship between training environments and API behavior. An API that presents a clean, documented interface may have behavior that is significantly shaped by an undocumented, closed, forgiving harness that the model was reinforced on. Schema compliance is not a purely logical property of the model; it’s a distributional property that depends on what the model was rewarded for during post-training.

Strict mode is the correct mitigation, and Anthropic should make it more accessible — higher complexity limits, more transparent documentation of what shapes it can and can’t handle. The model should also be evaluated against third-party harness schema compliance, not just task completion in the Claude Code environment. If the model is getting smarter in ways that make it worse for the rest of the API ecosystem, that’s a product quality problem, not just a theoretical concern.

Ronacher ends with a measured conclusion: “If the newest models get better at solving the task while getting worse at faithfully emitting an alternative tool schema, then the harness needs stronger guarantees somewhere.”

That’s correct. And right now, the harness is carrying more of that guarantee burden than is sustainable. The more capable the model gets at Claude Code’s own schemas, the more every other harness will need to engineer around the difference. That’s a tax on the entire developer ecosystem, and it compounds with every new model version.

For today, the practical advice is simple: use strict mode, flatten your schemas, test with long agentic histories, and log your schema conformance separately from your task success metrics. The bug is real, it’s in production, and it’s invisible if you’re not looking for it.

Original analysis: Better Models: Worse Tools by Armin Ronacher. Discussion on Hacker News.