Researchers Extracted Hidden Reasoning from Claude, GPT, and Gemini in Two API Calls. The Architecture Decision That Made This Possible Was Deliberate.

The paper dropped on a Tuesday. By Thursday morning it had 659 points on Hacker News and the comment section had turned into a slow-motion argument about whether this constitutes a vulnerability or a design choice. I want to be direct: it is a vulnerability, the providers knew the design had this property, and the interesting question is not whether it should be patched but why the architecture was built this way in the first place.

The paper is titled “Stealing Reasoning Traces from Proprietary LLM APIs.” The authors are from MATS Research, the ELLIS Institute at Tübingen, the Max Planck Institute, Snyk, and several affiliated institutions. The core finding is this: encrypted chain-of-thought blocks returned by frontier reasoning APIs from Anthropic, OpenAI, and Google can be replayed into weaker sibling models from the same provider, and those weaker models can be jailbroken into transcribing the encrypted content verbatim in plaintext.

The attack requires exactly two API calls. No access to model weights. No exploit of cryptographic weakness. No attack on the stronger model at all. The anti-distillation safeguards that Anthropic and OpenAI have built trigger on the strong model’s output, not on cross-model replay. The attack routes around the guardrails by never touching the guarded surface.

How the Encrypted Trace Architecture Actually Works

When you call Claude Opus 4 or GPT-5.x with extended thinking enabled, the API returns two types of content blocks: a visible text response and an encrypted thinking block. The encrypted block contains the model’s chain-of-thought—its actual reasoning process—encoded as a signed opaque blob. The signature and the block are both returned to the client.

Here is the design decision that creates the attack surface: the encrypted thinking block is designed to be sent back to the server in subsequent turns of the same conversation. This is not a bug. It is the mechanism by which stateless API serving can maintain continuity of reasoning across multi-turn exchanges without storing per-session state server-side. The client holds the reasoning state. The server can reconstruct context from it.

What the researchers discovered is that these blocks are portable across sessions, across users, and across models from the same provider family. An encrypted thinking block produced by claude-opus-4-8 can be injected into a conversation with claude-haiku-4-5. The signature validates. The weaker model accepts it as legitimate context.

From an engineering standpoint, this is almost elegant in its wrongness. The signature scheme authenticates the block as provider-produced content, which it is. It does not authenticate that the consuming model is the same model that produced it, or that the consuming session has any right to read the contents. The authentication and the authorization have been collapsed into a single primitive that only enforces one of the two properties you need.

The Two-Call Attack in Detail

The attack procedure is straightforward enough that I can describe it without reference to any exploit code:

1. Call a strong model (claude-opus-4, GPT-5.x, Gemini Ultra) on a task that requires extended thinking. Capture the encrypted thinking block from the response.
2. Construct a new conversation targeting a weaker model from the same provider family. Include the encrypted thinking block as assistant context. Add a user turn instructing the model to transcribe the content of the thinking block verbatim inside a wrapper element.
3. The weaker model, when jailbroken, outputs the plaintext reasoning from the stronger model’s encrypted trace.

The researchers verified this against 120 Codeforces problems. The correlation between the hidden thinking-token count reported by the strong model’s API and the token count of the decoded plaintext in the weak model’s output is high enough to confirm that the content being transcribed is genuine, not confabulated. The decoded reasoning tracks the hidden token count closely across all three providers.

I want to emphasize something that is easy to miss in the writeup: the jailbreak required is on the weak model, not the strong one. The stronger model’s anti-distillation defenses are never engaged. From the perspective of the production safety system, nothing unusual has happened. The strong model returned an encrypted trace. The weak model was jailbroken separately. These are two independent events from the provider’s monitoring perspective, which makes the attack difficult to detect through normal API anomaly detection.

The Secondary Finding Is More Alarming Than the Primary One

The primary finding—that proprietary reasoning can be extracted—is concerning for competitive reasons. A company using Claude Opus 4 to power a customer-facing reasoning system does not want competitors extracting its prompt engineering, its domain-specific reasoning patterns, or its intermediate analytical steps. Those are trade secrets expressed in inference, not in code, and this attack turns the provider’s own infrastructure into an exfiltration channel.

But the secondary finding is what kept me reading. The researchers collected 6,708 publicly available agent trajectories from GitHub and Hugging Face. These are sessions produced by Claude, GPT, and Gemini models that were uploaded to public repositories or datasets, often as examples, benchmarks, or agent evaluation traces. They still contained the encrypted reasoning blocks. The researchers applied their decoding pipeline to every signed block in the dataset.

From 315,320 reconstructed reasoning blocks, they recovered 704 distinct privacy artifacts in what they classified as genuine, non-benchmark user sessions:

Of those 704 artifacts, 64 appeared exclusively inside the encrypted reasoning blocks. They were not present anywhere in the visible session output. Someone reviewing the public trajectory would see no credentials, no PII. The secrets are invisible to the naked eye, encoded inside what looks like an opaque cryptographic blob. But they are there, and they are recoverable.

This is the data breach vector that nobody had fully articulated before this paper. Every developer who uploaded an agent trajectory to GitHub as a demonstration, every researcher who published benchmark sessions to Hugging Face, every company that shared evaluation datasets with consultants or auditors—if those sessions contained encrypted thinking blocks and if those thinking blocks captured sensitive content that was in the model’s context window, that content is now potentially accessible to anyone who runs the two-call extraction.

Why the Providers Built It This Way

I have spent some time thinking about why Anthropic, OpenAI, and Google all converged on essentially the same architectural decision: client-side storage of encrypted reasoning state with cross-turn replayability. The engineering motivation is real.

Server-side reasoning state storage at production scale is expensive. If every extended thinking session required the provider to maintain per-session state—storing the full chain-of-thought for each user, keyed to session ID, available for retrieval on subsequent turns—the infrastructure cost is non-trivial. You need storage, retrieval, TTL management, and session association logic. For workloads where most users do multi-turn conversations with reasoning enabled, this becomes a first-order infrastructure problem.

The client-hold pattern is architecturally cleaner for stateless serving. Each API call is self-contained. The server does not need to track which sessions exist or what reasoning state they have accumulated. This is the same pattern that JWT uses for authentication state: rather than maintaining a session store, you encrypt the state, hand it to the client, and have the client present it on each request. JWT has well-documented security properties and well-documented vulnerabilities. The reasoning block architecture has now received a similar security analysis, three years later.

The difference between JWT and these reasoning blocks is the portability boundary. A JWT is typically bound to an audience—it validates only for the service that issued it. The reasoning blocks are portable within the provider’s model family, which is a much larger attack surface. A JWT that validated for any microservice in a company’s internal infrastructure would be considered a serious design flaw. The reasoning blocks are the equivalent.

What the Providers Said and What They Actually Did

The paper does not describe the coordinated disclosure process in detail, which is notable. The researchers state that the attack works against current production systems from all three providers. They do not claim that patches have been deployed. As of the paper’s publication, the attack was live.

From an engineering perspective, fixing this is not trivial. You cannot simply stop returning encrypted thinking blocks to clients, because the multi-turn reasoning functionality depends on them. You cannot easily make the blocks non-portable without changing the authentication primitive, which would require updating client libraries, breaking existing integrations, and potentially re-architecting the serving infrastructure.

The most straightforward fix is model-binding: tie the signature to the specific model that produced the block, and have the server reject cross-model replay. This would break the attack immediately. It would not require changing the client-side storage pattern or the basic architecture. But it would require that the server validate the model identity on consumption, not just the signature integrity—a small change in validation logic that is large in deployment surface because it touches every reasoning API call.

A more aggressive fix is session-binding: tie the block to the session ID and user identity, and reject replay outside that context. This provides stronger guarantees but potentially limits legitimate use cases where applications pass reasoning context between different model tiers for cost optimization reasons.

I expect Anthropic to move first. Their public security track record includes faster response times to responsible disclosure than either OpenAI or Google, and the credential-in-reasoning-block finding creates reputational surface that accelerates internal prioritization. Whether “moving first” means a patch or a policy update or a documentation clarification is a different question.

The Practical Engineering Implications Right Now

While the providers work through whatever remediation they choose, practitioners building on these APIs need to understand several things that are not obvious from reading the headline.

First: do not put credentials in your system prompts or user context when extended thinking is enabled. This sounds obvious but it is not how people actually build agent systems. The most common pattern I see in production agent architectures is a system prompt that includes database connection strings, API keys for downstream services, and internal service URLs. The assumption is that the system prompt is the boundary of what the model “sees” and therefore what can leak. The reasoning block finding breaks that assumption: the model’s chain-of-thought may explicitly enumerate, analyze, and reference credentials from its context, encoding them in a trace that persists in the API response.

Second: treat encrypted thinking blocks as potentially sensitive data. If you are logging API responses for debugging, if you are uploading agent trajectories to version control, if you are sharing evaluation sessions with external parties—audit what encrypted thinking blocks are present in those artifacts and consider whether the content they might encode is acceptable for that audience.

Third: the anti-distillation defenses that Anthropic and OpenAI describe in their terms of service are not a complete defense against model capability extraction. This paper demonstrates one route around them. The architectural attack surface for capability leakage from frontier models is wider than the public discourse has assumed.

The Broader Design Lesson

The reasoning trace attack sits at an intersection that is underanalyzed in ML systems security: the gap between cryptographic guarantees and system-level trust assumptions. The encryption is real. The signature is real. The authentication is genuine. What is missing is the authorization model—the specification of who should be permitted to consume a signed block and under what conditions.

This failure mode is not unique to LLM APIs. It appears in OAuth implementations where tokens are too broadly scoped. It appears in JWT libraries where algorithm confusion attacks bypass cryptographic verification. It appears in Kubernetes where RBAC rules authenticate identity but fail to constrain what authenticated identities can do. In each case, a component that looks secure because it uses cryptographic primitives turns out to have a trust scope that is larger than intended.

What distinguishes the reasoning block case is the novelty of the artifact being protected. Credentials, sessions, and authorization tokens have been attacked this way before and the security engineering community has developed hard-won intuitions about where the authorization boundaries need to be. Encrypted model reasoning is new enough that those intuitions have not been applied systematically. This paper is the systematic application.

The implication for anyone designing AI infrastructure is that any novel artifact introduced by a new capability—whether that is reasoning blocks, tool call histories, agent memory snapshots, or multi-modal context caches—needs its own threat model. “We encrypt it” is not a complete threat model. “We encrypt it and we have defined a valid authorization scope for who can consume it and under what conditions” is closer to complete. The difference between those two statements is what this paper makes concrete.

What I Expect to Happen Next

These are falsifiable predictions, not opinions dressed as facts:

Within 90 days, at least one of the three major providers will deploy model-binding on reasoning blocks, making cross-model replay fail at the API level. This is the cheapest engineering fix that materially reduces the attack surface, and the credential-leak finding will drive internal urgency.

Within six months, at least one enterprise customer will disclose that encrypted reasoning blocks in shared evaluation datasets exposed internal credentials. The 704-artifact finding from public datasets represents a fraction of what exists in private or semi-private sharing contexts. The incident disclosure will come from someone who shared agent traces with a third-party auditor or vendor.

The anti-distillation terms in API provider agreements will be updated to explicitly address cross-model trace injection. The current terms were written before this attack class was publicly described. They will need to address whether a user who injects one provider model’s trace into another provider model’s context has violated any policy, and whether providers have monitoring obligations for this pattern.

Encrypted reasoning blocks will not be deprecated. The architectural benefit—stateless serving with multi-turn reasoning continuity—is too valuable to the providers’ cost structure to abandon. The fix will be in the authorization model, not the storage model. The encryption will remain. The portability will be restricted.

Finally: this paper will not be the last in this genre. The attack surface for AI system security is expanding faster than the security research community’s capacity to analyze it. Reasoning traces are one artifact class. Tool call logs, memory systems, retrieval-augmented context caches, and multi-agent message histories each have their own trust scope questions that have not been rigorously analyzed. The group at Tübingen and MATS has opened a research program, not closed one.

The Agent Trajectory Problem Is Larger Than One Paper Can Measure

I want to dwell on the dataset finding because I think it is receiving less attention than it deserves relative to the primary attack. The primary attack—extract a trace, replay it, jailbreak the weaker model—requires active effort. Someone has to decide to run it. The dataset finding is passive: it describes an attack surface that has already been created by normal developer behavior, sitting in public repositories, waiting to be scanned.

The researchers found 6,708 trajectories containing encrypted reasoning blocks in public GitHub and Hugging Face datasets. This is not a comprehensive search. It is a sample. The actual number of publicly available trajectories containing encrypted thinking blocks is almost certainly larger. GitHub has hundreds of millions of repositories. Hugging Face hosts thousands of datasets that contain agent traces, benchmark sessions, and example interactions. The researchers scanned what they could reach efficiently. They did not scan everything.

The developer behavior pattern that produces this situation is entirely predictable: you are building an agent system, you run a session that demonstrates some interesting behavior, you upload the session log to GitHub as documentation or to Hugging Face as a benchmark contribution. The log contains encrypted thinking blocks because you were using extended thinking. You did not know those blocks could be decoded. You did not know they contained recoverable content. You thought you were sharing a log of visible model behavior.

This is the same failure mode that produced the first wave of public API key leaks on GitHub circa 2014-2018. Developers were adding credentials to configuration files, committing those files, and pushing them to public repositories. The credentials looked like opaque strings—they did not obviously resemble sensitive data to someone unfamiliar with the format. It took years of tooling development—trufflehog, git-secrets, GitHub’s own push protection—to reduce the baseline rate of accidental credential disclosure on public repositories. We are about to begin the same process for encrypted AI reasoning traces.

The difference is that reasoning traces are not easily detected by pattern matching. A GitHub API key has a recognizable format: ghp_ followed by 36 alphanumeric characters. An AWS access key starts with AKIA followed by 16 uppercase alphanumeric characters. These patterns are well-known and trivially scannable. An encrypted thinking block looks like an opaque base64-encoded blob. There is no simple pattern to scan for it in a repository unless you know the specific encoding format used by each provider.

This means that the secret-scanning tooling that has been built to catch credential leaks will not, by default, catch reasoning block leaks. The tooling needs to be extended with provider-specific signatures for thinking blocks, and then repository owners need to run remediation on existing uploads. This is not a project that will be completed in 90 days. It is a multi-year remediation effort similar in scope to the API key cleanup that is still ongoing eight years after the problem was first widely documented.

Cross-Provider Analysis: Who Has the Largest Exposure

The paper does not rank the providers by exposure. It demonstrates the attack against all three and notes that the decoded reasoning closely tracks hidden thinking token counts across all of them, which is the fidelity verification. I am going to offer an analysis of relative exposure based on publicly available information about each provider’s reasoning architecture and deployment patterns.

Anthropic’s Claude Opus 4 and its extended thinking API have been available since mid-2025. The extended thinking feature is heavily used by enterprise customers for complex analytical workloads. The customer base is weighted toward developers and researchers who are more likely to share evaluation traces in public repositories. Anthropic also has the most explicit public documentation about extended thinking, which means developers understand they are getting reasoning blocks and are more likely to preserve them in logs. This combination—early availability, developer-heavy customer base, explicit documentation—likely means Claude thinking blocks appear disproportionately in public datasets relative to total usage.

OpenAI’s reasoning models have been available since GPT-o1 in late 2024, and the chain-of-thought exposure has been a topic of discussion in the safety community since then. OpenAI’s reasoning traces are partially hidden by design—the API has always returned only summaries of extended thinking to end users in the base tier. This architectural choice, made for competitive reasons, may have inadvertently reduced the surface area for reasoning block accumulation in public repositories: if the blocks are not returned to clients by default, they cannot be captured in session logs uploaded to GitHub.

Google’s Gemini reasoning architecture has been available in enterprise contexts since early 2025. Gemini’s extended thinking is more commonly used in the Vertex AI context, where enterprise security controls are more likely to be in place and developer trajectories are less likely to be shared publicly. The Vertex AI user base is dominated by organizations with data governance policies that would generally prevent sharing internal model traces. This likely means Gemini thinking blocks are underrepresented in the public dataset relative to usage.

None of this analysis changes the fundamental vulnerability. All three providers are affected. The exposure differences are about how much unauthorized data has already accumulated in public repositories, not about whether the attack works.

The Model Distillation Angle That Gets Glossed Over

The paper’s title focuses on privacy—”stealing” reasoning traces, recovering credentials and PII. But there is a second value category at risk that the paper addresses more briefly: the reasoning patterns themselves constitute intellectual property.

Frontier model providers have invested enormous resources in developing the chain-of-thought reasoning patterns that make their models effective at complex tasks. These patterns are not just emergent properties of training on large datasets. They are the result of specific training choices, RLHF fine-tuning, and post-training procedures. The specific way a model approaches a complex coding problem, the intermediate steps it checks, the failure modes it anticipates and avoids—these are differentiated capabilities that represent meaningful competitive advantage.

Anthropic and OpenAI have both published terms of service that prohibit using their model outputs to train competing models—this is the anti-distillation clause. The encrypted thinking block was supposed to be a mechanism for hiding the chain-of-thought from clients who might use it for distillation. Returning an encrypted blob instead of a readable reasoning trace was, in part, a business decision about protecting proprietary reasoning patterns.

The attack described in this paper bypasses that business protection entirely. Not just for one session, but systematically: anyone who has access to a corpus of encrypted thinking blocks—whether from public repositories or from their own API usage—can now decode those blocks and study the reasoning patterns of models they did not train. Whether this violates the terms of service is a legal question I will not resolve here. Whether it is the outcome the providers were trying to prevent with encryption is clear: yes, obviously, this is exactly what they were trying to prevent.

The practical consequence for model developers is that knowledge distillation from encrypted traces is now a real capability, not a theoretical concern. A team training a smaller reasoning model can use decoded traces from frontier models as training signal in a way that the API terms did not anticipate and the encryption was supposed to prevent. The next generation of efficient reasoning models may owe more to extracted Opus thinking than the training papers will acknowledge.

Historical Parallels in Systems Security

For engineers who have been around long enough to remember similar inflection points, this paper has the texture of several well-known predecessor moments in systems security.

The Heartbleed vulnerability in 2014 exposed the same fundamental failure: a piece of infrastructure that handled cryptographically protected data (TLS connections) was leaking content that should have been protected, not because the cryptography was broken but because the implementation of state management was wrong. Heartbleed leaked server memory content. The reasoning trace attack leaks reasoning memory content. In both cases, the underlying cryptographic primitives were functioning correctly. The failure was in the system design surrounding those primitives.

The SSRF (Server-Side Request Forgery) class of vulnerabilities has a similar structure: rather than attacking the protected resource directly, the attacker routes the request through a trusted intermediary that has legitimate access. The reasoning trace attack routes the extraction through the weaker model, which is a trusted intermediary from the perspective of the provider’s signing infrastructure. SSRF took years to be fully incorporated into standard security reviews for web applications. Reasoning trace injection will take similar time to become a standard consideration in AI system security reviews.

The JWT algorithm confusion attack is perhaps the closest structural parallel: an authentication mechanism (JWT with RS256 signature) could be exploited by switching the algorithm to HS256 and using the public key as the HMAC secret, because the validation code checked the signature but not whether the algorithm matched the issuer’s intent. The reasoning block attack exploits a similar gap: the validation checks the signature (provider-issued? yes) but not the authorization (is this the correct model to consume this block? unspecified).

In each of these cases, the vulnerability was apparent in retrospect but not obvious during design. The people who designed JWT, OpenSSL’s heartbeat extension, and SSRF-vulnerable web applications were competent engineers making reasonable decisions under incomplete threat models. The people who designed encrypted reasoning blocks at Anthropic, OpenAI, and Google are also competent engineers. The threat model was incomplete. This paper completes it.

Recommendations for Practitioners Building on Reasoning APIs Today

Given the current state—unpatched, affecting all three major providers, with credentials already leaked in public repositories—here are specific recommendations for teams running production systems that use extended thinking.

Audit your API logging infrastructure. If you are logging full API responses, including encrypted thinking blocks, determine what storage systems those logs go to, who has access, and what your retention policy is. The thinking blocks in those logs may contain content that should be classified at the same sensitivity level as your system prompt—which should be treated as confidential.

Review your system prompt construction for reasoning-enabled calls. Avoid placing live credentials, connection strings, or personal data directly in system prompts for sessions that use extended thinking. If your agent needs to access a database, pass the connection through a secure lookup at runtime rather than embedding the connection string in the system prompt. This is good hygiene regardless of the reasoning block vulnerability, but the vulnerability makes it urgent.

If you have shared agent trajectories externally—with auditors, benchmarking collaborators, dataset contributors, or as public examples—audit those trajectories for thinking blocks. If you find blocks, consider revoking any credentials that were in the model’s context during those sessions. The conservative assumption is that anything in the context window during a session with extended thinking enabled may have been encoded in the reasoning blocks.

For teams building agent evaluation frameworks: before publishing benchmark datasets that include API session traces, strip or null-out encrypted thinking blocks. A benchmark dataset that includes reasoning blocks from production sessions is a credential leak waiting to be discovered. The researchers found live credentials in benchmark datasets from this exact pattern.

Finally, watch the provider security advisories. When the patches come—and they will—they may require changes to how your applications handle multi-turn reasoning context. If your system currently relies on passing thinking blocks back to the API across turns, a model-binding patch will require you to ensure you are presenting blocks to the same model tier that produced them. Plan for this in your architecture review.

What This Means for the Future of Model Transparency

There is a deeper question underneath this paper that I do not see being asked in the Hacker News comments: what does this finding imply for the trajectory of chain-of-thought transparency?

The providers made a choice to hide reasoning. They presented it as a safety choice—preventing bad actors from extracting chain-of-thought for distillation, preventing users from reverse-engineering model behavior. But the encryption creates its own problems, as this paper demonstrates. The encrypted traces accumulate in API responses, in logs, in public datasets. They are opaque to humans but readable by the extraction attack. They contain sensitive content that users cannot audit before it is embedded in the trace.

The alternative—returning reasoning in plaintext but adding content policies and monitoring for distillation use—has different tradeoffs. Plaintext reasoning would allow users to audit what the model recorded about their data. It would allow secret-scanning tools to detect credential capture in real time. It would prevent the silent accumulation of opaque blobs in repositories. It would also make model behavior more transparent, which is arguably better for the ecosystem even if it is worse for any individual provider’s competitive position.

I am not arguing that providers should return reasoning in plaintext. The competitive and safety concerns are real. I am pointing out that the choice to encrypt was not without cost, and this paper has just made one of those costs legible. Every design decision in a complex system creates a threat surface somewhere. The reasoning block architecture chose to create the threat surface described in this paper rather than the threat surfaces created by plaintext reasoning. That was a reasonable choice under the threat model available at the time. The threat model is now updated.

The next version of reasoning APIs—whether from these providers or from new entrants—will be designed with this attack in mind. The question is what design choices the threat model update produces. Model-binding is the minimum viable patch. Session-binding is a stronger version. User-controlled deletion of reasoning state is a privacy-preserving extension. Full auditability of what the model recorded in reasoning traces is the most transparent option. Different providers will land in different places on this spectrum, and the choices they make will reflect their priorities—competitive protection, user privacy, regulatory compliance, engineering cost—more than any pure security analysis would suggest.

What I am confident about is that the encrypted reasoning block architecture as currently deployed is not the final form. This paper has ensured that.