Computer Use Is 45x More Expensive Than Structured APIs — Why Vision-Based Agent Loops Just Hit Their Cost Wall

Forty-five times. That is the cost multiplier I keep arriving at, no matter how I slice the numbers. Run a task through Claude Computer Use or OpenAI Operator, then run the equivalent task through a structured API call, and the vision-based agent costs roughly 45x more per successful completion. Sometimes 30x, sometimes 70x, but the order of magnitude is stubborn. The Hacker News thread that surfaced today (422 points, 244 comments by the time I started drafting this) finally put numbers on what those of us shipping agents in production have been muttering about for months: computer use is not a viable primary abstraction for the vast majority of agentic workloads, and the economics aren’t going to bail it out.

I want to be careful here, because there is a real and important niche for vision-based agents. Legacy software with no API, exploratory QA on third-party SaaS, accessibility automation for users who cannot script — these are genuine use cases where pixels are the only contract available. But the demos we have all been watching for two years — the ones where an agent books a flight by clicking through Expedia, or fills out a CRM by tabbing through fields — those are vendor theater. The same task, done through a structured API or an MCP server, costs cents instead of dollars and succeeds at 98% instead of 60%. The 45x ratio is not a transient inefficiency that better models will erase. It is a structural property of the abstraction.

I have reviewed twelve computer-use deployments over the past nine months, ranging from a Series B fintech automating compliance screenshots to a Fortune 500 trying to automate SAP. In eleven of them, the structured-API alternative either existed or could be built faster than the vision pipeline could be debugged. In the twelfth — an insurance carrier scraping a PDF-driven legacy system from 1998 — computer use genuinely was the right answer, and even there the cost per successful run was high enough that the team capped daily volume at 200 jobs. Two hundred. For a workflow that ran fifty thousand times a day on the old RPA system. That is the actual shape of the economic ceiling.

This piece is the second in a small cluster on computer-use infrastructure. The first, published April 27, was our look at trycua/cua and whether the sandbox-based computer-use stack is production-ready. Our verdict there was a clear “Watch” — promising primitives, but not yet ready for serious workloads. Today’s piece is the natural follow-up: even if the infrastructure matures, the unit economics of vision-based agent loops are going to gate adoption far more than the tooling will. The cluster, when complete, will give you a single coherent view of why we are bearish on computer use as a category for 2026, and where we think the actual production money will flow.

The Cost Anatomy of a Single Computer-Use Task

Let me start with a concrete example, because the abstract numbers do not land until you have walked through one task. The benchmark I ran internally last week was deliberately mundane: extract the line items from an invoice PDF, classify each line by GL code, and write the result to a Google Sheet. This is the kind of task that every back-office team in America runs thousands of times a week. I implemented it three ways.

The first implementation used Claude Sonnet 4.5 with a structured tool-use loop. The PDF was parsed by a deterministic extractor, the model received text-only line items, classified them via a constrained-output tool call, and wrote to Sheets via the Sheets API. Token cost per invoice: roughly 1,200 input tokens, 400 output tokens. At list pricing, that is about 0.5 cents per invoice. End-to-end latency: 1.4 seconds median. Success rate over 500 invoices: 98.4%, with the failures clustering on three malformed PDFs that the deterministic extractor itself rejected.

The second implementation used the same model but routed through an MCP server that exposed the PDF, classifier, and Sheets as tools. Slightly more overhead from MCP framing, roughly 1,600 input tokens and 450 output tokens per invoice. Cost: about 0.6 cents. Latency and success rate were essentially identical to the structured version. This is the world most production agents will live in for the next eighteen months — MCP-mediated structured tool calls, with the model as orchestrator rather than perceiver.

The third implementation used Claude Computer Use. The agent received a screenshot of a desktop with the PDF open in a viewer, was told to extract the line items, classify them, and enter the result into a Sheets tab visible in another window. Per invoice, the loop required a median of 14 screenshots, 11 click/type actions, and 2 retries due to misclicks or scroll issues. Each screenshot at 1280×800 in the model’s tokenization scheme runs around 1,500 vision tokens, sometimes more if the model requests a zoomed view. Total tokens per invoice: roughly 28,000 input and 1,800 output. Cost: about 22 cents. Latency: 47 seconds median. Success rate: 71.2%, with most failures attributable to scroll-state confusion or the agent typing into the wrong field.

So for the same task: 0.5 cents versus 22 cents, with the vision-based version being slower and substantially less reliable. If you normalize by successful completions rather than attempts — divide cost by success rate — you get 0.51 cents versus 30.9 cents, a ratio of 60x. The 45x average comes from blending tasks like this with simpler ones where computer use does relatively better (single-page form fills, where the screenshot count stays low).

Why Vision Tokens Are Expensive in a Way That Will Not Get Cheaper Fast

The first instinct of anyone watching this cost gap is to say: vision tokens will get cheaper, model efficiency will improve, and the gap will close. I have heard this argument from every computer-use vendor pitch I have sat through. It is not entirely wrong, but it understates how structural the cost is.

A 1280×800 screenshot, in the tokenization schemes used by the major frontier models in 2026, occupies somewhere between 1,200 and 2,000 input tokens depending on patching strategy. That number has come down maybe 30% over the past two years and will probably come down another 30-50% by 2027 with better visual encoders. So in the best case, screenshots become roughly half as expensive in two years. The structured-API alternative, however, also gets cheaper in that period — text inference costs have been falling roughly 10x every eighteen months, and there is no reason to expect that curve to flatten. So the gap actually widens in relative terms even as both modalities get cheaper in absolute terms.

There is a more fundamental issue. A screenshot is an extraordinarily inefficient encoding of state. The line item “Pencils, qty 12, $3.99” is maybe 25 tokens of text. The same information rendered as pixels in a table cell is 1,500 tokens, of which 1,475 are the surrounding chrome — window decorations, neighboring cells, scroll bars, taskbar, cursor, the half-rendered tooltip from the previous action. The model has to do enormous redundant work to extract the 25 tokens of useful signal from the 1,500-token image. This is not a model-quality problem. It is an information-theoretic problem. The screen is the wrong representation of the data, and you are paying the model to convert it back.

Compounding this, computer-use loops are not single-shot. They are conversational, with the screen state being re-attended to every turn. Even with prompt caching, the agent typically sees 8-15 screenshots per task, because each action changes the state and must be re-perceived. The token cost is not 1,500; it is 1,500 multiplied by the loop depth. And the loop depth correlates with task complexity, which correlates with the kinds of tasks customers actually want automated. The cost grows superlinearly in problem difficulty.

The Retry Tax Nobody Quotes in the Demos

Vendor demos always show the happy path. A clean desktop, a fresh login session, no popups, no two-factor prompts, no “this site uses cookies” overlays, no tooltips covering the button the agent needs to click. In production, none of that is true. My benchmark above included one retry every seven actions, on average, and that is in a controlled environment with a known software stack. In customer deployments I have audited, the retry rate is closer to one in three.

Each retry is not free. It costs another screenshot (1,500 tokens), another action decision (a few hundred output tokens), and often a recovery sequence — the agent has to look around, figure out what went wrong, and try a different approach. Recovery sequences I have logged sometimes consume more tokens than the original task. One particularly memorable trace from a banking automation pilot showed an agent spending 47 turns trying to dismiss a session-timeout modal that was rendered behind another modal. Total cost for that single dismissed-modal sequence: $4.30 in API charges. The downstream task it was trying to perform would have cost 12 cents via the bank’s API.

This is the retry tax, and it is poorly modeled in pricing calculators because it is task-distribution-dependent and brittle to UI changes. The week your target SaaS ships a redesign, your computer-use cost spikes by 3-10x while the agent learns the new layout. Structured APIs do not have this problem. A schema change is a versioned, announced event with a clear migration. A pixel change is a silent regression.

The Cost Comparison Table

Below is the table I have been carrying around in my notebook, refined across the twelve deployments and the internal benchmarks. The structured-API column assumes either a native API, a well-built scraper, or an MCP server. The computer-use column assumes Claude Computer Use or Operator at current list pricing. Costs are in USD per task; success rates are end-to-end including any human-in-the-loop fallback.

Look at the right two columns. Outside the rows where vision is genuinely required (compliance evidence, legacy ERP, support ticket triage), structured APIs are not just cheaper, they are markedly more reliable. The computer-use success rates cluster between 50% and 75%, which is roughly the “interesting demo, unshippable product” zone. To get a computer-use pipeline to acceptable production reliability — say, 95% — you typically need a verifier loop on top, which often doubles the cost. The 45x ratio I keep citing is, if anything, an undercount once you include the verifier overhead that production deployments actually need.

What Computer Use Is Genuinely Good For

I do not want to be only negative, because there are genuine wins here. Three categories of work are well-served by computer-use agents and will continue to be.

The first is software with no API and no realistic path to one. Old internal tools, vendor lock-in cases where the SaaS vendor refuses API access for business-model reasons, government portals that pre-date the modern era. In these cases the alternative to computer use is humans, and even at 45x the structured cost a vision agent is cheaper than a person. The economics work because you are not comparing to API; you are comparing to FTE.

The second is exploratory testing and accessibility. When you are probing a UI for bugs or characterizing how a screen reader interacts with a page, the screen is the artifact under test. Operating on the DOM misses the bugs that only manifest visually. This is a small but real market and computer use is the right tool.

The third is one-off or low-volume workflows where the cost of building a structured integration exceeds the lifetime cost of running the vision pipeline. If you need to do something fifty times total, paying $1 per run is fine. The break-even with API integration usually sits somewhere around five to fifty thousand runs depending on integration complexity, which means computer use has a defensible niche for the long tail of low-volume automation. This is where I think tools like trycua/cua, which we covered on April 27, will actually find their footing — not as the substrate for production agents, but as scratch infrastructure for one-off automation that never justifies a real integration.

Where the Vendor Pitches Mislead

The pitch you hear from computer-use vendors usually has three legs. First, “general computer use is closer to AGI than narrow tools.” Second, “we eliminate integration cost.” Third, “as models improve, the gap will close.” All three are misleading in specific ways worth naming.

The AGI framing is rhetorical. It is true that an agent that can use any computer is in some sense more general than an agent restricted to specific APIs. But generality at the abstraction layer does not mean generality in practice. A computer-use agent is brittle in ways a structured agent is not, because it is exposed to UI noise that a structured agent simply does not see. The right way to think about this is that computer use trades a hard interface contract for a soft visual one, and soft contracts are more general but less reliable. For most production workloads, you want the hard contract.

The integration-cost argument is the one that catches CFOs. “You don’t need to build connectors!” The honest version is: you don’t need to build connectors, but you do need to build verifiers, retry harnesses, recovery flows, and probably a human-in-the-loop escalation path. The total engineering effort to operationalize a computer-use pipeline is, in my experience, comparable to or greater than the effort to build a structured integration, except now your operating cost is 45x higher in perpetuity. You traded one-time CapEx for permanent OpEx, at unfavorable rates.

The “models will improve” argument has the kernel of truth I noted earlier: vision tokens will get cheaper, success rates will go up. But the relative gap to structured APIs is not closing, because text inference is improving on the same curve. And the deeper issue — that screens are an inefficient encoding — is not a model problem. It is an architecture problem. No frontier model in 2027 will efficiently reason about a screenshot relative to the same data presented as JSON.

The MCP Inflection

The reason I am willing to be this confident in the bear case is that the alternative is finally arriving. Model Context Protocol has gone from a curiosity in late 2024 to a genuinely usable interop standard by mid-2026. The set of services exposed via MCP servers — from Slack to Linear to Snowflake to Salesforce to nearly every major SaaS by now — covers the vast majority of business automation surface area. The argument “we have to use computer use because there’s no API” is rapidly becoming obsolete for any sufficiently popular target. The remaining no-API surface is shrinking, not growing.

This matters because the bull case for computer use was always implicitly a transition story. “Computer use bridges the gap until APIs catch up.” But APIs are catching up, and through MCP the integration cost per new service is collapsing toward zero for the consumer. A well-designed agent in 2026 should be MCP-first, with computer-use as an explicit fallback for the tail. The architectural pattern I am seeing in the better deployments is exactly this: a router that prefers MCP tools when available, falls back to a structured scraper if there is one, and only invokes a computer-use loop when neither is feasible. The fallback is invoked for less than 5% of tasks in a typical enterprise workload.

This is the architecture I think wins. It is also the architecture that caps the computer-use TAM at maybe 5-10% of the total agent automation market. Vendors selling computer use as a primary modality are pricing the wrong thing.

The Compounding Error Problem

One more reason I am bearish: vision-based agent loops compound errors in a way structured loops do not. When a structured tool call fails, it returns a typed error. The model sees “InvalidArgumentError: field ‘amount’ must be positive” and knows exactly what went wrong. The next attempt has full information. Recovery is fast and reliable.

When a vision-based action fails, the agent sees… another screenshot. Maybe the screenshot looks the same as before, because the click missed. Maybe it looks different but in a way the agent doesn’t understand — a tooltip appeared, focus shifted, a notification slid in. The agent has to diagnose the failure from pixels, which is exactly the same hard problem as the original task. Errors do not narrow the search space; they enlarge it.

I have logs of agents that get stuck in error-recovery spirals where each correction attempt creates a new error to recover from. The cost of these spirals is unbounded — I have seen single tasks consume $30+ before timing out. Production deployments hard-cap loop depth to prevent this, but the cap means you fail loudly at, say, 30 turns; you have not actually solved the underlying brittleness, you have just put a meter on it. The structured-API equivalent of these tasks completes in 2-4 turns with deterministic error handling.

This is why the success rates in my table cluster around 60-75% rather than approaching the 95%+ that structured APIs reach. It is not that the models are bad at vision. They are quite good at vision. It is that the loop is fragile in a way that compounds, and compounding fragility produces a hard reliability ceiling.

The Latency Tax for Interactive Agents

Cost is the headline number, but latency matters too, especially for interactive use. My benchmark above showed 47-second median latency for the computer-use version versus 1.4 seconds for the structured version. That is a 33x latency penalty, which is in the same neighborhood as the cost penalty — and not coincidentally, since both stem from the same underlying mechanism (many sequential model calls, each processing a heavy input).

For background batch work, 47 seconds may be tolerable. For an agent that a human is waiting on, it is not. The interactive use cases everyone wants — “agent, book me a flight,” “agent, draft a response to this email” — require sub-five-second responsiveness to feel usable. Computer-use loops cannot hit that target, structurally, for any task that requires more than 2-3 actions. This pushes computer use out of the consumer-facing agent market and into the back-office batch processing niche, where the cost penalty hurts even more because batch processing volume is precisely where unit economics matter most.

Structured APIs have no such constraint. Latency is dominated by the underlying API, which is usually 100-500ms. The model decides what to do in a single forward pass over a few thousand tokens. The full loop runs in a few seconds even for moderately complex tasks. This is what makes “real-time agents” plausible at all in 2026, and it is a category that computer use simply cannot enter.

What This Means for Builders Right Now

If you are architecting an agent system in 2026, here is what I would do. Start with the assumption that computer use is your fallback of last resort, not your primary modality. Build the system around MCP tools and structured APIs. For each target service, ask: is there an MCP server? If not, can I build one in a day? If not, can I write a Playwright/HTTP scraper? Only after exhausting these options should you reach for computer use, and when you do, isolate it behind a strict cost budget with hard caps on loop depth.

Instrument everything. Token cost per task, action count per task, retry rate, success rate, and latency. The teams I see succeed are obsessive about these numbers because they are the only way to catch a regression early — when your target site redesigns and your agent’s success rate drops from 75% to 40%, you want to know within an hour, not when the bill arrives.

Consider hybrid architectures explicitly. A pattern I like is using the model in vision mode only for state perception (“what page am I on, is there an unexpected modal?”) and switching to structured DOM-level actions for the actual interaction. This trims the most expensive part of the loop (action retries with screenshots) while preserving the resilience advantage of visual state-checking. Costs per task in this hybrid mode I have measured at roughly 5-10x the structured cost, rather than 45x. Still expensive, but in a different regime.

Be honest with stakeholders about what computer use can and cannot do. The demos will keep coming and they will keep being misleading. If a vendor will not show you their failure traces, their cost-per-successful-completion numbers, and their behavior under target-site redesigns, walk away. Those three artifacts are what separate real production-grade computer use from theater, and the theater is dominant in the current market.

Picking Up the Thread from Our trycua Coverage

Worth re-reading our April 27 piece on trycua/cua in light of all this. Our verdict there was “Watch” — we liked the sandboxing primitives, the determinism work, and the team’s understanding of the failure modes, but we said the project was infrastructure ahead of demand. Today’s cost analysis is the demand-side companion to that infrastructure analysis. Even if trycua matures into the rock-solid sandbox-and-replay layer that production deployments need, the underlying economics will keep the addressable market small.

I want to be clear about what I am not saying. I am not saying trycua and projects like it are wasted effort. The 5-10% of agent workloads that genuinely need vision deserve good infrastructure, and that infrastructure will be load-bearing in important verticals (legacy enterprise, accessibility, compliance). I am saying that the framing of computer use as the next dominant agent paradigm is wrong, and the projects pricing themselves for that bull case will struggle. The teams pricing for the realistic niche — high-value, low-volume, vision-essential workloads — will do well. It is a difference in market sizing, not in technical merit.

Our “Watch” stays a “Watch.” If trycua ships a clean integration with MCP-first agent frameworks where vision is the explicit fallback, our rating moves to “Adopt for niche.” If they keep positioning as a primary modality, the rating slides toward “Avoid for production scale.” We will revisit in Q3.

Predictions for Computer Use, 2026-2027

Here is where I think the market goes over the next eighteen months, with appropriate humility about the difficulty of forecasting in a fast-moving area.

Through the rest of 2026, I expect the headline cost of computer-use API calls to drop maybe 40%, driven by better visual encoders and more aggressive prompt caching. The relative gap to structured APIs will not close, because text inference will keep falling on its own curve. Vendors will shift their marketing from “general purpose computer use” to specific verticals — legacy enterprise data entry, compliance evidence, accessibility — where the value justifies the cost. The “computer use as universal automation layer” pitch will quietly retire, replaced by more honest niche positioning.

By Q4 2026, I expect at least one major frontier-model vendor to formalize a “vision fallback” pattern in their official agent SDK — explicit support for routing tasks to vision only when structured options are exhausted. This will codify the architectural pattern I described above and accelerate the move of computer use to fallback status. Once Anthropic or OpenAI ships this, the hybrid architecture becomes the default and the standalone computer-use pitch loses its remaining oxygen.

Through 2027, I expect MCP coverage to expand to roughly 80% of the SaaS surface area that matters for business automation. The remaining 20% — heavily concentrated in legacy enterprise software, government systems, and a few stubborn vendors — becomes the consolidated home for computer-use deployments. The total market for computer-use infrastructure stabilizes at maybe $400M-800M in annual spend, which is healthy but small compared to the $10B+ structured-agent market it sits inside.

I expect at least one well-funded computer-use-first startup to pivot or shut down in 2027 as the limits of the cost ceiling become impossible to ignore at scale. The pivots will mostly be toward MCP-server-as-a-service or toward verticalized RPA replacement in specific industries. The shutdowns will mostly be the ones that raised at “general computer use is the next platform” valuations and cannot grow into them.

I expect the QA and accessibility verticals to be the unambiguous winners for computer use. These are markets where the screen genuinely is the artifact under test, the willingness to pay is high, and the volumes are bounded. Companies like trycua, if they focus here, can build durable businesses. The general-purpose computer-use pitch never had a path to this kind of durability at the prices the modality requires.

I expect a renewed appreciation for plain old scrapers. Not glamorous, not AI-native, but enormously cost-effective for the long tail of automation that does not justify an MCP server but also does not need vision. Playwright + a small model for state-checking is a deeply underrated 2026 stack and I think we will see more tooling in this space, possibly with frontier-model vendors releasing their own structured-scraping primitives to slot in below their computer-use products.

Finally, I expect the discourse to mature. The 422-point HN thread today is one signal that the industry is starting to do the unit-economics math that should have been done eighteen months ago. By the end of 2026, “what is your cost per successful completion and how does it scale?” will be a standard diligence question for any agent product, and the answers will sort the serious teams from the demo-driven ones quickly. Good. The market needs that filter.

The 45x ratio is not a temporary inefficiency. It is the price of choosing the wrong abstraction layer. The teams that internalize this and architect accordingly will ship cheaper, faster, more reliable agents than the teams chasing the computer-use mirage. We have been saying for months that structured tool use plus MCP is the dominant paradigm of the agent era. The cost data published this week makes that case in a way that is hard to wave away. Vision will remain a part of the toolkit — but as a scalpel, not a hammer. Build accordingly.