Cactus Shipped a 14MB Agentic Model. The Architecture Paper Behind It Is More Interesting Than the Size.

A 14MB Model That Does One Thing Well. The Architecture Behind Needle2 Is More Interesting Than the Size.

Cactus shipped Needle2 last week: a 14MB binary, 45 million parameters at 2-bit compression, running full inference sessions in 28MB of RAM. On Hacker News it hit 419 points and 153 comments, which is a reasonable proxy for “this touched a nerve.” The reaction was split between genuine engineering excitement and fair skepticism — the web demo produced some memorably bad outputs, including setting a thermostat to “cool” mode when asked to make a room warmer, and locking the front door when a user typed “HN.”

I want to get past both the hype and the mockery. The failures are real but they are also beside the point if you misidentify what kind of system this is. And the architecture that powers Needle2 — built on a July 2026 paper from the same team titled “A Controlled Study of Attention-Only Transformers” — is doing something genuinely non-obvious that deserves careful reading.

The short version: Cactus found that deleting feed-forward networks from transformers and reallocating the freed parameters into attention depth costs almost nothing in perplexity when parameters are matched, but produces a model with a distinctive failure mode at parametric recall and a corresponding strength at context-grounded inference. That failure mode, normally disqualifying for a general-purpose assistant, happens to be exactly the right failure mode for a model whose only job is mapping natural language to function calls in a schema you define at runtime.

What Simple Attention Networks Actually Found

The Cactus paper (arXiv:2607.18363) is a controlled ablation study. The research question: do feed-forward networks — which hold two-thirds of a transformer’s non-embedding parameters — actually need to be there? The setup runs attention-only decoder transformers (they call them Simple Attention Networks, or SANs) against standard transformers matched on three separate axes: parameter count, training FLOPs, and depth. Training goes up to 105 billion tokens, from 6 million to 87 million parameters, 2 to 48 layers.

The naive result is bad. If you delete the FFN layers in place without adjusting anything else, the standard transformer leads by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. That is a meaningful difference. FFNs are doing real work and you cannot just cut them out.

The interesting result comes when you reallocate the freed budget. At matched parameters — meaning the attention-only model gets more layers and wider attention heads to compensate for the removed FFN parameters — the perplexity gap collapses to 0.006 nats. That is 0.27% of total loss. The paper reports this as reproducible to one part in ten thousand across seed pairs, and the gap shrinks across 5B, 30B, and 105B token budgets, holding near 0.02 nats across a 29x size range.

That is a striking result. The FFN layers, which dominate parameter counts in standard architectures, can be replaced by equivalent compute distributed across deeper attention stacks with almost no loss in aggregate perplexity.

But the paper does not stop at perplexity. It localizes where the remaining gap concentrates, and this localization is the key engineering insight for Needle2. Attention-only models are better at context-grounded answers — queries where the correct response is determined by the input context — and worse at parametric recall — queries where the correct response requires facts stored in model weights. The weight spectra analysis shows why: routing matrices (Q/K) crystallize early in training, content matrices accumulate rank slowly, and removing FFN layers relocates rank accumulation to the attention output projection rather than losing it entirely. The residual gap concentrates on low-context query prediction and localizes there entirely at the largest budgets tested.

QK-normalization, not FFN layers, is what keeps 48-layer attention-only stacks trainable. That is a secondary finding worth noting for anyone building in this space.

Why This Architecture Is the Right Failure Mode for Tool Calling

Here is the design argument that Cactus is implicitly making, and that the HN post statement makes explicitly: when your model’s only job is mapping natural language to function calls with typed parameters, you do not need world knowledge. You need context-grounded inference.

Consider what a tool call dispatcher actually has to do. A user says “set the living room lights to 60%.” The model receives: (1) that sentence, and (2) a schema defining available functions and their parameters. The correct output is a structured call to set_lights(room="living room", brightness=60). No fact retrieval required. No knowledge of whether 60% is a reasonable brightness. The answer is entirely determined by the input context — the sentence and the schema.

This is precisely the regime where SANs match or outperform standard transformers. The 0.006 nat difference in aggregate perplexity hides an asymmetry: SANs are better at context-grounded tasks and worse at recall tasks. For general-purpose language modeling this is a wash that turns net-negative when parametric recall matters. For tool dispatching, you opt into the strong side of the asymmetry and never need the weak side.

The other architectural component is Engrams, which Cactus describes as O(1) lookup with disk space as the main constraint. In the HN thread, the Cactus team confirmed that ablating Engrams showed they recover world knowledge similar to FFN but without compute expenditure — the ablation confirmed they add parametric storage without contributing to attention compute overhead. This is their mechanism for adding back a small amount of factual anchoring without paying the FLOPs cost of full FFN layers.

The result: 70 MFLOPs per token, versus 87 MFLOPs for a comparably-sized conventional transformer squeezed to the same parameter count, versus 164 MFLOPs for a standard-width transformer at the same depth. The “7x to 85x fewer FLOPs” claim in the HN post is relative to larger performant models, not to parameter-matched transformers — that framing is optimistic. The honest comparison is roughly 2x FLOP reduction against an equivalent-parameter standard transformer, which is still meaningful on a device with no NPU.

The Size Arithmetic and What It Implies

Let me work through the numbers. 45 million parameters at 2-bit quantization. Standard float32 would be 180MB. Float16 would be 90MB. INT8 would be 45MB. INT4 would be 22.5MB. Actual file size: 14MB. That implies something close to 2.5-bit effective storage including overhead, consistent with the stated 2-bit compression and packing alignment costs. The 28MB RAM figure at inference accounts for activations, KV cache (small for a shallow session context), and the quantized weights themselves.

The tradeoff at 2-bit is accuracy loss. The Cactus team chose 2-bit specifically because Needle1 users were deploying on microcontrollers, and moving to 4-bit would have doubled the size and excluded devices they had committed to supporting. This is a real constraint driving a real design decision, not an arbitrary choice. The ESP32-S3 and ESP32-P4 targets have 32MB of PSRAM, which is the binding constraint, and 2-bit keeps the session within it with 4MB to spare.

The token generation numbers: 500 tokens/sec on Raspberry Pi 5 (ARM Cortex-A76, no NPU), 400-1500 tokens/sec on VR headsets like Meta Quest 3S and Apple Vision Pro, 300-700 tokens/sec on sub-$200 Android phones without NPU. These are not fabricated — the WASM demo in the HN thread showed commenters reporting 310 tok/sec in browser, which is lower than native ARM but consistent with the overhead of running without hardware acceleration and through WebAssembly’s memory model.

For tool call dispatch, 500 tok/sec is massively faster than needed. A typical tool call response is 50-200 tokens. At 500 tok/sec that is 100-400ms, well within interactive latency for a voice assistant pipeline where the STT step already adds 200-500ms. The speed is not the main point; the power budget is. Every token is milliwatt-hours on a battery-constrained device, and 70 MFLOPs/token versus 500+ MFLOPs/token for the smallest performant transformer is a legitimate battery argument for always-on assistants.

What the Demo Failures Actually Tell You

The HN thread has a collection of demo failures worth cataloging systematically because they reveal specific engineering gaps rather than general model incompetence.

“Make it a little warmer in here” → set_thermostat to temperature 65, mode “cool”. Reasoning: “‘warmer’ implies need for cooling; set_thermostat with temperature 65 (typical warmth).” Confidence: 0. This is an antonym confusion failure. The model conflates “warmth” as a semantic concept with the “warm” setting on a thermostat, then inverts the mapping. At 45M parameters with no world knowledge, semantic nuance of this type is learned from training data distribution. The low confidence score is correct — the model knows it is uncertain — but the output is still wrong.

“Lock out the vacuum salesman at the front door” → lock_door with door=”front door vacuum salesman”. This is a named entity boundary failure. The model cannot reliably extract the device name from a sentence containing additional noun phrases. At this parameter count, parsing natural language compositionally is not a solved problem. The model is doing approximate template matching with soft boundaries.

“HN” → lock_door, front door, confidence 0. This is the expected behavior for out-of-distribution input: the model has a default action bias (locking the front door appears frequently in training examples) and falls through to it when no better match exists. The confidence gate is supposed to handle this: if your application threshold is above 0.6, you reject this output and escalate. Whether the confidence is calibrated is a separate question — calibration requires holdout evaluation on your specific tool vocabulary, which Cactus does not provide pre-computed.

“I’m hungover” → lock_door, reasoning: “‘hungover’ implies a security door.” Confidence: 0. This is the same failure mode as the previous case: default action with attached rationalization and a correct zero-confidence signal.

The pattern across all these failures is consistent: the model fails on queries that require semantic reasoning about the input rather than slot-filling against a tool schema. When the tool vocabulary is narrow and the query is unambiguous (“lock the front door,” “turn on the living room lights”), the model performs correctly. When the query requires compositional understanding (“make it warmer”), synonym resolution, or entity extraction from complex sentences, it fails.

This is not surprising for a 45M parameter model trained on tool call templates. What is worth noting is that the confidence gate is working as designed — all the obvious failures report confidence near zero. The question for any deployer is whether their application can tolerate false positives at low confidence thresholds, and whether they will invest in the fine-tuning pipeline Cactus provides to push performance up for their specific tool vocabulary.

The Competitive Landscape at This Size

The Cactus comparison table puts Needle2 against LFM2.5 230M and Apple Foundation Model. Both are 5x to 70x larger by file size. The benchmark being compared is tool call and mobile device use accuracy, measured at float16 for the competitors versus 2-bit for Needle2. “Trades wins” is the diplomatic framing for “sometimes Needle2 wins, sometimes it loses.”

The honest framing is that Needle2 is not competing with any of these models for general capability. It is competing with the alternative of deploying no AI at all on devices that cannot run even the smallest standard models. The 21 billion IoT devices Cactus cites — of which only roughly 1.5 billion are PCs and Macs — is the real market claim. Most industrial sensors, smart home controllers, budget Android phones shipped in Southeast Asia and Africa, Raspberry Pi-based projects: none of these can run LFM2.5 230M in production with reasonable power budgets. Needle2 can run on most of them.

Whether Needle2 is good enough at tool calling to be useful in production is a separate question from whether it is technically impressive. The answer depends entirely on application. For a narrow tool vocabulary with tight definitions — a thermostat with three actions, a door lock with one action — fine-tuned Needle2 is a plausible production choice. For open-ended natural language understanding of the type users expect from modern voice assistants, it will fail frequently enough to be frustrating.

The Confidence Gate Is the Most Important Feature Nobody Is Talking About

The confidence score is the system’s most important feature and the one that gets the least attention in the HN post. Henry Ndubuaku says the calibration threshold from experiments is around 60%, but acknowledges it requires tuning per workload. This is correct and undersells the engineering problem.

Calibration is the question of whether a 60% confidence score actually corresponds to 60% empirical accuracy on held-out test cases. Neural network confidence scores are systematically miscalibrated — well-documented since at least Guo et al. 2017 on modern neural network calibration. A model that reports “confidence 0.6” on a set of examples may in practice be correct 40% or 80% of the time on those examples, depending on how the confidence head was trained and what temperature scaling was applied at inference.

For a smart home use case where a false positive means your door locks when you said “make it warmer,” calibration matters considerably. For a commercial deployment where incorrect tool calls are expensive — robotics, medical devices, industrial control — calibration is the difference between a useful product and a liability. Cactus provides the confidence gate and sets a default threshold, but the calibration audit on your specific tool vocabulary is engineering work that every deployer needs to do independently.

The Cactus Hybrid architecture — where Needle acts as a local filter and escalates low-confidence calls to a cloud model — is the right long-term design for this problem. It solves the calibration question by design: if you are uncertain, escalate. The engineering question is where to set the uncertainty threshold, which requires per-deployment empirical measurement. Their combination with DeepSeek-v4-Flash for low-confidence escalation is a concrete example of the pattern working in practice.

From an architecture standpoint, Needle + cloud fallback is more interesting than Needle alone. It means you are running a 14MB local model for the high-confidence majority of queries, and a much larger cloud model only for the edge cases. The economics improve nonlinearly: if 80% of your queries are simple, unambiguous tool calls that Needle handles correctly with high confidence, you have reduced your cloud API spend by 80% while delivering the same accuracy on the 20% of hard cases. That is a genuine cost argument for enterprise IoT deployments at scale.

The IoT Gap Is Real and the Market Is Not Where People Think

I want to push back on one framing that appears in both the Cactus post and some HN comments, which treats “21 billion IoT devices” as an AI opportunity in the same sense that “1.5 billion PCs” is an AI opportunity. It is not.

Most industrial IoT sensors do not have a natural language interface and do not need one. A temperature sensor reporting to a SCADA system does not benefit from LLM integration. The use case where Needle2 is genuinely compelling is narrow: voice-driven control of devices in environments where cloud connectivity is unreliable or undesirable, and where users speak in natural language that needs to be mapped to a fixed tool vocabulary.

The concrete cases: smart home without cloud dependency, industrial equipment in RF-shielded environments, wearables with intermittent connectivity, budget phones in markets where cloud AI APIs are expensive relative to local income levels. These are real markets, each smaller than the IoT total would imply.

The hearing aid comment in the HN thread — from user anr0 — is the most interesting specific application I saw. A hearing aid processing speech in real time, recognizing commands like “boost bass,” “switch to crowd mode,” “answer call,” and dispatching to device firmware functions: this is exactly the narrow, high-volume, power-constrained use case where Needle2’s architecture makes sense. The device has no NPU, runs on a coin cell battery, and cannot phone home for every user interaction. The tool vocabulary is fixed (ten to twenty actions), the queries are short and unambiguous, and the cost of a false positive is recoverable (re-issue the command). That is the design space Needle2 is built for.

The robotics use case that appeared repeatedly in the HN thread is more complicated. Robotic control loops have real-time latency requirements that Needle2’s 100-400ms response time may not satisfy for fast motion primitives. More importantly, the failure modes I cataloged above — entity extraction errors, antonym confusion — become safety-critical in physical systems. A robot arm that moves the wrong direction because “make it warmer” triggered “cooling mode” in analogy is a different category of problem than an annoying thermostat. I would not deploy Needle2 in robotics without demonstrating calibrated performance on the specific robot’s command vocabulary, with confidence thresholds tuned conservatively.

Fine-Tuning and What It Actually Changes

The Cactus fine-tuning pipeline changes the capability equation significantly. The claim is that you can fine-tune Needle2 for a custom tool vocabulary on a Mac or PC in minutes to a few hours, using an automated data generation pipeline that synthesizes augmentations from a few sample inputs.

This is plausible at 45M parameters. Fine-tuning compute is trivial — training a model this size costs less than a month’s rent in GPU time, and fine-tuning a fraction of the parameters for a specific task should run in under an hour on a modern laptop. The data synthesis pipeline — using a larger model to generate variations of your canonical tool call examples — is the standard approach for specializing small models on narrow tasks. Model collapse concerns apply to large-scale training but are largely irrelevant at this parameter count and domain specificity.

The practical result is that Needle2’s out-of-the-box demo performance is not representative of production performance with fine-tuning. The thermostat antonym failure — “warmer implies cooling” — is the kind of systematic error that largely disappears with fifty examples of “make it warmer” mapped to temperature-up actions in the fine-tuning set. Domain-specific failures shown in the demo are mostly training distribution failures, not architectural limits.

From my experience working with small specialized models: the difference between a 100M parameter model fine-tuned on 500 domain-specific examples and the same model out-of-the-box is often larger than the difference between the 100M model and a 10B model in the same narrow domain. Task specialization at small parameter counts is underrated as a capability multiplier. The Cactus fine-tuning pipeline — if it works as described — is potentially the most important feature of the Needle ecosystem, more consequential than the SAN architecture or the size claim.

What the SAN Architecture Implies for Edge AI More Broadly

The broader implication of the attention-only transformer paper is not just that Needle2 works. It is that a large class of on-device AI tasks — anything where inference is context-grounded rather than knowledge-requiring — may be achievable with architectures significantly more compute-efficient than standard transformers at matched capability.

The standard framing of edge AI is: smaller, worse. You pay a capability tax for running locally. The SAN paper suggests a different framing: for tasks where parametric recall is not required, attention-only models match standard transformers at matched parameter counts, then you can quantize aggressively without sacrificing the capabilities that matter for your use case.

This reframes the tradeoff. You are not accepting worse tool calling to get smaller size. You are accepting worse general-purpose language modeling — a capability you never needed — in exchange for a model that fits in 28MB of RAM. That is not a compromise; it is a design choice that happens to align with the requirements of the application.

The H3-metal project from antirez — also on HN this week at 351 points — is working in related territory: native MiniMax-H3 inference for Apple Silicon, also an attention-heavy architecture optimized for specific hardware. The convergence of multiple teams toward attention-only or attention-heavy architectures for edge inference, each arriving independently from different hardware targets, is a signal worth tracking. The FFN layer is not sacred. The allocation of compute between attention and feed-forward layers is a design variable, and the optimal allocation depends on the task distribution you are targeting.

The pre-registration in the Cactus paper is also worth noting: they predicted a 0.02-0.05 nat gap on knowledge-dense web text, and measured 0.040. Pre-registered results that confirm the prediction are more credible than results discovered post-hoc. For a small team publishing a first architecture paper, that discipline signals that the result is real rather than cherry-picked.

What I Would Actually Do With Needle2

If building a product on this today: first, does the target device have 28MB of available RAM for the inference session? If no, Needle2 is not an option. If yes, is the tool vocabulary fixed and narrow — fewer than 20 distinct function signatures? If no, use a cloud API or a larger on-device model. If yes, can you generate 50-200 training examples per tool to support fine-tuning? If no, evaluate the base model on your specific tool calls before committing. If yes, fine-tune Needle2 on your vocabulary, calibrate the confidence threshold on a held-out set of at least 200 examples per tool, set escalation to cloud for anything below your calibrated threshold, and ship.

The escalation path is not optional. Needle2 alone is not a complete product for any application where incorrect tool calls have consequences. Needle2 as the first layer in a hybrid inference stack — handling the high-confidence majority of queries locally, escalating the remainder — is a credible production architecture. The local cost approaches zero in battery terms. The escalation cost is a cloud API call you would have paid for all queries in a fully-cloud architecture. Net result: better battery life, same or better accuracy on the full distribution, lower cloud spend.

The Language Stack Problem Nobody Is Solving

One thing the HN thread surfaced that deserves more attention: the language stack required to make Needle2 useful in a voice interface is not Needle2 alone. It is at minimum STT + Needle2 + TTS, and on a Raspberry Pi 5 that means three models running in sequence. The latency budget is roughly: Whisper tiny (~150-300ms for short utterances), Needle2 (~100-200ms at 500 tok/sec for a tool call), TTS synthesis with Piper (~200-400ms), plus system overhead. Total end-to-end: 450-900ms. That is usable for non-time-critical interactions but perceptibly slow for anything that feels conversational.

The power arithmetic is different. Whisper tiny at float16 uses roughly 50 MFLOPs/second of audio, Needle2 uses 70 MFLOPs/token, Piper TTS uses comparable compute to Whisper. For an always-on device that processes voice queries ten times per hour, the majority of power consumption is idle listening (the wake word detector, which runs continuously) rather than any of these three models. Needle2’s contribution to total device power is small in the full stack, but it is the piece that determines whether the answer is right once the wake word is detected.

The integration work to connect these three components — particularly on constrained hardware without an OS that manages audio pipelines gracefully — is where projects like this tend to stall. The models work; the plumbing is hard. The Home Assistant ecosystem has standardized Wyoming protocol for this pipeline on Pi-class hardware, and Cactus’s best growth path is probably building an official Wyoming-compatible Needle2 integration rather than shipping standalone Python packages. That is the distribution channel for the embedded enthusiast market.

The commercial embedded path — where a hardware vendor bakes Needle2 into a product’s firmware — is a different integration problem. The vendor needs a C or C++ runtime (Cactus Engine appears to be this), build toolchain integration, memory layout guarantees, and support for their specific RTOS or bare-metal environment. That is real systems work that is invisible from the Python package and the web demo, and it is the work that determines whether Needle2 ships in products or stays in hobbyist projects.

Falsifiable Predictions

By the end of 2026: at least one commercial smart home product ships with Needle2 or a model in its weight class embedded in firmware, marketed explicitly as “AI without cloud.” The hearing aid application appears as a published benchmark or prototype. The Cactus Hybrid commercial offering reaches paying customers, and confidence calibration becomes their primary support issue for enterprise deployments.

By mid-2027: the SAN architecture from the Cactus paper is replicated by at least two other research groups targeting different hardware profiles. At least one replication finds that 4-bit SAN models outperform 2-bit SANs on tool calling quality while fitting on ESP32-P4-class hardware, shifting the Needle default quantization. The ollama integration that Henry Ndubuaku confirmed is coming ships and drives the majority of developer adoption — because most developers will interact with Needle2 through existing tooling rather than the Python package.

The benchmark that does not yet exist but should: fine-tuned Needle2 versus a 1B parameter quantized model (phi-3, Gemma-2) at INT8, head-to-head on calibrated accuracy and confidence gate performance for a specific smart home tool vocabulary. My prediction: fine-tuned Needle2 wins on confidence calibration because its confidence head has less noise from irrelevant capabilities, and loses on compositional natural language understanding. The crossover point — the vocabulary complexity and query distribution at which a larger model becomes worth its resource cost — is the number the embedded AI industry needs and does not yet have. Building that benchmark would be more useful than another demo.