Qwen 3.6 27B Is Called the Sweet Spot for Local Dev. We Ran the Numbers. Here Is What That Actually Means.

A blog post from Quesma hit the top of Hacker News on June 29th with 1,017 points and 660 comments. The headline: “Qwen 3.6 27B is the sweet spot for local development.” The author, Piotr Migdał, ran it on a MacBook Pro M5 with 128GB of RAM and came away convinced this is “the first local model that actually makes sense as a general intelligence.”

The 660 comments that followed told a more complicated story.

That is the starting point for this piece. Not “is Qwen 3.6 27B good?” — it is, plainly. But what does “sweet spot” actually mean when you pull the numbers apart? Who is it a sweet spot for? What hardware assumptions are baked into the claim? And what do the people actually running it daily say when they disagree with each other in public?

ThesisBench is a benchmark-driven blog. So let us benchmark the claim itself.

What Qwen 3.6 27B Actually Is

Qwen 3.6 is Alibaba’s sixth-generation open-weight model series. The “27B” refers to 27 billion parameters in a dense architecture — every parameter is active for every token. This matters because Qwen also released a mixture-of-experts sibling: the Qwen 3.6 35B A3B, where “35B” is the total parameter count but only roughly 3 billion are active per token during inference.

Dense vs. MoE is the first trade-off you have to understand before any “sweet spot” claim makes sense.

A dense 27B model activates all 27 billion parameters for every forward pass. A MoE 35B A3B model activates approximately 3 billion per pass. This makes MoE dramatically faster at token generation — Migdał’s own benchmarks on an M5 MacBook with 128GB show 85–93 tokens/second for the 35B A3B vs. 17–18 tokens/second for the 27B dense. With multi-token prediction (MTP) enabled, those figures move to 105 vs. 32 tokens/second respectively.

So why does the author prefer the dense 27B at roughly one-third the generation speed? His answer: “I’d rather generate a third as much code, but of higher quality.”

That preference is defensible. But it is a preference, not a universal truth — and the comment thread made that tension explicit.

The Hardware Assumption Nobody Leads With

Migdał’s benchmark machine is a MacBook Pro M5 with 128GB of unified RAM. That machine currently starts at $6,699 USD. The article does disclose this, though late and without emphasis.

This is load-bearing context that changes almost every claim in the piece.

The M5 Max’s unified memory architecture gives it approximately 614 GB/s of memory bandwidth. For LLM inference on a dense model, memory bandwidth is the primary bottleneck once the model fits in memory — the GPU must stream billions of weights from DRAM on every forward pass, and the speed at which it can do that determines tokens per second. Apple Silicon’s unified memory means the GPU and CPU share the same high-bandwidth pool, which is why a Mac with 128GB of unified RAM substantially outperforms a PC with 128GB of DDR5 and a discrete GPU with only 24GB of VRAM.

The HN thread surfaced this hardware calculus immediately. One commenter noted the M5 Max achieves 614 GB/s of memory bandwidth. An M4 Mac Mini with 64GB — Apple discontinued the configuration on June 25th, just four days before this thread — runs at 273 GB/s, less than half. A single RTX 3090, the GPU most commonly proposed as the budget alternative, has 24GB of VRAM at 936 GB/s of bandwidth but can only hold a portion of the 27B model at 8-bit precision.

The actual performance picture by hardware tier, assembled from verified user reports in the thread:

Hardware	RAM / VRAM	Bandwidth	27B 8-bit tokens/s	Cost (approx.)
MacBook Pro M5 Max 128GB	128GB unified	614 GB/s	18 (llama.cpp), 32 with MTP	$6,699+
MacBook Pro M5 Pro 48GB	48GB unified	307 GB/s	12–14 (Q4, MLX)	~$3,499
Mac Mini M4 Pro 64GB	64GB unified	273 GB/s	~25–30 (35B A3B)	~$2,800–3,000 used
DGX Spark / GB10	128GB	~273 GB/s effective	~50 (35B A3B, vLLM+MTP)	$3,999–4,500
RTX 5090 (32GB VRAM)	32GB	1,792 GB/s	~67 (Q5_K_XL, 256k ctx)	~$4,000+ GPU alone
Single RTX 3090 (24GB VRAM)	24GB	936 GB/s	41 (Q4, llama.cpp)	~$1,000–1,200 used
Dual RTX 3090 (48GB total)	48GB	936 GB/s each	variable (11–41 t/s reported)	~$2,000–2,400

Several things stand out here. First, the RTX 3090 numbers are confusing because they depend heavily on quantization and whether the model fits entirely in VRAM. At Q4_K_XL quantization, a single 3090 can fit the 27B model (roughly 17GB) with room for KV cache — and one commenter running llama-bench documented 41 tokens/second with MTP pushing that to 65–70. Another user with dual 3090s was seeing only 11 tokens/second, suggesting configuration issues rather than hardware limits. The 3090 route is genuinely viable at Q4 but requires more setup expertise.

Second, the DGX Spark’s numbers look worse than the 5090 despite similar pricing. The Spark has 128GB of memory but runs at around 273 GB/s bandwidth — roughly equivalent to an M4 Mac Mini. For the 35B A3B MoE model, one commenter achieved 110 tokens/second on a Spark using MTP speculative decoding, but the more common reports clustered around 50 tokens/second. For Qwen 3.6 27B dense specifically, a commenter directly tested it and found the performance “dog shit” relative to expectation.

Third, and most importantly: the Mac Mini M4 Pro with 64GB, the machine the article recommends over the expensive MacBook, was discontinued four days before the article was published. Apple removed the 64GB option on June 25th. The only current Mac Mini options top out at 48GB, which technically fits the 27B model at Q4 but with minimal room for KV cache at long contexts.

The Heat Problem Is Real and the Solutions Are Funny

The top comment in the 660-comment thread, from a user with an M5 Max 128GB MacBook, opened with: “DO NOT buy this MacBook if you plan on doing serious coding using local LLMs with it. The reason is simple: your fingers will burn and your head will explode from the noise.”

The article includes a thermal camera photo of the laptop mid-inference, described as making the author’s “knees start to melt.” This is meant to be charming. It is also a product description of a $6,700 laptop that becomes physically uncomfortable to use during its primary advertised workload.

The recommended solution, from multiple HN commenters converging independently: “Just buy a Mac Mini and put it in the basement.” A headless server that you SSH or Tailscale into, running the model there, using your actual working machine as a thin client. One commenter described running models on a Mac Mini via Tailscale over hotel Wi-Fi and finding it perfectly usable since LLM token output is low bandwidth.

Another commenter discovered a quieter workaround: macOS low-power mode. With low-power mode enabled, the fan never spins up. Token generation speed on the 35B A3B dropped from 80 tokens/second to 38 tokens/second — but with MTP speculative decoding re-enabled, that gap narrows substantially, and you get silence. A third commenter described their M5 Max running Qwen with Unreal Engine in the background with no thermal issues, suggesting workload matters significantly.

A user with a Linux box running two RTX 3090s described a more elaborate solution: lowering each card’s power limit to 250W, building a ducting and fan system to vent heat outside, and achieving 110 tokens/second with a silent machine. This is not “put it in the basement” — this is a custom HVAC project for a GPU cooling rig.

The thermal problem is not unique to Qwen 3.6. Any dense 27B model saturating a high-end GPU or Apple Silicon chip will generate sustained heat. What is notable is how normalized the heat problem has become in this conversation. Comments about burning fingers and melting knees are treated as table stakes, not disqualifying flaws.

The Benchmark That Grounds the “Sweet Spot” Claim

The article includes a benchmark from Artificial Analysis that places Qwen 3.6 27B at score 37 on their composite intelligence index. For reference, the article claims this is comparable to “mid-2025” frontier models — GPT-5 and Claude Sonnet 4.5 territory. The 35B A3B scores 32, comparable to “early-2025” models like o3 and Claude Sonnet 4.

This framing is powerful because it converts “local model” into a temporal claim: you can run frontier-quality AI from six months ago, today, on your own hardware, with complete privacy.

How accurate is that framing? It depends on what you mean by “frontier-quality.”

Aggregate benchmark scores compress a lot of variation. The HN comments reveal several specific domains where the consensus on Qwen 3.6 27B breaks down:

Coding: Strong, with Caveats

Qwen 3.6 27B is broadly agreed to be excellent at coding tasks — one-shot code generation, refactoring, multi-file edits. The article demonstrates a hexagonal minesweeper built in a single prompt. HN commenters working with OpenCode (an open-source coding agent) report consistent success for typical coding tasks.

But two specific failure modes surfaced repeatedly:

Looping. Multiple commenters reported the model getting stuck in repetitive loops, especially on longer tasks. The mitigation is llama.cpp’s --repeat-penalty flag set above 1.0, plus explicitly disabling thinking/reasoning mode in the system prompt. Without these settings, agents using Qwen 3.6 27B as their backbone are prone to spiraling in place rather than making progress.

Context fragility. One commenter using LM Studio discovered that the default 32K input window was truncating OpenCode’s system prompt, silently cutting instructions from the middle of the context window. The model was receiving half its instructions. This is a configuration issue, not a model flaw, but it illustrates how local inference stacks require significantly more tuning than API calls where the context window is managed for you.

For agentic workflows specifically, one commenter raised a point that deserves attention: using 64K context with sub-agents (as Migdał’s benchmark does) is not how most serious agent workflows operate. Real multi-agent architectures with long-running tasks regularly fill 128K+ context windows, spawn sub-agents with their own contexts, and hit KV cache collisions when running multiple concurrent slots. The 27B dense model uses 41GB of RAM for the model weights at 8-bit, leaving only 7GB free on a 48GB machine for KV cache — which means roughly 16K tokens of context headroom per session before you start paging. At 256K context with Q8_0 weights, you need 53GB of VRAM, which requires a 64GB+ unified memory machine or a 5090.

Tool Calling: Better Than Alternatives at This Size

The HN thread surfaced an extended comparison with Gemma 4 31B, which is the other commonly discussed model in this parameter range. Multiple commenters noted that Gemma 4 31B is actually stronger than Qwen 3.6 27B on tasks requiring nuanced reasoning and general intelligence — but substantially weaker on tool calling and agentic coding tasks.

The cause appears to be Gemma’s non-standard tool calling format. Unsloth’s quantized versions attempt to fix this, but one commenter described persistent issues with edit tool calls specifically — the model would fall back to rewriting entire files from context rather than making targeted edits. For anyone building an agent that uses the filesystem, this is a significant practical problem.

Qwen 3.6 27B’s tool calling is more reliable out of the box, which is why it dominates in agentic coding benchmarks even when Gemma 4 31B scores higher on aggregate intelligence benchmarks. This is the correct framing for a “sweet spot” claim: not “best at everything” but “best combination of intelligence and reliable tool use for coding agents at this hardware tier.”

Security Research: Surprisingly Capable

One of the more interesting data points came from a commenter who works in security research. Running Qwen 3.6 35B A3B on a DGX Spark, they had it perform reverse engineering and disassembly of MS-DOS assembly language games from the 1980s — and found it handled the task well. The same session revealed a significant failure mode: the model entered an argument loop insisting the user could not be the author of the articles it was analyzing, because “the user is the user, the article author wrote the article.” Looping again, but in a different direction.

A separate commenter specifically running Gemma 4 31B for security bug hunting described it outperforming Qwen 3.6 27B by a significant margin on their specific benchmark of hard security bugs (drawn from Mythos-reported vulnerabilities). Qwen did better on general coding; Gemma did better on reasoning about subtle security flaws. The choice depends on your use case.

The Quantization Trade-Off Nobody Explains Clearly

The article recommends the Unsloth Q8_0 quantization of Qwen 3.6 27B with MTP support. This is a reasonable default for users with sufficient RAM. But the quantization landscape for this model is complex and meaningfully affects both quality and hardware requirements:

Quantization	Model Size	VRAM/RAM Required	Quality Impact	Notes
BF16 (full precision)	~54GB	64GB+ unified or dedicated	Reference quality	Rarely practical for inference
Q8_0 (8-bit)	~29GB	41GB total with KV cache	Minimal (~0.1% degradation)	Recommended default; needs 48GB+ machine
Q5_K_XL (Unsloth dynamic)	~22GB	~32GB with KV cache	Small degradation	Good balance on 32GB cards
Q4_K_XL (Unsloth dynamic)	~17GB	~24GB with KV cache	Moderate; noticeable on long tasks	Fits on single 3090 or 24GB GPU
Q4_K_M (standard)	~16GB	~22GB with KV cache	More degradation than dynamic quants	Less preferred than Unsloth dynamic
Q2–Q3 (aggressive)	~8–12GB	~16–18GB	Significant quality loss	Only for very constrained hardware

The key insight from the HN comments: Unsloth’s “dynamic” quantization formats (UD-Q4_K_XL, UD-Q5_K_XL) outperform standard quantizations at the same bit depth. The reason is that dynamic quantization preserves higher precision for the model layers most sensitive to quantization noise, while aggressively quantizing layers where the degradation is imperceptible. A Unsloth Q4_K_XL behaves meaningfully better than a standard Q4_K_M at approximately the same model size.

This also explains why the NVidia DGX Spark’s NVFP4 quantization — a 4-bit format — performs poorly on dense models like Qwen 3.6 27B despite the Spark’s 128GB of memory. NVFP4 is not a dynamic quantization; it applies uniform 4-bit compression and sacrifices quality significantly on this model. The Spark’s strength is running the 35B A3B MoE model efficiently, where the active parameter count during inference is low enough that 4-bit precision degrades less visibly.

For users on a 24GB GPU: the Unsloth UD-Q4_K_XL fits in roughly 17GB of VRAM, leaving 7GB for KV cache. At typical prompt/output sizes for coding tasks, this is tight but workable. One commenter on a single RTX 3090 documented 41 tokens/second at this quantization — genuinely fast, though your context window is limited. For agentic workflows that need 64K+ context, you need either a larger GPU or unified memory architecture.

The Context Window Problem Nobody Wants to Talk About

Qwen 3.6 27B supports up to 256K tokens of native context. This is a remarkable number. In practice, running at 256K context on most hardware is either impossible or extremely slow.

KV cache memory usage scales with context length. At 256K context with Q8_0 weights, you need approximately 53GB of VRAM total — which requires a 64GB unified memory machine or two high-end GPUs. The Artificial Analysis benchmark and the article’s performance claims are measured at 64K context, where the model uses approximately 41–42GB of RAM.

The practical ceiling for most hardware configurations:

• 48GB machine (M5 Pro MBP, M4 Mini 48GB): Q4 quantization only for comfortable 64K context. Q8 is possible but leaves very little KV cache headroom.
• 64GB machine (M4 Mini 64GB, Mac Studio): Q8 at 64K context is comfortable; 128K context is feasible; 256K is tight.
• 128GB machine (M5 Max 128GB): Q8 at 256K context works. This is the configuration the article is benchmarking.
• Single RTX 3090 (24GB): Q4 only, ~32K–64K context maximum before significant slowdown from KV cache overflows.
• Single RTX 5090 (32GB): Q5_K_XL at 256K context, ~67 tokens/second. Best performance-per-dollar for context-heavy workloads.

A commenter who works with sub-agents made the most pointed observation in the thread: “What tasks you would really be able to do with 64k context, 1 agent? For sure some quick edits but not complex planning where you need to ingest a lot of files and end up losing 80% of the ingested files to compaction.”

This is accurate. Claude Code, for example, can fill 100K+ tokens in a single session of moderate complexity. Agentic workflows that spawn sub-agents accumulate context rapidly. If your “local LLM for coding” use case involves complex multi-file projects with planning phases, 64K context on Qwen 3.6 27B will hit limits within minutes on most hardware configurations.

The counterargument, also present in the thread: for many common tasks — targeted code edits, quick refactors, single-function implementations, document summarization — 32K–64K context is more than sufficient. The KV cache constraint is a limitation for power users doing sustained agentic work, not for developers doing typical task-by-task coding assistance.

The Cloud vs. Local Cost Arithmetic

One of the most substantive sub-threads in the 660 comments was a debate about whether local inference makes economic sense at all. The argument against local inference, stated bluntly by one commenter: “Cloud models are faster, don’t consume power, have much bigger context, are more precise, and have wider engineering context of the problem. Except for privacy and use cases blocked by cloud models, local LLMs are currently an expensive toy.”

The arithmetic is not entirely wrong. A $6,699 MacBook Pro with 128GB represents roughly 13 years of a $40/month frontier model subscription, or approximately $200,000 in Claude Pro / OpenRouter credits at current pricing. The per-token cost of running a local model approaches zero after hardware amortization, but the breakeven period is long for anyone who uses models intermittently.

But this arithmetic misses several things that the local inference advocates correctly identified:

The subsidy question. Multiple commenters noted that current cloud model pricing reflects massive venture capital subsidies, not sustainable economics. “Other frontier models run at a massive subsidy, where paying $100 a month gives us thousands worth in tokens. Let’s use the discount while it lasts!” The original article makes this point explicitly. If frontier model pricing normalizes upward over the next five years, the local inference breakeven point shortens substantially.

The privacy ceiling. For businesses with proprietary code or sensitive data, cloud APIs require sending that data to external servers. GDPR constraints, competitive concerns, and contractual data handling requirements create a non-negotiable use case for local inference regardless of cost. One commenter specifically mentioned medical data and personal secrets as domains where “I don’t feel comfortable sharing with the US or China” — and this is a real and growing category of inference demand.

The availability ceiling. Cloud APIs have rate limits, outages, and pricing changes. Local inference has none of these. For automated workflows running overnight or batch processing tasks, the predictability of local inference has real operational value.

The opportunity cost of learning. A substantial subset of the thread was people learning local inference as a skill investment. One commenter described running models locally on existing hardware as the entry point that made the entire local LLM stack comprehensible. “I no longer feel like I am running away defeated” is a concrete outcome for someone who needs to understand how these systems work to stay current in their field.

The economic case for local inference is strongest for: (a) high-volume users who saturate API rate limits, (b) privacy-constrained deployments, (c) users in regions with poor API latency, and (d) people who need to run models continuously for agentic tasks and want predictable costs. It is weakest for: (a) intermittent users, (b) developers who only need code completion, and (c) anyone who needs frontier-tier intelligence on complex reasoning tasks rather than strong-but-not-frontier intelligence on coding tasks.

The Inference Stack Matters More Than Most Reviews Acknowledge

The article recommends llama.cpp over Ollama, with a parenthetical: “I would recommend against using that on ethical grounds.” This is a reference to ongoing concerns in the local LLM community about Ollama’s licensing, update practices, and lack of transparency compared to the fully open-source llama.cpp.

But the more practically significant point is that different inference stacks produce meaningfully different performance on the same hardware. The HN thread documented this repeatedly:

On Apple Silicon, llama.cpp consistently outperformed MLX for the 27B dense model in Migdał’s own benchmarks — 18 tokens/second vs. 17 tokens/second at the model level, and 32 vs. 17 with MTP (MLX does not support MTP for this model). One commenter noted that llama.cpp was using 95% of GPU resources on the M5, indicating efficient utilization. MLX is Apple’s own inference framework and was expected to have home-field advantage; the fact that llama.cpp matches or beats it is notable.

For NVIDIA hardware, vLLM vs. llama.cpp is the relevant comparison. Multiple commenters noted that llama.cpp’s performance has converged with vLLM for single-GPU inference on most workloads, while llama.cpp offers much easier model switching, better compatibility with Unsloth’s dynamic quantization formats, and lower setup complexity. The DGX Spark user who achieved 70–90 tokens/second on the 35B A3B model did so by switching from vLLM with EAGLE3 speculative decoding to MTP — a change that required moving to llama.cpp.

The inference stack decision also affects feature availability. MTP (multi-token prediction) is only available in llama.cpp. EAGLE speculative decoding is the vLLM equivalent but requires a separately hosted draft model. Flash attention (-fa on in llama.cpp) is essential for performance at longer context windows. KV cache quantization (allowing higher precision for the main model while compressing the cache) is available in llama.cpp as --cache-type-k q8_0 and similar flags.

The full recommended llama.cpp invocation for Qwen 3.6 27B with MTP is more complex than the article implies:

llama-server -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 \
    --spec-type draft-mtp \
    -ngl 999 \
    -fa on \
    -c 65536 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --repeat-penalty 1.1 \
    --port 8080

The --repeat-penalty 1.1 flag matters. Without it, extended coding sessions are prone to the looping behavior multiple commenters flagged. The KV cache quantization flags are optional but save significant RAM at longer contexts. The article does not mention either of these.

What “Sweet Spot” Actually Means Here

After reading the article and its 660-comment thread, a clearer picture of the “sweet spot” emerges — but it is narrower than the headline implies.

Qwen 3.6 27B is the best locally-runnable model for agentic coding tasks on hardware between approximately $2,500 and $4,000, under the following conditions:

• You have a machine with at least 48GB of unified memory (Apple Silicon) or 32GB of VRAM (discrete GPU)
• Your primary use case is coding assistance, code generation, and tool-using agents
• Your context requirements fit within 32K–64K tokens per session
• You are willing to run llama.cpp with appropriate flags rather than Ollama or LM Studio defaults
• Privacy, cost predictability, or offline availability matter to you

It is not a sweet spot for:

• Users with 32GB of unified memory or less (the model runs, but slowly and with severe context limits)
• Users who need 128K+ context for complex multi-file agentic work
• Users primarily doing nuanced reasoning, research, or general intelligence tasks where Gemma 4 31B or frontier APIs are stronger
• Users who want frontier model intelligence — Qwen 3.6 27B is mid-2025 frontier-equivalent on benchmarks, which means it trails current top models on hard reasoning tasks
• Users who can access current frontier APIs at subsidized prices and have no privacy or offline requirements

The comparison to the MoE sibling deserves a direct statement: for most users, the 35B A3B MoE is the better practical choice. It runs at 3–5x the token speed, fits in similar memory at 8-bit (37GB for MoE vs. 41GB for dense, though MoE requires less active compute per token), and scores within 10–15% of the dense model on coding benchmarks. The author’s preference for the dense model is a quality-over-speed trade-off that makes sense for someone who values correctness over throughput — but for an agent running unattended, generating five times as many tokens per second is a significant advantage.

The Ornith-1.0 project, which appeared on HN the same week with 229 points, is relevant here. Ornith is a post-trained version of Qwen 3.6 (and Gemma 4) specifically optimized for agentic coding and self-improvement. Early benchmarks suggest it underperforms the base Qwen 3.6 27B on most tasks — an interesting result for a model whose entire premise is improving on the base. But it demonstrates the direction the open-weights community is moving: rather than waiting for new base models, fine-tuning existing ones for specific agentic workflows.

The Hardware Treadmill and the Coming M6

One commenter offered the most structurally interesting observation in the thread: “Running potentially SOTA open-weight models locally only became a thing in fall 2023. If a hardware cycle takes ~3 years, fall 2026 would be the first possible device generation where Apple exploits its advantage with the unified RAM architecture. More realistically, spring 2027.”

This is worth sitting with. Apple Silicon’s unified memory advantage for LLM inference was not designed for LLM inference. It was designed for video editing and creative workloads that require large textures and buffers. The fact that it incidentally produces the highest memory bandwidth per dollar for models in the 27–70B parameter range is an architectural accident that Apple is only now beginning to acknowledge and design around.

The M5’s “neural accelerator” additions — which improve prefill speed significantly (one commenter cited 3–4x faster prompt processing vs. M4) but don’t dramatically change token generation bandwidth — are the first evidence of Apple deliberately targeting LLM inference. The M6, expected in late 2026 or 2027, will likely see further optimization of this path.

For Qwen 3.6 27B specifically: if you buy a machine today capable of running it well, the model will look significantly less impressive in 12–18 months when the next generation of open-weights models arrives. The Qwen 3.6 “beats mid-2025 frontier models” framing is accurate today. In mid-2027, mid-2025 frontier quality will likely be achievable on a $1,200 laptop. The hardware investment today is betting that local inference at this tier of quality creates enough value to justify the purchase before it is commoditized.

For some users, that bet is correct. For others, waiting makes more sense.

The Honest Verdict

Qwen 3.6 27B is genuinely excellent. The “sweet spot” claim is defensible — with caveats about what hardware you have, what you are using it for, and what your alternatives are. The Quesma article is honest about the benchmark source and the hardware used, even if those details are buried in ways that make the headline claim seem more universal than it is.

The 660-comment thread reveals the actual distribution of experience: enthusiasts with high-end hardware who are delighted, developers with mid-range hardware who find it adequate for targeted tasks but limiting for agentic workflows, and skeptics who calculate that API costs remain cheaper than hardware for their usage patterns.

All three groups are right for their own situations.

If you have a machine with 64GB+ of unified memory or a 32GB+ discrete GPU, Qwen 3.6 27B is worth running. Configure llama.cpp properly, use Unsloth’s dynamic quantizations, add the repeat penalty, enable flash attention, and you will get a coding assistant that is meaningfully better than GPT-4-class models from 12 months ago — running entirely on your own hardware.

If you have a 48GB machine and primarily do targeted, context-bounded coding tasks, the 35B A3B MoE variant is probably a better match than the 27B dense. Faster generation, similar quality for most tasks, lower memory pressure at moderate context lengths.

If you have a 32GB machine or less, or if you primarily need intelligence on complex reasoning tasks rather than coding specifically, the case weakens substantially. You are either running an aggressively quantized model that degrades noticeably or spending money on hardware that delivers a worse experience than the subsidized API you already have access to.

The “sweet spot” is real. It is just sitting on top of hardware that costs $2,800 to $6,700, requires a dedicated server setup to avoid burning your lap, and demands significantly more configuration than any API call you have ever made.

That is the benchmark. The claim holds up — at the asterisk-laden resolution where most claims hold up when you actually measure them.