Tailscale Spent Six Months Chasing Database Corruption. The Bug Was in SQLite and Had Been There for Sixteen Years.

Tailscale Spent Six Months Chasing Database Corruption. The Bug Was in SQLite and Had Been There for Sixteen Years.

Yesterday Tailscale published a postmortem that deserves more attention than most incident reports get. Between August 2025 and February 2026, they experienced 19 separate database corruption incidents. Each one caused real downtime — their control plane shards going dark for up to an hour, customers unable to onboard new devices, trust eroding even among users on unaffected shards. The root cause was a data race in SQLite that had existed, undetected, since approximately 2009.

I have spent the last decade working with systems that depend on SQLite at various scales, and I find this incident instructive in ways that go well beyond “SQLite has a bug.” The more important story is about what happens when you take a correct technology and operate it in a way that slightly deviates from the standard path — and why those deviations almost always find the edge cases that years of production testing missed.

The Architecture Tailscale Was Running

Tailscale’s control plane is sharded: each shard is a Go process that exclusively owns an SQLite database holding configuration data for a set of tailnets. A single writer. No multi-process contention. This is textbook SQLite deployment — the “single-writer, single-process” pattern that SQLite is explicitly designed for and that countless deployments run without incident.

The non-standard part was their checkpoint configuration. SQLite in WAL mode works as follows: writes go to the WAL file first, not directly to the main database file. At some point, those WAL pages get copied back (“checkpointed”) to the main database file. By default, SQLite manages this automatically. Tailscale took manual control of the checkpointing process to support their backup pipeline — they take full database snapshots every few minutes and upload the SQLite file to S3. To make those backups consistent, they needed to control exactly when checkpoints ran and run them aggressively.

Everything about this was documented, supported, and reasonable. It is not an obscure or unsupported feature. The SQLite documentation describes the manual checkpoint API, the WAL mode, and the tradeoffs involved. Tailscale’s engineering team was not doing anything naive.

But by deviating from the default checkpoint behavior, they stepped into a code path that the standard operational envelope almost never exercises. And in that code path, a sixteen-year-old data race was waiting.

Why This Bug Survived for Sixteen Years

The WAL-reset bug is a race condition between a checkpoint operation and a concurrent write transaction. If a write happens at a specific moment during the checkpoint — specifically when the checkpoint process is resetting the WAL — the checkpoint can become confused about which pages it has already copied. It believes pages have been moved to the main database file when they have not. The result: those pages are never written to the main database file, the data vanishes without error, and when other pages that reference the missing pages do get written, the database becomes internally inconsistent.

The reason this survived sixteen years is that triggering it requires an uncomfortably specific timing condition. In most deployments, checkpoints are triggered infrequently — when the WAL grows beyond a default threshold, which is 1,000 pages. In most deployments, the checkpoint is not running aggressively or continuously. And in most deployments, the probability that a write transaction and a WAL-reset coincide in the narrow window where the race is possible is vanishingly small.

Tailscale’s aggressive, manual checkpointing raised the frequency of that window dramatically. They were running checkpoints far more often than typical deployments, which meant that the narrow race window appeared far more often. Even with that increased frequency, it still took years of operation to accumulate enough exposures for the bug to surface — and then 19 times in six months once conditions aligned.

The SQLite developers themselves had to add code specifically designed to trigger the race condition artificially in their testing environment, because it could not be reproduced under normal synthetic load. That is how precise the timing requirements are.

What Their Forensic Investigation Actually Looked Like

I want to be specific about the investigative methodology here because it illustrates a class of debugging problems that most engineers never encounter in their careers: bugs where the symptoms are clear, the severity is obvious, the trigger is unknown, and the reproduction is impossible.

Tailscale’s initial approach was standard: review recent code changes, look for common factors across incidents, try to reproduce synthetically. None of it worked. The incidents were not correlated with specific shards, customers, tailnet features, time of day, or load level. There were no relevant recent code changes. The low-level SQLite interaction code had been written years earlier and had been running without incident. The corruption could not be reproduced in a test environment.

This left them with one investigative option: deploy passive forensic telemetry into their live production environment and wait for the next incident. That is an uncomfortable place to be. You cannot speed up the investigation. You cannot run controlled experiments. You are at the mercy of a stochastic process that might trigger tomorrow or six weeks from now — and indeed they had a six-week gap between October and December 2025 with no incidents, before they returned.

The transaction logging pipeline was the key breakthrough on the diagnostic side. They streamed every SQL write to a separate log file. Because SQLite is a single-writer system with fully serialized transactions, replaying those logs against a known-good backup should produce a deterministic result. In two incidents, it did not — data written and committed by a transaction was invisible to later transactions when replayed. This was the tell. A committed write that disappears is a data race fingerprint. It cannot happen through any mechanism other than concurrent state mutation at a level below the transaction abstraction.

The SQLite developers’ contribution — the tmstmpvfs shim — was the diagnostic tool that closed the loop. This is a wrapper around SQLite’s virtual filesystem layer that adds detailed tracing of checkpoint behavior. Deployed into Tailscale’s production environment, it captured exactly what was happening during the faulty checkpoint: the WAL-reset occurring while a write was in flight, the checkpoint miscounting the pages it had moved.

The SQLite 3.52.0 False Alarm Is Its Own Lesson

The fix for the WAL-reset bug was released in SQLite 3.52.0. Tailscale deployed it carefully — canary shards first, then fleet-wide rollout. Their backup integrity monitor immediately flagged corruption in 13 databases. This looked catastrophic.

It was not real corruption. It was a second unrelated bug: SQLite 3.52.0 also included an optimization that subtly changed how text-to-floating-point conversions were rounded. Tailscale was storing high-precision timestamps as text and using a virtual generated column with a computed floating-point value. The optimization changed the rounding, which meant the virtual column values no longer matched what the index had been built on. SQLite’s PRAGMA integrity_check correctly flagged this as index corruption — but it was false corruption in the sense that no data was actually lost. The index was stale, not the data.

The SQLite developers withdrew 3.52.0 and published 3.51.3, which contained only the WAL-reset fix. Tailscale fixed their side by reducing timestamp precision to integer seconds, where text-to-integer conversion has no ambiguous rounding behavior. SQLite 3.53.0 later added a self-healing index feature that prevents this class of problem in the future.

From where I sit, the 3.52.0 incident illustrates a principle I have seen violated repeatedly in production systems: the migration from a broken state to a fixed state can produce its own class of failures that look like the original problem. Tailscale’s canary deployment caught some of this, but the canary shards happened not to have timestamps that triggered the changed rounding behavior, so the canary gave a false all-clear. This is a hard problem. Your canary data distribution may not include the edge cases that expose the bug you are trying to fix — especially when the edge cases involve floating-point rounding.

What “Boring Technology in a Non-Standard Way” Actually Means

Tailscale’s postmortem includes this observation: “running boring technology in a non-standard way is a risk.” That sentence is correct but undersells the specific mechanism. Let me be more precise about what actually happened.

SQLite’s test suite is enormous. The SQLite developers run billions of test cases. The default checkpoint behavior — auto-checkpoint at 1,000 WAL pages — has been exercised under that test load for sixteen-plus years. The race condition in the WAL-reset path was not exercised by that testing because the timing required aggressive, manual checkpointing at a frequency far higher than the default. The test suite did not cover that pattern with sufficient intensity to surface the race.

This is not a criticism of SQLite’s testing. It is a description of a fundamental limit: the space of possible operational configurations is too large to exhaustively cover. You can be comprehensive over the standard envelope. You cannot be comprehensive over all deviations from it. The more a production system deviates from the standard operational path, the higher the probability it will encounter edge cases that standard testing does not cover — not because the code is low quality, but because the test coverage implicitly follows the operational assumptions of the standard path.

I have seen this pattern in systems built on PostgreSQL, Redis, Kafka, and custom storage engines. The standard configuration is deeply battle-tested. The moment you start tuning aggressively — adjusting fsync behavior, tweaking WAL settings, taking manual control of compaction, configuring non-default replication factors — you are entering territory where the combination of your specific configuration and the underlying implementation has seen far less cumulative real-world load. The bugs that exist there are often older and more subtle than the bugs in the standard path, because the standard path bugs got found and fixed years ago by the millions of systems running that path.

The Case for Professional Open Source Support

One detail in Tailscale’s postmortem that I think deserves emphasis: they engaged a professional SQLite support contract early in the investigation. This was a decision that mattered. The tmstmpvfs shim — the virtual filesystem wrapper that ultimately caught the race condition — was developed by the SQLite core team specifically for this investigation. Without the support relationship, Tailscale would not have had access to a SQLite developer creating a custom diagnostic tool for their specific problem.

The open source ecosystem has an uncomfortable tension around this. Most teams consuming open source infrastructure behave as if professional support is optional — something you pay for after things break, or not at all. But for foundational infrastructure like a database engine, the people who wrote the code and know its internals at depth are irreplaceable when you hit the kind of bug that cannot be reproduced synthetically and cannot be debugged with standard tooling. Tailscale’s decision to pay for that access saved months of investigation time, and possibly saved the business-critical relationship between their stability problems and customer trust.

I have worked on teams that tried to debug deep storage engine bugs through public GitHub issues and documentation alone. The experience is qualitatively different from having direct access to the engineers who wrote the code. The knowledge of what should be impossible — “a committed write disappearing cannot happen unless…” — compresses the hypothesis space dramatically. That knowledge is not in the documentation.

There is also a broader point about open source sustainability here. SQLite is public domain software. There is no commercial entity with a revenue stream that funds SQLite development — the developers are funded primarily through consulting and support contracts. When Tailscale paid for professional support, they were not just buying access to expertise; they were funding the development of the tmstmpvfs shim that is now available to every SQLite user. The fix for the WAL-reset bug is available to every SQLite user. The broader ecosystem benefits from companies that are willing to pay for support rather than treating open source infrastructure as a free good with no cost of ownership.

The Verification Strategy After the Fix

The most technically interesting part of Tailscale’s postmortem to me is how they verified the fix. They did not simply declare victory after rolling out 3.51.3 and observing no corruption for a few weeks. The absence of corruption incidents during a six-week period was already in their record — it had happened before the fix, when the bug was still present. Absence of symptoms is not proof of absence of cause.

Instead, they patched their SQLite driver to log a warning whenever the specific conditions for the WAL-reset race were met: a write transaction overlapping with a WAL-reset during a checkpoint. This warning would fire when the race conditions occurred in production, but — if the fix was correct — without resulting in corruption. They deployed the warning and waited.

Two months later, the warning fired. This was the proof they needed: the race conditions are still occurring in production. The fix is preventing them from causing corruption. If the warning had never fired, they would have had no way to distinguish “the fix works” from “the race condition is not occurring.” The warning firing proved the former.

This is a principle I try to apply in post-fix verification: wherever possible, instrument not just the symptom but the causal condition. If you can detect when the conditions that caused the bug are occurring without the bug’s consequences, you have positive proof of the fix rather than absence of evidence. For timing-sensitive concurrency bugs, this is especially important because the conditions may not occur frequently enough to validate the fix by simply watching for clean operation.

Most engineering teams declare victory after a period of clean operation and move on. Tailscale’s approach — waiting two months for positive confirmation rather than relying on absence of negative evidence — is more rigorous and more honest. The engineering culture that produces that kind of patience is also the engineering culture that catches the bug in the first place.

What This Means for Teams Running SQLite at Scale

The WAL-reset bug was fixed in SQLite 3.51.3, released in 2026. If your system is running an older version of SQLite with manual checkpoint control and aggressive checkpoint frequency, you are potentially exposed. The probability of hitting it depends on how aggressively you checkpoint and how much concurrent write activity you have.

More broadly, if you are running SQLite in WAL mode with manual checkpointing for backup consistency — a common pattern in production systems that need point-in-time consistent snapshots — here is what I would evaluate:

First, upgrade to at least SQLite 3.51.3. The 3.52.0 release introduced the stale expression index issue; 3.51.3 is the safe fix-only version. 3.53.0 includes the self-healing index feature if you want that as well. Second, audit any virtual generated columns that involve text-to-numeric conversions. If you have computed columns based on text timestamps or string-to-number coercions, the precision of those conversions can produce stale indexes after a SQLite version upgrade. Third, consider whether aggressive manual checkpointing is necessary for your specific backup strategy. Tailscale’s use case — uploading a complete SQLite file to S3 every few minutes — required consistent snapshots, which drove the aggressive checkpointing. If your backup strategy has more flexibility, you may be able to reduce checkpoint frequency and reduce your exposure to this class of race condition in future bugs.

None of these are criticisms of SQLite as a database. SQLite remains the correct choice for the single-writer, embedded database use case. The 16-year age of this bug is evidence not of poor code quality but of how comprehensively the standard use cases have been exercised — this bug survived because it was essentially invisible to standard operational patterns.

The Trust Damage Calculation

There is a business dimension to this incident that Tailscale is candid about. Their status page shows a global incident even when only a subset of shards is affected. Most customers who saw those status page entries were on unaffected shards. The actual impact radius of each individual incident was narrow — but the repeated pattern of incidents visible to all customers erodes trust beyond the literal set of customers experiencing downtime.

This is a specific failure mode for sharded, distributed infrastructure: the global status signal amplifies per-shard incidents to the entire customer base. There is no easy fix for this. You can make the status page more granular — “shard 7 is experiencing an issue” — but most customers do not know what shard they are on, and “shard 7” does not communicate anything meaningful to them. A more granular signal might reduce the trust impact for customers who are not affected, but it also makes the status page harder to parse for customers who want to know “is my Tailscale working right now?”

Tailscale’s approach — being transparent about the issue on their public blog, including the admission that they “didn’t live up to their promise” of reliability for several months — is the right call. Customers can tolerate bugs in complex distributed systems. What they cannot tolerate is a company that obscures those bugs behind vague status page language or post-hoc framing that minimizes the impact.

I have watched companies lose significant customer trust not because they had incidents, but because their incident communication was defensive. Tailscale’s postmortem does the opposite: it gives you the timeline, the technical details, the false alarm during the fix rollout, and an honest accounting of what went wrong. That is the kind of transparency that actually rebuilds trust rather than just performing it.

Predictions

Within the next eighteen months, at least two other companies operating SQLite at production scale with non-default checkpoint configurations will publicly disclose corruption incidents that they have now traced to the WAL-reset bug in pre-3.51.3 versions. The disclosure will happen because Tailscale’s postmortem gives them the vocabulary and the template. Most companies experiencing this bug have no idea it is a known SQLite issue; they are blaming their own code or their storage layer.

The tmstmpvfs shim that the SQLite team built to diagnose this issue will be adopted by at least one major SQLite-based commercial database product as a standard debugging tool within 12 months. The tool solves a real diagnostic gap that affects everyone running SQLite under production load with non-default checkpoint behavior.

SQLite’s dominance in the embedded single-writer database category will not be threatened by this incident. The combination of reliability, performance, and operational simplicity that makes SQLite the right choice for embedded databases is not changed by the existence of a rare race condition in a non-default configuration. The developers found the bug, fixed it, and published both the fix and the diagnostic tooling. That is the correct outcome. What this incident will do is accelerate the adoption of professional SQLite support contracts among companies running SQLite at production scale — which is a healthy development for the open source ecosystem.

The more uncomfortable prediction: this is not the last 16-year-old bug in a widely-used open source infrastructure component. The SQLite WAL-reset race survived for sixteen years because the standard operational envelope did not expose it. Every foundational open source component — PostgreSQL, Redis, RocksDB, LevelDB — has code paths that are similarly under-exercised by the standard configuration and that have not accumulated enough real-world exposure to surface their own latent race conditions. We will find them the same way we found this one: a production system operating in a non-standard configuration will hit it, spend months diagnosing it, and eventually give us both a fix and a postmortem worth reading.

Tailscale’s transparency here is genuinely useful. The postmortem tells you what the bug is, which SQLite versions are affected, how to verify the fix, and what operational patterns make you more likely to encounter it. That is the correct use of an incident report: not to demonstrate that everything is now fine, but to give the broader engineering community the information they need to assess their own exposure.

The Concurrency Model That Made This Hard to See

SQLite’s concurrency model is often described as “single writer, multiple readers.” That is accurate but incomplete in a way that matters for understanding this bug. The single-writer constraint means that at any given moment, only one thread or process can be writing to the database. But within that constraint, there is still concurrency between the writing thread and SQLite’s internal housekeeping operations — specifically the checkpoint thread.

When you run SQLite in WAL mode with PRAGMA wal_autocheckpoint enabled (the default), SQLite will automatically trigger a checkpoint when the WAL grows beyond the threshold. This checkpoint can run concurrently with ongoing read operations. More importantly, if you take manual control of checkpointing using sqlite3_wal_checkpoint_v2(), you introduce a specific interaction pattern where your application code is explicitly triggering checkpoint operations while the database is also potentially receiving write transactions.

The WAL-reset race occurs in the following sequence: a checkpoint operation completes the process of copying pages from the WAL to the main database file, and then resets the WAL — marking it as empty so it can be reused. During this reset, if a write transaction begins, the write will see the WAL as available and start writing new pages. The checkpoint code, having initiated the reset but not fully completed it, can become confused about which pages belong to the pre-reset WAL and which belong to the new write. In the specific failure mode Tailscale encountered, the checkpoint would report pages as having been moved to the main database file when they had not actually been written there yet.

The fix in SQLite 3.51.3 adds an additional atomic check during the WAL-reset process that detects when a new write has started. When this condition is detected, the checkpoint backs off rather than proceeding with a potentially inconsistent reset. It is a small code change — a handful of lines — applied to a code path that runs correctly in the vast majority of conditions. The difficulty was not in writing the fix but in identifying where the fix needed to go.

This is the nature of concurrency bugs in storage systems. The code that needs to change is often small. The challenge is understanding the precise execution interleaving that produces the wrong outcome, and then designing a synchronization point that prevents that interleaving without introducing excessive contention or new performance problems. The SQLite developers’ solution — an atomic check that allows the checkpoint to detect and abort a race rather than trying to prevent the race from occurring — is the right approach for a code path where the race is genuinely rare and the cost of false positives (unnecessary checkpoint aborts) is low.

Why S3-Backed SQLite Backup Architectures Will Keep Hitting This Class of Issue

The pattern Tailscale was using — SQLite as a primary production database with periodic full-file uploads to S3 for backup and replication — is increasingly common. I have seen it in companies building internal tooling, in startups avoiding the operational overhead of PostgreSQL or MySQL, and in infrastructure companies using SQLite as the state backend for control plane services. The pattern has a lot going for it: SQLite is operationally simple, the backup mechanism is straightforward, and the consistency model is easy to reason about.

The challenge is that this pattern almost invariably involves taking manual control of the checkpoint process. If you are uploading the SQLite file to S3 at a consistent point in time, you need a consistent snapshot — which means you need to know that the WAL has been fully checkpointed before you copy the file. This pushes you toward manual checkpoint control. And manual checkpoint control, at aggressive frequencies, is exactly the operational configuration that increases exposure to WAL-related race conditions.

The alternative patterns — using SQLite’s streaming replication extensions, or using sqlite3_backup_init and sqlite3_backup_step for online backups that do not require a clean checkpoint — are more complex to implement correctly but avoid the manual checkpoint dependency. If you are building a new system on this pattern, I would evaluate sqlite3_backup_init seriously. If you are running an existing system with manual checkpointing, understanding exactly which versions of SQLite you are running and whether you have upgraded past 3.51.3 should be a priority.

None of this changes the fundamental correctness of SQLite as a database. But the combination of “SQLite plus S3 backup plus manual checkpointing” has a specific risk profile that the Tailscale incident has now quantified. You should understand that risk profile before assuming you are in the safe zone.