Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Great writeup, and it was great to see them step in an pay the developers of SQLite to help them fix the bug. I get tired of corporations asking open source authors to fix problems that affect the corporation for free. And while I'm sure it was frustrating for folks to have these outages, I find such puzzles pretty fun to get to the bottom of.by ChuckMcM
- See perhaps recent video "Reliability Lessons From SQLite - Richard Hipp | SSW 2026":
> Abstract: SQLite is a C-language library that implements a self-contained, in-process relational database engine supporting full-featured SQL, an advanced query planner, and ACID transactions. By many estimates, SQLite is the most widely used software library in the world today.
> Over its 26-year history, SQLite has gained a reputation as software that "just works". This talk goes over the design choices and development practices that have, at least in the opinion of the lead developer, resulted in that reputation.
by throw0101a - Block device upfuckery layers are powerful against databases. Years ago some colleagues wrote one that provides most of the hazards described by "Parity Lost and Parity Regained"[1] to test FoundationDB, which immediately uncovered several flaws in a project that described itself as well-tested. It's easy to do this with all the probing features that Linux (and others) provide today.
1: https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
by jeffbee - I'm definitely going to use the word "upfuckery" instead of fault injection the next time I need it.by jnwatson
- With opus 4.7-ish to fable 5, immediately after release - before they locked it down, it was shockingly easy to find crash bugs in a lot of very heavily used DBs and other software.by dilyevsky
- For this particular category of bug, SQLite's existing testing methodology is demonstrably outclassed by modern deterministic concurrency testing.by colomo
- If you already know the bug is caused by a race between writing and checkpointing then it's easy. I'm unsure how the linked article makes your point at all though.by Kwantuum
- >In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups.
> running boring technology in a non-standard way is a risk.
It was a good read and reminder that the industry is loosing experts gradually. I am not a DBA and yet I have heard about this behavior at least couple times in the past as something to avoid. Its just one of those things which didnt get a chance to be documented cause experts avoided it and regulars didn't get into
by sandeepkd - In other words there exists a concept of HOT and COLD backups for this reason only.by sandeepkd
- I've never heard of any reliability reason you shouldn't run checkpoints whenever you want - only performance reasons. Can you elaborate?
Backing up sqlite by copying the file (e.g. rsync) while it's open is a surefire way to eventually get corruption caused by a race condition, but it seems like tailscale wasn't doing that. They were probably using the proper sqlite backup API.
But you don't need checkpoints for consistency and I think the backup API will not copy both the old and new versions of pages just because they're in the WAL, in other words I think checkpointing then backing up should give you the same pages as backing up without checkpointing. So the whole thing seems unnecessary.
by inigyou - Great read. So glad they took the time to tell this story. (And glad they, as a for profit corporation, took out a support contract with SQLite. I hope they continue to do so even though this problem is resolved.)by bobtheborg
- This was really, really interesting - what a triumphant adventure.
A few (very, very, very pedantic) things that stood out:
> We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky).
(Emphasis mine) - it would be "risky", not "potentially risky" - then the "calculated risk period" starts and it's "potentially problematic".
In the SQLite report[0] (11.2) I wish they downplayed this less - a mention of the rarity, then technical details - I'm friendly with a few of the devs/previous-devs, have the utmost respect for their skill and accomplishments (and by extension, faith that the developers I do not personally interact with are also excellent), appreciation and fondness for the huge accomplishment that is SQLite, and on and on... this is world-class work. Maybe section 11.2 wasn't really aimed at me, or I'm too critical. To be fair to all involved, what a minor quibble for such an interesting problem/fix. I hope my comment isn't a fly in the ointment.
Last bugfix point[1] - ugh. What a sinking feeling that must've been to deploy a fix then be flooded with not-green - and a lesson[2] against smuggling other changes in a changeset "just because we're already here"? Happy it turned out non-catastrophic, but did result in a rare (not remembering other instances of top of head) recall[3] from SQLite. That it was throwing errors at the same time SQLite and Tailscale were testing the other WAL-issue bug must've upset some stomachs for a moment.
[0] https://sqlite.org/wal.html#the_wal_reset_bug
[1] https://tailscale.com/blog/sqlite-wal-reset-bug#fixed-with-a...
[2] Nobody conceptually learned anything here - we're all just reminded of what we know: that sometimes "perfect storms" do actually occur.
by bch - Very nice article, and I appreciate SQLite's explanation of the bug too. And how extremely cool Tailscale appears to have been about it (paying for the VFS shim, etc.).
I'd have liked to have heard more about the decision to checkpoint so frequently that put them on this path though. Presumably that's to keep the WAL tiny for very fast recovery. Trying to mitigate some of the deleterious effects of inserting a DBMS into your network layer, I suppose? Tricky stuff. Wonder how that compares to typical etcd snapshot frequencies too.
by procflora - Was wondering the same, doesn’t mention if they tried checkpointing less frequentlyby nujabe
- Agreed, I was also wondering about this. Maybe the aggressive checkpointing was for preventing WAL-overflow?
- SQLite: 92 million lines of tests
Dijkstra: Tests can only prove the presence of bugs, never their absence!
by andai - by _justme
- probatio diabolicaby altilunium
- The idea of testing is to sample and test the paths you care about, mostly business workflows, not to enumerate infinite combinations.
It's an art to come up with a great test suite that covers just enough and minimizes overlap, not only survives but also helps with refactoring.
by namelosw - Testing:
The union of a lot of necessary conditions is not a sufficient condition. But it might be good enough for software.
by fghorow - It can prove absence of specific bugs though.by 0x457
- I admit to curiosity as to whether static analysis could have caught this. E.g., Rust's type system (yeah yeah I know) catches all data races, unless they originate in unsafe code, which this one might or might not have; a hypothetical Rust SQLite would probably need a lot of unsafe (https://github.com/tursodatabase/turso has 556 unsafe blocks in the core), and I don't have a sense of whether the particular part that contained this bug would be included in that.
- Donald Knuth: Beware of bugs in the above code; I have only proved it correct, not tried it.by dist-epoch
- Everyone knows that tests don't prevent all bugs. But they are very good at preventing known bugs from recurring in the future.by otterley
- It says a lot about sqlite that a bug becomes front-page news on HN. I'm impressed that Tailscale took this seriously enough to engage with a commercial support contract. I'd love to work for a company that cared so much about correctness.by anitil
- Indeed, SQLite has got to be one of the best tested pieces of software with famously 100% test coverage.
https://sqlite.org/testing.html
It's amazing that a bug could exist for 16 years but it is sobering.
by Lio - Well written post, really enjoyed reading it.
> A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used.
This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQLite page[0] outline that it can only ever occur if there are multiple connections open, so the writer and the checkpointer must have been on different threads.
- I don't know enough about the scale of Tailscale's operations to comment strongly on this, but if they're fairly significant shouldn't that have read "is exactly how MariaDB is meant to be used" or "exactly how Postgres is meant to be used"? SQLite has a "lite" in the name for a reason, but it's often pushed into places where it's being asked to do things it was never really designed for.
- > The bug only affects databases in WAL mode when there are two or more database connections open on the same file, in separate threads or processes
To be honest, I'm surprised that someone using SQLite would try to access it directly from multiple threads or processes without fear of data racing.
by Naru41 - Database corruption is due to 2+ peer connects with 744 file permission entering header rwxr -.- WAL write new content into secondary header tag: inter-element whitespace.
Bug details: