Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- 1 month later: "Re-writing DuckDB in Rust with Claude Ultra". There is a curios phenomenon, big tech uses great open source projects as a guinea pig for experimenting agent-based development approaches, see Bun, AstroJS.by log101
- Wouldn't be the worst thing, when I last used it, I got some c++ specific errors.by dunefox
- I'm aware of the Bun Rust rewrite, but what has Astro done with experimental agent-based work?
- Maybe they learned AWS were planning on releasing a managed clone without their involvement?by fidotron
- And forced AWS to acquire them instead? Nice trick.by bornfreddy
- That seems unlikely because AWS needs to build proprietary features within their services to lock in paying (enterprise) users. The problem now is that labs will inevitably focus on this work, which I view as opposed to local, in-process when your paycheque comes from selling hosted services.by skeeter2020
- Surprised that there isn't more discussion of DuckDB source distributions.
Why do we need a source distribution for a well regarded MIT licensed project? Because it's not easy to contribute code to DuckDB if you don't work at DuckLabs. The CI used to take 5 hours for a simple bug fix last I looked (may have improved since I flagged it on social media).
There are two that I'm aware of:
Pygmy-Goose is focused on making agentic workflows faster by splitting the repo, making git worktrees cheap and 5 minute cached CI on 3 major platforms.* Haybarn: https://rusty.today/blog/duckdb-extension-distribution-gap/ * Pygmy-Goose: https://github.com/Pygmy-Goose/pygmy-gooseby adsharma - Several "Graph on DuckDB" efforts started on DuckDB and ended up reinventing a basic columnar codebase to innovate on because of these reasons. Even though people didn't document why, lack of outside contributor friendly flows is likely one of the reasons.
KuzuDB folks worked on something called GRainDB in 2022: https://vldb.org/cidrdb/2022/graindb-a-relational-core-graph...
But circa 2023 decided to write their own. Work continues as LadybugDB. One of my long term goals is to find an integration point with DuckDB's table implementation as the "node table". Conversely at some point DuckDB could implement all the join algorithms and LadybugDB's REL table in their code base.
For now, any talk of Graph on DuckDB is limited to DuckPGQ and the more recent entrant DuckGQL (both of which don't touch the storage layer which is the main reason why LadybugDB exists).
by adsharma - I'm an idiot, and a lot of Hacker News goes over my head, but I still read it. Would anyone mind answering this question?
Are databases not a solved problem? Why are there lots of different databases? Why is one faster than the other? What's different between them?
by 0898 - DuckDB is really great. It's definitely the best tool for working with any type of tabular laptop-sized analytical data. You can do some of what DuckDB does with SQLite, but DuckDB is much more versatile and performant.by xnx
- There’s trade offs for performance, access patterns, throughput, latency, concurrency, workloads, rigor, type systems, extensibility… really every characteristic you could imagine.
It’s not a solved problem because each iteration of technology doesn’t just fix the mistakes of the past, it’s an evolution to solve the problems of the present.
by koolba - My very high level take:
Every tool is a trade off between effort to create vs power of the solution.
Effort is generally expensive so most things settle on some general purpose local maximum. If you had infinite effort available, you could build bespoke hardware and software from the ground up to solve every problem. It would be faster and more power efficient than any solution available today.
CPUs win out over integrated circuits because the same CPU can be used for ~every software problem, so by using a CPU you benefit from everyone pooling their efforts to improve the general purpose CPU rather than their own specific niche. But when you reach a certain scale/requirements it makes sense to do something more specific. This is one reason why we have standardized GPUs. Still general purpose but more specialized than a CPU. Or think about how Bitcoin mining moved to ASICs, because they need to do one specific thing as fast and as power efficiently as possible.
So for databases, when you get to specific scale and requirements the same kind of specialization starts to make sense. DuckDB or Clickhouse for analytical loads, TigerBeetle for high scale transactional stuff, etc. And that scale is aggregated across ~all software users, i.e. scale of analytical workloads being big enough to support analytical DBs.
Also as time goes on and industries develop the cost to develop specific solutions can go down.
by pythonaut_16 - It's all about tradeoffs. Same reason there are so many kinds of wheeled transport and so many kinds of bridges.
There's no single set of requirements and desired properties that people have for databases.
What queries does it accept? How does it persist data? How does it manage replication and partitioning across multiple servers? Are questions with many answers and the right one varies by application.
by toast0 - This is an enormously wide question but the quickest way to give you an idea would be rephrasing as:
"Are cars not a solved problem? Why are there lots of different cars? Why is one faster than the other? What's different between them?"
I think the best way to approach the subject in an easy to grasp way is to ask Gemini or another frontier AI to teach you the basics, they will do a surprisingly good job and they'll be able to react to your questions with INFINITE patience.
by luckydata - It's a fair question!
My attempt at an answer: no, databases are not solved.
Specifically, different databases are better or worse for different use-cases.
For example, Postgres is a great "all around database" - you can use it for a lot of different things. As a "relational" database, it's really good if you have a table full of users, a table full of order, and you want to see all orders made by a user with ID=123. You need to answer questions like that a lot (eg every time someone on a website loads a page) and you need the answer fast (hundreds of miliseconds at most)
However, say your use-case is more like... you've got 100 billion rows of billing data ("joe was charged $123.45 on 2026-03-07 for a shirt, blue, size 11, brand foobar") in one table. You don't care much about joe, but you want to be able to find out how much was billed, total, in 2026-03 for blue shirts (or all year for brand foobar, or all time, for size 11). Postgres would struggle with data of that volume - you'd need a really big expensive database. A "columnar" database like duckdb (or clickhouse) might be able to answer those questions better.
Anyway, different databases are better/worse for:
- Large piles of data that you need to query in seconds
- Huge (petabytes) of data that you need to query in minutes, but can query in parallel
- Many related piles (like a standard relational database)
- Cases where you're mostly getting or retrieving single items (key-value stores)
- Huge piles of data that represent a long stream of events in time (time-series datbases)
- Piles of data that look and act more like files (object stores)
- When you need strict transactions
- When your need is very write-heavy
- When your need is very read-heavy
- and probably many others - I'm not even a huge data guy :)
So it all depends on your use-case. There are still cases that are not served well by any existing database - eg "filtering billions of rows, in milliseconds, by an arbitrary portion of several dozen very-high-cardinality columns" (to use an example that came up recently for me IRL) :)
by fishtoaster - This is a real shame. I love DuckDB, and we've adopted/extended it and ducklake very heavily at Krea for our research.
I know the projects will remain open, and ostensibly still contributed to in the same direction, but surely AWS thinks of this as another piece of a product suite to compete with Databricks.
Nontheless, I think the open embedded query engine approach DuckDB is spearheading is larger than one project, and I remain excited about the broader ecosystem (especially datafusion).
by willbeddow - Title is "DuckLabs to Join AWS, Projects to Remain Open Source"
- This gave me a shot of panic when I read the headline - thanks for correcting the record.by ray_v
- I was a real fan from the very beginning, but now DuckDB 2.0 "as a server" and 5. asynchronous I/O [1] looks aimed at something very different from what I thought first.by janpeuker
- Agreed that DuckDB as a server feels far from their initial mission, but how does async IO divert from it?by tuvix
- agree - I was really uneasy with the lede in their v2.0 preview:
"DuckDB has been an in-process database since day one. But people have asked us – very persistently – for a client/server mode, and we have finally caved"
This was already satisfied by numerous projects and products, and feels like a "me too" attempt to capture AI-based workflows. DuckDB always felt like "SQLite for Analytic Data" but I fear these changes and now acquiring the org leading technical direction is where they deviate for good. AWS is so unnecessary for what DuckDB can (and should, IMO) be; MongoDB jumps to mind as a cautionary tale.
by skeeter2020 - Congrats to the founders on their new generational wealth, but also what a shame.
- Very much doubt it’s generational wealth. I’m not sure how much people think open source companies are worth, but the path to VC-scale profitability is incredibly rare.
My guess is that Amazon wants official hosted versions and doesn’t want to go through something like the Redis fiasco with licensing. In that case, they ended up having to support their own development anyway (with Valkey), so they might as well just buy the team.
DuckDB team probably gets a nice package and pay bump, but it’s really unlikely they’re getting hundreds of millions from this.
by tyre - If there's no price tag mentioned, it's likely not generational wealth.by echelon
- Seems like a good time to recommend Apache Datafusion. It's designed to be a library, but works pretty well standalone as well. There's a CLI version, but also Python and Java bindings, as well as a Rust library, of course.
From my own experience I can say it integrates far better into your Rust app than DuckDB does.
Well over 100 monthly contributors, too.
by tormeh - Came here to post the same thing. DataFusion is fantastic.by colinmarc
- Is Datafusion related to Polars at all? I ask because you mentioned Rust integration.by asa400
- It's juts not as user friendly as DuckDB.by dtnm
- You can even use DuckLake with Datafusion: https://github.com/tobilg/datafusion-ducklake-providerby tobilg
- Datafusion has all of the individual pieces, especially if you just want to data analysis, but is far from an embedded database.
Indexes, transactions, a first-class storage format are all things that come included with DuckDB, that you won't have with Datafusion.
by hobofan