Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I'm curious why the test needed so many router hosts:
512 shards, each with one Postgres primary each on an r8g.16xlarge
480 Neki routers, each on its own 8xlarge instance
That's ~250K queries/sec per router which seems lowish for this type of workload? The routers won't be doing very much (parse query, route it to proper shard?).
by AdamProut - that's over 13k queries/sec per router core, about 50% of it is spent doing syscalls, the other 50%: parsing, doing grpc, tls, go gc, resolving the shards, waiting for the responses
neki is still in platform preview, this experimentation allowed us to collect profiles at such scale and ship some nice optimizations, more are coming
by svuiv - The fact that you can just pay to scale out point reads is not news to anyone.by jeffbee
- yeah, this is a definitely a "best case" workload for a sharded database. Single row reads on the key used to shard with no hotspots (no shard to shard network traffic at all).by AdamProut
- If this is closed source then i have zero interest in it.by znpy
- Someone posted a twitch conversation yesterday about this, I poked around on the page realised there was no open source version and noped out immediately.
I'm sure it's a great product (it seems like planetscale do good engineering and the folks I know who use them seem fine with it) but I don't do vendor lock-in as a service personally, I'll use whatever employer uses because that's the deal but for personal stuff, well this isn't designed for that really, wrong order of magnitude on scaling.
by noir_lord - What company do you work for that makes its tech decisions based purely on ideology without regard to engineering merits?by throw10920
- I believe multigress is the similarly aged open equivalent from Supabase. Haven't used it myself and don't know what the differences are in usability, but I'm a bit more interested in that since it's open.by jjice
- It cost $250,000 to do this run but it feels worth it.by samlambert
- I did not know men could build such things.
- I estimated the cluster to achieve this was ~$3-4k per-hour. I am thinking there is a typo on the r8g.16xlarge and they are actually r8gd.16xlarge (notice the d) which comes with directly attached nvme disks.by whalesalad
- That's roughly 250x more than it would cost to perform this stunt using on-demand Cloud Bigtable, if my math checks out (~1150 nodes @ 85¢/hour for 1h).by jeffbee
- Its great to see scale and progress, but being closed source is HUGE DEALBREAKER.
Clickhouse is also on the right track of building some amazing opensource integrations with postgres, they have superior*[1] managed postgres looks like from their recent blog. I hope they do some OSS sharded postgres solution.
[1] - https://clickhouse.com/blog/benchmarking-nvme-managed-postgr...
by weekendcode - If you want an OSS sharded Postgres, just use Citus. Feature-full, proven, mature, boring.by ahachete
- Sai from ClickHouse here, I lead the Postgres efforts at ClickHouse. Expect news from us on this soon! Many of us here are ex-Citus and have done this for Postgres before.by saisrirampur
- 87.3 % served from cache. Does that mean it returned a result existing in the cache because the very same query was executed before? Probably still a relevant result, if you have to process millions of queries every second, it seems not unlikely that you will see a lot of repeated queries. But at that point you are measuring cache performance more than query performance. But unless you run some standardized query benchmark, a single queries per second number is not that informative anyway because query complexity and therefore execution time can span many others of magnitude. Looking up a name by ID and aggregating across a billion rows from seventeen tables joined together are both a single query.by danbruc
- I did some quick ChatGPT math for the same performance/storage as the benchmark:
Neki 1 primary + 2 replicas: ~$5.0M/month (just the AWS bill)
Google Spanner w/ 3 replicas built in: ~$3.85M/month
by cbg0 - In 2015, MySQL Cluster (NDB Cluster engine) benchmarked 200m transactions/second on commodity hardware [ref]. It was read-committed transactions, not snapshot isolation, but still impressive. NDB has now become RonDB, but is still based on a non-blocking 2-phase commit protocol and is GPL-v2.
RonDB now has support for infiniband, so ought to blow through the 1B ops/sec. For reference, that is 1 GHz of transactions/sec.
[ref] https://www.slideshare.net/frazerClement/200-million-qps-on-...
by jamesblonde - I watched an interview that Casey Muratori did with Tyler Cloutier (SpacetimeDB founder and spokesperson) [1]. One of the points that Tyler is that, given modern CPU architecture with cache lines, a distributed database needs to fan out to at least 50-100 nodes to beat the throughput of a cache-optimized, single node database.
It’s cool to see the flip side of that argument. Planet scale is answering the question, “what does it look like when you DO fan out your workload to >100 nodes?”
There’s a place for both technologies. Very cool stuff.
by stephenlf - IME people usually go multi-node for availability and durability reasons, not throughput.by titanomachy
- > I watched an interview that Casey Muratori did with Tyler Cloutier (SpacetimeDB founder and spokesperson) [1]. One of the points that Tyler is that, given modern CPU architecture with cache lines, a distributed database needs to fan out to at least 50-100 nodes to beat the throughput of a cache-optimized, single node database.
Do you have the timestamp where they are talking about this? The claim doesn't pass the smell test for me. If you're talking about latency, then perhaps. On throughput, I don't understand how a single-node system could deliver higher throughput than a three-node system
by malisper - In 2015, I was able to get to 1 million read/write queries per second on only a couple nodes and tested this with multiple databases, it required (at the time) decent network tuning and node placement inside AWS but it cost me about 10 to 15 dollars per run if I recall correctly, obviously there is the matter of scaling such performance and so I want to recognize the engineering effort gone into this but this is too much money. This reminds of when one of my teams used Hadoop to process only a a few terabytes of offline data and were able to process the WHOLE THING in only a few hours. I did not have the heart or courage to tell them during the demo that this was overkill, but I did write a very simple (and small) piece of code that could extract all the signals from the offline files in mere seconds with careful network planning and storage optimization and invited them for a demo/lunch and learn next week.by farazbabar
- What happened next?by kinduff
- > all the signals from the offline files in mere seconds
That sounds unlikely unless those machines had access to crazy disk I/O. A local RAID 5/10 with 8 drives would still take 30-50 minutes just to read that much data. Even with a mid-range SAN you still would have spent 15-25 minutes just reading data. This assumes 7.2K SAS/SATA since SSD/NVMe were not ubiquitous in 2015, but even with 2015-era SSDs you're still looking at half that much time spent reading.
by tomnipotent