Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Honestly the network-attached storage only ever made sense with HDDs, it's ridiculous to have to pay 10x penalty for both iops and latency just for some "operational flexibility". You get way simpler and much more robust architecture if you just stick to local NVMe and replication -- not that you can forgo one in the (also replicated) network block storage solution anyway...
    by nasretdinov
  • > Examples from production #

    > This pattern is not only theoretical, the talk cites public examples from Instacart, which has described PostgreSQL on NVMe for a latency-sensitive search workload

    article basically teaches that computers with local storage has better latency. Wisdom of back to basics.

    Not that it teaches a bad thing or in a bad way, just me expected from Clickhouse guys less obvious things

    by CoolCold
  • I recently interviewed for a company (for the purpose of not making this an advertisement, I will not name it) that uses FPGAs to speed up SQL queries. They claimed at least two orders of magnitude speedups for joins!

    They also told me that they've started as a h/w shop and that they have competitors who are building ASICs rather than programming FPGAs to do the same (with, potentially, even more performance gains).

    This completely changed the perspective for me. Before, I had this idea that if I need to optimize database performance, I'd first look at indices. Then, at how much memory is available to the database for various tasks. Only after that, I'd look at I/O performance, but, mostly, because of fsync configuration.

    It wouldn't even appear to me as a possibility that the compute can become the culprit of optimization. But, apparently, it's a new hot thing.

    * * *

    From my experience with storage testing, which also included quite a bit of pg-bench exposure... My impression was this: the sort of query that is used for performance benchmark matters the most. It will define the optimization direction. Blindly trying to add or speed-up one resource or another is not going to be an efficient strategy. In the database world, there's this idea of an "ideal" benchmark. Without too much detail, it imagines an "ideal" business accounting back-end, the sort of queries it would have to execute. It defines an "ideal" unit that roughly encompasses one "sale", and then it measures how many such "sales" a business can perform per unit of time. This will translate into some ratio of selects / inserts / deletes / updates with some index hits and some linear searches etc. But it's all extremely vague and it's very hard to adapt this benchmark to the actual business needs of the database being tested.

    There's also a question what to optimize for: the resource usage? the response speed? the concurrency? In some scenarios, I can see how storage would factor in, but there are also certainly scenarios where using a faster, in some sense, storage will not result in any meaningful gains.

    by crabbone
  • "Back in the day", we used to have to carefully plan the underlying storage array to eek out every ounce of performance from the database server.

    I think there's a generation of devs that just takes big memory pools and SSDs for granted; there's a mental disconnect to the fact databases map the underlying storage.

    It becomes obvious once I bring it up with them in discussion and I can almost see the bulb go off (happened while discussing partitioning and sharding), but they never make the connection that a query is mapping to some artifacts sitting on storage.

    by CharlieDigital
  • Recently moved our RDS instances from r8g. to r8gd., which adds local NVMe, and I think the average I/O latency went from 1.0ms to 0.1ms. Adds up in big queries!
    by iancarroll
  • CloudNativePG supports Kubernetes Local Storage as one of its main features [0]

      Persistent volume management, including:
    Support for Local Persistent Volumes with PVC templates.
    Reuse of Persistent Volumes storage in Pods.
    Separate volumes for WAL files and tablespaces.
    Chapter Temporary PostgreSQL Cluster Degradation [1] describes how to set it up and starts with this

    > While the standard approach ensures service reliability and leverages Kubernetes' self-healing capabilities, there are scenarios where operating with a temporarily degraded cluster may be acceptable. This is particularly relevant for PostgreSQL clusters relying on node-local storage, where the storage is local to the Kubernetes worker node running the PostgreSQL database. Node-local storage, or simply local storage, is employed to enhance performance.

    [0] https://cloudnative-pg.io/docs/devel/#main-features [1] https://cloudnative-pg.io/docs/devel/kubernetes_upgrade/#tem...

    by smartbit
  • SQLite + NVMe feels like cheating sometimes. If my #1 requirement is lowest possible latency, I can't imagine picking anything else.

    Kestrel (AspNetCore) + SQLite + NVMe all on the same machine is where things start to get really spicy in my experience. Imagine handling an entire form submission that mutates database state in tens to low-hundreds of microseconds. Total wall clock time required to service any HTTP request can be well under a single millisecond. If your average cpu time per request is 30uS, you could approach half a million per second on a 16-32 core machine.

    by bob1029
  • Cloud is insanely expensive and inefficient. Too many engineers don't understand how slow I/O back and forth is on network storage, or VMs with chatty neighbors are compared to bare metal...
    by mmnfrdmcx

Explore Birbla archives