Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Zstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be very effective, especially for more situations where CPU is a bottleneck.
It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data by protobuf field (well, recursively). It doesn't even have to know the message schema. Shame the project is not more widely known.
by chickenbig - Curious how this application scheme compares to filesystem + transport compression. You'd end up potentially compressing and decompressing more often but the higher software doesn't need to know what's happening and the compression happens in kernel space.
ie btrfs
You could also layer on out of band dedupe and probably push out cache updates with btrfs snapshots although maybe that ends too convoluted
by nijave - > We initially considered limiting transcoding to popular content
Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.
by MayeulC - I'm surprised the faster decompression speed of zstd wasn't mentioned. Feels like a big win for cache (assuming decompress on read).
A shame browsers don't support concatenating zstd frames (which zstd does support). Then you wouldn't need to decompress, just serve zstd straight to the browsers after concatenating the frames you care about.
by andersmurphy - Why not serving files compressed if the client supports it even though the origin served an uncompressed file?by thinkindie
- I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.by CodesInChaos
- One thing that bothered me enough to comment on: "transcoding" doesn't seem like the right term in this context. "Encode/decode" is technically correct, but "compress/decompress" would have made the intent much clearer.by nithril
- Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files.
For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks.
There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!
by r3trohack3r