Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- So, what’s the effect on memory usage?
And for decompression, the effect on memory usage and timings?
by Someone - For decompression, nothing changes because DEFLATE is asymmetric; compressor can spend however much time to optimize the compressed stream independently from decompressor.by lifthrasiir
- OpenZL is the future: https://openzl.org/
"OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. OpenZL takes a description of your data and builds from it a specialized compressor optimized for your specific format."by pella - The future may be ~ AI-assisted format detection + OpenZL
(~ OpenZL-AI-LLM recognises the data structure, then guides OpenZL toward the best lossless compression path )
by pella - OpenZL is nice, but it's often less useful than you think - it requires that you know the structure of your data, and don't care about inspecting that data outside of your program. I've extracted one too many png files from a word document (by renaming .docx to .zip) to desire OpenZL everywhere... It might be better as a short-term "data in transit" compression than for long term storage.by Retro_Dev
- It's interesting to see just how far Deflate can be taken, and to know that even after decades there is still some (admittedly tiny) room for improvement. Optimal LZ is well-known, and so is static Huffman, but their combination creates some additional inefficiencies(opportunities).
...and of course it's written by someone with a Russian name, and has that characteristic style common to many other articles about data compression.
by userbinator - As someone currently exploring grid searches of encodings + compressor combos, and currently looking at neural compressors that reduce size almost half that of a traditional compressor yet take order from ms -> minutes to operate in either direction, I appreciate a good compression post!by blobbers
- reminds me of the x264 “placebo” encoder settingby jedbrooke
- There is also zopfli and it's decadent ECT that allow for more extreme tradeoffs.by tobijdc
- I love it. So much in computers is trade offs and this was a fun read exploring it.
It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?
by jbosh - Process-intensive, but higher compression has clear strategic value. Distant satellites such as Voyager, where bandwidth is severely limited, could transmit more data using such capabilities. Equally, for long-term archival storage, improved compression would allow far greater volumes of data to be preserved on durable, life-long media formats.by Zenst
- zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themselves.by a_t48
- I also wonder how brotli/lzma would compare here.
Far better, just like anything else based on arithmetic coding. The main distinction here is that the output can still be decompressed with a standard Inflate implementation.
by userbinator