bzip3

Join the discussion

• Hacker News
• Why LGPL when the original license is more permissive? (https://sourceware.org/bzip2/manual/manual.html)
  by red_admiral
• Please stop questioning people's choice of license.

  If you don't like it, you're entitled to not using it.

  by LtWorf
• The author of bzip3 probably doesnt like pushover licenses so made user rights stronger
  by greyw
• bzip3 is not made by the same author as bzip2, that's why there is a licence disparity.
  by atiedebee
• For data recovery of archival stuff it's better to keep it uncompressed, no?
  by sehw
• Depends on the specific scenario. E.g. efficient compression can speed up recovery from slow media and provide space for increased parity.
  by zamadatix
• So apparently zamadatix's point wasn't well taken, but I think it could be an effective approach to store the same data twice over to increase the likelihood of successful recovery. Could be even be even ten times, or e.g. 6+4 with redundant recovery coding.

  Although you do have a point that the compressed data might be more difficult to decipher, if it doesn't have sufficient redundancy to skip bad parts, or if it is essential that the data is aligned in a certain way (e.g. disk images, and probably many other formats) and the format doesn't take this into account. Shorter window sizes, window reset markers, and explicit offset information could mitigate those problems.

  by _flux
• "DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE."

  So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.

  by sergiotapia
• This big warning gave me pause, too. Why not have a compression option that automatically checks the data after compression (by decompressing and checking against a hash of the original data), which would reduce the probability of undetected errors to that of a hash collision? i.e. like `7z a` followed by `7z t`, but in one command.
  by fmx
• Is that all that different than the standard MIT License:

  > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE....

  by jubilanti
• Tool author here. bzip2 also has this clause, in fact it has been lifted from its dist tarball README verbatim. So does lzma, xz, or in practice any open source program that you use.
  by palaiologos
• I would be interrested in a comparison with openzl
  by ThiraSoft
• That's like comparing apples to pears. OpenZL is not general purpose. You specify a format for data and it compresses that format. Specifying a general "could be anything" format would be interesting, but I doubt it would compress as well.
  by Retro_Dev
• It's distasteful to use such a name when it's not created by the bzip authors. For some reason open source developers love using would be trademark infringing names instead of coming up with something unique.
  by charcircuit
• Besides the oppressive trademark law how isn't it bzip3?
  by mitxela
• Other than the oppressive trademark law how isn't it bzip3?
  by mitxela
• > However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program.

  Sounds like perhaps a nice testcase for formalization + AI?

  by amelius
• It's beyond me why such foundational libraries don't have formal correctness proofs attached these days.
  by teiferer
• I think this should include benchmarks zstd with larger windows and long range mode. I wouldn't be surprised if the window they used is smaller than an individual version tar file, which would prevent useful compression.
  by CodesInChaos
• “Additional benchmarks on the same dataset” (2025) explores various compression levels with and without long range mode: https://news.ycombinator.com/item?id=42901476
  by altairprime
• An interesting unintentional benchmark is to go to https://github.com/iczelia/bzip3/releases and see to what degree bzip3 compresses its own release archives; and go to https://github.com/iczelia/bzip3/blob/master/.github/workflo... to see what options have been chosen for the other compressors here.
  by JdeBP
• I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.

  I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...

  by 8organicbits
• I'd recommend trying openzl for jsonl.
  by woadwarrior01
• For structured logs and json I've found a lot of success with PPM-style schemes.

  If your JSON file has many of the same object, you could see ratios in the single digits.

  by bob1029
• zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems.

  I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

  by nvme0n1p1
• For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.
  by retrac
• The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors.

  If we match the window size of zstd to that of bzip3 we get dramatically different results:

      % gzcat *.gz | time zstd -T8 -16 | wc -c  # baseline
       2819113884
      zstd -T8 -16  2054.50s user 3.47s system 783% cpu 4:22.80 total
  
      % gzcat *.gz | time zstd -T8 -16 --long=29 | wc -c
       196405076
      zstd -T8 -16 --long=29  1083.06s user 2.41s system 783% cpu 2:18.55 total
  

  Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do).

  (the baseline number is slightly different because I don't have the exact Perl version set used by the author)

  Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.

  by ot
• Can you tell me what the zstd invocation is that corresponds to the default invocation of bzip3, which uses block size 16 MiB (according to the man page)?

  I got some really good results with bzip3 compression Wikipedia XML dumps, and I would like to check if it's actually better or if I was just calling zstd wrong.

  by sltkr
• Wow, that is widely disingenuous, I don't really think there is any excuse for that, I don't believe someone deep in compression algorithms wouldn't know they could adjust the block size, and 512GB is a huge block size for bzip3, as it needs to basically all be in memory so you can't pretend that's just 'the standard value'.
  by CJefferson
• This was my point too, but related to LZMA(2?) / xz, as the exact parameters were not specified while it supports setting compression level up to -9e, and the dictionary size can be controlled directly as well (in addition to quite a lot of fine-tuning knobs), increasing it up to 1536 MiB.
  by pajko