Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Lo and behold, a nice arithmetic coding implementation that wasn't written by an LLM! A sight for sore eyes – a treat, even. Looks like it was written by someone else though.
Check it out: https://github.com/samyak112/pym-particles/blob/main/arithme...
by jxmorris12 - Ohh yeah , I took it from Project Nayuki as mentioned in the file as well, i tried to pip install it but there were some issues so just took the file and kept the copy right as it is.
Its not an issue is it? I am not sure.
by spidy__ - Neat approach. Since the 900KB model ships with the compressed file, is there a file size below which the model overhead just eats the gains? Curious where the crossover is.by tae0086
- For the model overhead to become significant enough to eat into the gains, the file size would need to be fairly small, right? I assumed nobody would use this for compressing anything below 100 MB.
I tested with 100 MB files because anything larger takes a long time to evaluate. The actual target was at least 1 GB, and in that case I would use a 100 MB model (Shannon entropy rules).
I also tried it on a 100 MB Photoshop file and was able to compress it down to 45 MB, whereas ZIP could only get it down to 60 MB. So yeah still not losing gains.
by spidy__ - Dumb question: can you train a model to predict the next byte of ANOTHER MODEL
So apply this same logic to compressing a bigger model within a smaller model
I know this is absolutely regarded, but humour me please
by purple-leafy - If there's any redundancy in the model that can be compressed (parallel to how RLE is used to compress the static Huffman tree in FLATE) that's possible, but it's not necessary if the model is being trained on the input dynamically, like what Bellard's NNCP does.by userbinator
- Not dumb at all. It's a whole field of active research - Speculative Decoding. A recent paper goes one level deeper with Speculative Speculative Decoding - https://arxiv.org/abs/2603.03251by anyg
- I've had this idea of building a codec that would similarly overfit to specific images. But the codec itself would not be a fixed size transformer... instead you could just mess around with the sizing to get better quality/smaller size.
So the codec would be something like: <header describing image size + transformer layer shape> <transformer data itself>
I've seen experiments where people have a "fixed" pipeline but I think having something more dynamic would work quite well.
by rtpg - Likely doable with metaparameter tuning (used to work on a team with data scientists that were routinely doing this in various situations). Seems like a cool idea.by dvt
- Great work. Just Yesterday I thought about LLMzip and asked myself if this is something which could vastly improve HTML compression when done at Google scale and shipped with browsers. I haven't done any research though.by VorticonCmdr
- I mean neural compressors provide great compression, BUTT the issue is they are really slow like in my project it takes around 45 minutes for de compression of 100 mb so I doubt if it would be useful, also using a transformer in user's browser sounds like a heavy task.by spidy__
- Recently read about some upcoming GPU chip vendor that speeds up multiply-and-add (the core of [pretty much any neural network, including LLM's) oepration by transforming them to log scale where the multiplication changes to addition.
There issue was the size of such log tables that would be needed, and hence, they settled for some trade-off.
My question is: can the expensive log-domain addition / correction function be implemented as fixed hardware lookup tables or approximate units?
Update: found the vendor: https://www.tensordyne.ai/silicon-and-math
by freakynit - Three questions:
1. How much was AI used to generate documentation for this project?
2. The 100MB CSV data sources are not provided in the repo so it doesn't seem possible to reproduce your results. The enwik9 dataset says it is a "slice" of the larger data set, and there are many NYC taxi trip record datasets that exist. Can you provide the datasets used to generate your results?
3. I am surprised to see performance comparisons only between your transformer and WinZIP. What were your results when comparing your transformer to more modern approaches like LZMA2 (level 9), BZIP2 and ZPAQ (max effort)?
- 1. I wrote the content as what i want to mention in the documentation and just used AI to polish it so that its easy to understand, is it hard to understand the documentation right now?
2. Have added the link for downloading both the enwik9 slice and the nyc dataset. Apologies I forgot to add it.
You can get it from here - https://github.com/samyak112/pym-particles/blob/main/README....
3. Other than zip i tested it with zstd19, and now that you mentioned LZMA2 and BZIP2
I got results on enwik9 100mb slice as
zstd - 28mb bzip2 - 30mb lzma2 - 26mb
I will mention these and results from ZPAQ in the readme for both files, thanks for pointing them out!!!
But the thing is this neural compression approach cant be used right now, as it takes hours to compress and de compress a 100mb file so not really usable and more of a fun project.
by spidy__ - Somewhat related is stavros's method to compress 500KB to something like 50 bytes https://www.stavros.io/posts/compressing-images-with-stable-...
main drawback is that it's not lossless ;-)
but this is great. I hope this actually becomes a format that wraps the weights and transformer module (maybe this can also be NAS-optimized too?). Maybe it would even work for video?
It's like calling gzip but instead of compression level you choose kolmogorov complexity level
by whacked_new - Maybe it would even work for video?
While clearly satirical, it's definitely quite thought-provoking from various angles including the basis of information, representation of data, and even copyright. It's like watching a movie, writing a book based on it, and then making another movie based on that book.
by userbinator - > There are some minor kinks that need to be worked out, such as the fact that each image takes around a day to generate on mobile, but this is more than acceptable in certain domains. Website visitors, for example, are well-accustomed to such loading times, and would barely notice any difference.
Just amazing, wow
by isoprophlex - The model is the important part, a huffman code or adaptive huffman or other sorts of encoders would be much better on a dataset based on the model. You need the model to also decode. And on a dataset of sufficient size, embedding the model and the benefit of it's memorization of the file can be offset.
A non-general compression algorithm (model - I don't mean a distinct llm, but "modeling data") targeted at a specific dataset will always do better than a general algorithm.
The reason I mentioned the "encoder" doesn't matter - arithmetic coding, for the data it is presented, will beat huffman/adaptive huffman every day, but it's the model that is where the real "compression" comes into play.
I've implemented enough "coders" over the years, including arithmetic for both commercial and research purposes (was a student of Glen Langdon).
by jmspring - What does it compress the full 1GB file to? http://prize.hutter1.net/by 7373737373
- Maybe everyone should compress the 1st 100MB worth of digits of pi, for an apples-to-apples comparison?
Edit: oh wait that's too easy. Need to generate /publish random digits so everyone can use it.
by cellular - Thanks for the link!by purple-leafy
- I tried it on a enwik9 100 mb slice and was able to compress it to 20 mb + 900kb transformer so 21mb.
I know the top submission was able to get it to 13 mb.
Still trying some ideas to get better compression.
by spidy__ - What do those compress to with conventional approaches? For comparison.
I am curious. A classic machine learning ensemble approach is to overfit a collection of small models then bag them (e.g. voting) allowing the models to generalize.
I'm sure someone's tried to overfit a bunch of transformers for compression like this, then bag them to see how well it does?
- conventional algorithms https://www.mattmahoney.net/dc/text.htmlby fsiefken
- Ensembling is not compute or parameter-efficient, so compression per se is a terrible application. (This is related to why people train ever larger LLMs like 1 10t-parameter LLM, rather than 100 GPT-3-scale LLMs.)by gwern
- Fabrice Bellard may have been the first to do this, 7 years ago: https://news.ycombinator.com/item?id=27244004by userbinator
- Yeah yeah, I just found the idea kinda interesting so wanted to implement itby spidy__