Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I think language itself is compression, so the arxiv paper tracks for me.

    Viz. if Language is compression (of thought / culture / the tacit je ne sait quois of being-to-being communication etc.), then definitionally, Language Modelling must also be Compression.

    Except, language is an arbitrarily lossy compressor, who's "compression-prediction equivalence" is indeterminate and unstable, because Language co-evolves constantly; both as a function of or response to culture, as well as an influencer of culture.

    So, the subjective-objective goodness of Language Models (of any kind of language) would be, at best, upper-bounded by the compression-prediction equivalence of the Languages corpus itself. And that is assuming the language corpus is perfect in every way---it captures all knowledge expressible by language and it is always in-sync with live evolution of all language expression and evolution (i.e. LLM training is not a batch job, but a real-time present continuous process).

    For example, to my layperson eyes, the mathematical language of proofs actively weeds out ambiguity of subjective interpretation. Ideally, a proof ought to lead to the exact same conclusion on every single reading by any reader who can follow the steps. A proof also holds only if the rest of the formal, explicit, inviolable, internally-consistent set of axioms and results holds.

    So it stands to reason that mathematical prose of proofs, being optimised as mechanical procedure of taking an open question to a deterministically closed solution, has better odds of approximating the tacit aspects of mathematical derivation.

    Which makes an LLM able to construct a mathematical proof, which is mind-melting to say the least.

    However, I wonder, can LLMs dream of mathematical sheep?

    by adityaathalye
  • I'm more interested in the converse question: how well does an LLM perform as a compressor, compared to gzip (ignoring its insanely lower speed)?
    by tromp
  • This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
    by montebicyclelo
  • Good article.

    Compression is a property of language.

    A seemingly simple sentence like "I had lunch" has enormous amount of information compressed inside it.

    The word lunch is a compressed form of "having food at noon" while "noon" in turn is a compressed form of "Sun's position against Earth's rotation" and so on and so forth.

    Every sentence has layers of compressed sentences. How many layers one chooses to decompress is up to the person.

    by elendilm

  •     give it a normal text prompt, and it
    continues that prompt by searching
    for the byte sequences that compress
    best.
    One moment, how are we supposed to know how well that search was done? There is no way to search a meaningful part of the search space.

    So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.

    The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?

    by mg
  • This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
    by Culonavirus
  • 3blue1brown did a series on this topic: https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=GlYgs6v2YfU (i think one more is yet to release)
    by GodelNumbering
  • Yes: you can classify a test file by topic with gzip as follows:

      gzip -9 sports.txt   testfile.txt

  gzip -9 politics.txt testfile.txt

  gzip -9 business.txt testfile.txt
    (ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)

    The test file belongs to the topic with the smallest size *.gz file.

    Witten's group at Waikato uni were perhaps the first to work on this.

    Also check out the Hutter prize if you are interested in this.

    by jll29

Explore Birbla archives