Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- FF bytes are an easy way to identify an invalid UTF-8 file. This idea doesn't have that property.by Dwedit
- At some point it just collapses into a sort of Huffman coding of every possible 4096 bit embedding vector.by bastawhiz
- > Ken Thompson: "...i really dont think it is useful. it is like replacing ipv6 with ipv50"by achille
- > No special cases introduced. All properties preserved.
I don’t actually know if this is LLM-generated, but phrasing like this is weirdly triggering to me now
by stbenjam - Just limit it to 8 bytes at which point you always do 'know the number of follow on bytes' from the first byte.
Nobody needs more than 4.47 trillion characters. (famous last words)
by yyyk - UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(by Sharlin
- > UTF-8000 is in no way endorsed by or representative of the Unicode Consortium.
Not until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.
Someone at the consortium is rubbing their hands with glee with all the newfound space.
But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?
by sph - On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
by 2shortplanks