F3

Join the discussion

• Hacker News
• Great! I'll use it.

  In the "future."

  Nimble? Lance? Also in the future. Maybe.

  I'll use Parquet in the present.

  by thisisauserid
• File format for what? Text, graphics, compiled code?
  by krzyk
• The future.
  by meindnoch
• For columnar data storage I think. The description references Parquet and they appear to benchmark against Parquet, Vortex, and Lance.
  by ghkbrew
• My concern is, if decode fails I need to debug WASM added by some other party maybe containing random bugs. Maybe a library of standard decoders maintained and tested by the project could help, but then not sure if it kills the advantage of the flexibility it provides.
  by anygivnthursday
• But Wasm has deterministic execution, so if decode fails for you, it should have failed for them. I.e. it's not a problem that your system has introduced; they should be able to reproduce the failure independent of any client.
  by titzer
• If I am archiving PBs of data for 10+ years, I don't want to rely on a WASM interpreter being available and performant in the future just to read a file. I want a dead-simple, heavily documented byte specification like Parquet.

  Additionally, putting the decoding logic inside an WASM binary introduces an active execution layer into what should be a cold storage.

  by coffeecoders
• You don't want to run a custom 10-year-old data parsing function every time you read a single data record?
  by 0xbadcafebee
• WinRAR format does include RAR VM bytecode as part of the archive to achieve state of the art compression in media files. it was sandboxed and well accepted by everyone.

  the same sandboxing capability exists for WASM as well.

  it is actually better for long-term archival: you dont need to carry decompression program, since it will be a part of the archive file itself

  by bijowo1676
• Nice! The world can always use a better data format.

  I think you might get some traction if you post the advantages over parquet and other files directly on the readme, so that if someone goes to https://github.com/future-file-format/f3 the see why they should try it.

  Mention the advantages and post metrics. Cherry pick the metrics! There's probably a good use case for this but, from the current readme, it's not clear who should use this and why.

  by owentbrown
• One nice thing about some modern formats is that there are tools that read them at extraordinarily high effective speed. For example, DuckDB can do all manner of nifty optimizations while reading its own native format or Parquet. And I’m not sure that those optimizations can be effectively applied to a format that needs a WASM blob to be run to understand it. By the time you run a non-SIMD or even a SIMD-optimized pass over app the data, if that pass doesn’t understand your query, you may have already lost.

  I admit I only skimmed the beginning of the paper, and maybe the format is less general than it sounds.

  by amluto
• My understanding is it’s a fallback mechanism
  by hahahacorn
• Hm. I can kinda see it replacing self-extracting EXEs, but a lot of why you choose specific file formats is for specific features they offer - any self-describing system can fall into "there are too many competing features and nobody handles them all" exactly as easily as any other format.

  Like, can this file be efficiently mmap'd? Maybe if it emulates tar internally, but you don't know until you run it. Can it be seeked to specific bytes to only decompress part? It only supports a pre-release version of ISO-36898533 seeking, and your file library dropped support for it 6 years ago. If I rewrite 1MB in the middle, can it only change those pages on disk (and maybe an index), or do I have to rewrite the whole thing? Well the wasm blob supports 97 different APIs for it (there are 35 copies of one with different names), so it's larger than the data (but nobody paid attention to that), so you have 19 options that you recognize, but your CPU's native WASM accelerator only handles two or three so you've still got to specialize your code heavily.

  At least with "*.tar.gz" you have some idea of what's possible.

  by Groxx
• Some folks described it as genius. I guess it's my turn to play the role of an annoying HN skeptic: I find it somewhat silly. Data compression formats are secondary to what you're planning to do with the data once decoded. An audio file is completely different than an SVG image. An embedded VM that decompresses video to raw pixels doesn't magically let you play that video in a text editor, so there's no radically new kind of interoperability. Each new format still needs to be handled in a format-specific way.

  I guess one use case is that I come up with a video compression scheme that's better than H.265, but not all platforms support it, so I embed a decoder that would allow me to play it back on legacy hardware. But that also shows the weakness of the idea: it's unlikely that legacy hardware will perform well doing software-only decode for video formats from the future. If we rolled this idea out in the 1990s, it would not have allowed watching Netflix on an i386.

  In the same vein, I doubt this would have allowed me to open Word 2021 files in Word 97. There's no 1-to-1 mapping between the data structures. So if this kind of compat isn't slam-dunk, what's the goal?

  The downsides are clear. First, it's probably a maintenance nightmare: if your decoder has a bug that needs fixing, how do you patch all the files that already embed it? And then, there's size overhead and security risks. We're adding a considerable attack surface to every format parser. It's more opportunities for remote code execution, resource exhaustion attacks, and so on. Again, this is not always wrong, but what's the benefit?

  by zerobees
• I don't think you have encountered the problems that this class of formats solves. Try looking up columnar storage formats, the pros and cons are pretty well defined these days. It is not meant for video decoding, indeed.
  by vouwfietsman
• This could use a bit more "why".

  Shortcomings of Parquet are mentioned as overcome by this, which ones? Certainly not wide tool support...

  Why should one leave Parquet or ORC for this structure?

  by largbae
• Paper mentions Parquet, ORC, Nimble, Lance, TSFile, Bullion, and BtrBlocks.
  by dj_axl
• Yeah it seems like most of this can be handled by some more dev hours to Parquet
  by skrtskrt
• I also had no idea what they were talking about, but there's good points about how hardware oblivious and somewhat global is Parquet around metadata.

  I found this post interesting,

  - https://medium.com/@reliabledataengineering/f3-the-future-pr...

  by dietr1ch
• The ‘why’ is referenced in the bibliography at the end of the readme; this repo is not meant to be consumed standalone. Start with the paper instead:

  https://doi.org/10.1145/3749163

  by altairprime
• I don’t know what are people commenting on. I see a README with little to no information about what this is, what problems it solves, just links to its Flatbuffer description and a directory full of source code.

  What context am I missing?

  by sph
• There is a linked paper: https://dl.acm.org/doi/epdf/10.1145/3749163
  by burkaman
• This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist:

    > "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native decoders are unavailable. "

  by gavinray
• except you need flatbuffers to access that blob
  by vouwfietsman
• How is wasm better than C bindings?
  by grodes
• Applets redux.
  by cbm-vic-20
• Is embedding executable code into a file a security risk? My assumption is a yes
  by verdverm
• I would call it clever. I'm not sure I'd call it genius.

  When I'm working with data I'm working in a specific set of languages. Usually one. Yeah, other people might be working in other languages, but no individual author really needs a language-agnostic way of accessing data beyond compile time. Add to that the likely runtime boundaries that may need to be crossed instead of e.g. inlined by the compiler because it's in-language and dealing with known offsets or tags (depends on the data format of course). To the other commenter's point, am I going to have to sandbox all data access code just to be sure it's not able to do something unexpected? There's a lot of complexity here. And the inherent risk is going to slow down the operation that should be the simplest and fastest: interpreting bytes.

  by andrewstuart2
• It sounds neat, but feels like it might fall apart with higher-complexity formats. What does an embedded decoder for a PDF look like? I guess since they are tightly-coupled to the file bytes themselves, the author of the file gets to choose what formats make sense, but not all formats have a one-true-decode-step.
  by rebeccajae
• I don't understand how that's supposed to work. What does the decoder decode into? That's gonna depend entirely on the kind of data, right? For some formats, it's gonna be a stream of bytes; for others, a 2D plane of pixels; others again will need vertexes, 2D planes of pixels and UV maps; for some, an object graph will make more sense.
  by mort96