Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Use for vectors and stream smaller projections first?
- it could be really useful for cases where youre repeatedly processing similar JSON structure like in case of analytical events but any plans for language bindings beyond the current implementation?by noobcoder
- > any plans for language bindings beyond the current implementation?
If we need more bindings for the projects we work on - we will implement and opensource them. E.g. recently we added rudimentary JS support (no cursors, just encoder/decoder).
For many reasons, we avoid working on something we don't use ourselves and we are not paid for. But your contributions are very welcome. Also we would be happy to have you as a paying client.
by pshirshov - So... isn't this just a database, and a bespoke one at that?by slashdave
- Yes, it's a specific JSON repiesentation and a tiny database based on it.by pshirshov
- I started on something a bit like this, but using sqlite instead of a custom serialisation format: https://github.com/DavidBuchanan314/dag-sqlite (it's intended to store DAG-CBOR objects but it is trivial to represent a JSON object as DAG-CBOR)by Retr0id
- SQLite was bit too heavy for our usecase unfortunately. We tried it as one of the first options.by pshirshov
- This sounds quite similar to Amazon Ion which is one of the few binary JSON formats that allows sparse reads, and deduplicated keys.
However I found that in cases where you have the requirements of streaming/random access and every entry is the same... SQLite is a really great choice. It's way faster and more space efficient than it has any right to be, and it gets you proper random access (not just efficient streaming), and there are nice GUIs for it. And you can query it.
by IshKebab - SQLite was assessed for our usecase. SQLite representations of our data were much bigger, it was much slower with our access patterns and we didn't need all the power of SQL.by pshirshov
- It is a bit confusing that JSON is being mention so much when in reality this has nothing to do with it - except to showcase that JSON is not suitable for streaming whereas this format is.
Secondly, I fail to see advantages here as the claim is that it allows streaming for partial processing compared to JSON that has to be fully loaded in order to be parseable. Mainly, because the values must be streamed first, before their location/pointers in order for the structure to make sense and be usable for processing, but that also means we need all the parent pointes as well in order to know where to place the children in the root. So all in all, I just do not see why this is advantageous format above JSON(as that is its main complaint here), since you can stream JSON just as easily because you can detect { and } and { and ] and " and , delimiters and know when your token is complete to then process it, without having to wait for the whole structure to finish being streamed or wait for the SICK pointers to arrive in full so you can build the structure.
Or, I am just not getting it at all...
by gethly - Most existing JSON parsers don't support streaming but that's not inherent in the format. It is definitely possible to stream writes easily but it's just as possible to stream parsing.by xenadu02
- I think this quote explains the efficient streaming:
> There is an interesting observation: when a stream does not contain removal entries it can be safely reordered.
So if I'm understanding, the example in the readme could be sent in reverse, allowing the client to immediately use root:0 and then string:2 while the rest streams in.
I was looking for something like this, but my use case exceeds the 65k key limit for objects.
by 8organicbits - > when in reality this has nothing to do with it
It's a specific representation of JSON-like data structures, with an indexed deduplicated binary format and JSON encoders and decoders. Why "nothing"? It's all about it.
Mostly it's not about streaming. More efficient streaming is a byproduct of the representation.
> because you can detect { and } and { and ] and " and ,
You need a pushdown automaton for that. In case of SICK you don't need potentially unbounded accumulation for many (not all) usecases.
> the values must be streamed first, before their location/pointers
Only to avoid accumulation. If you are fine with (some) accumulation, you can reorder. Also think about the updates.
But again, streaming is a byproduct. This tool is an indexed binary deduplicating storage which does not require parsing and provides amortized O(1) access time.
by pshirshov
I've worked on _many_ applications which have needed those features. Object keys is a per implementation detail, but failing at 65k keys seems like a problem people would likely hit if this were to be used at larger scales.Current implementation has the following limitations: Maximum object size: 65534 keys The order of object keys is not preserved ... These limitations may be lifted by using more bytes to store offset pointers and counts on binary level. Though it's hard to imagine a real application which would need that.by qixxiq- Isn't the order of JSON keys not guaranteed by the official spec? I don't remember when I learned that but I have always behaved as if that cannot be relied upon.by ericmcer
- .net has a polymorphic serializer where the output json contains a $type field for deserializer to choose the concrete type.
It needs to be the very first key in the object. I’ve been bitten by this because postgresql’s jsonb also does not preserve the key ordering.
I believe the latest .net release addresses this but key ordering does matter sometimes.
by eknkc - I'd say that it's generally unwise to use fixed-width integers in a data structure where this width can vary widely, and has no logical upper limit. Arbitrary-size integers are well known, used in practice, and not hard to implement.by nine_k
- In our usecase, for which we created the library, we made this tradeoff to save several bytes per pointer and keep binary form more compact. The application splits large objects into smaller chunks. 99% of the structures there are relatively small but there are tons of them. Most likely you can do the same - just split large structures into smaller ones.
If you need support for larger structures, you may create your own implementation or extend ours (and I would really like to hear about your usecase).
SICK as a concept is simple. SICK as the library was created to cover some particular usecases and may be not suitable for everyone. We would welcome any contributions.
by pshirshov - I don't know what kind of data you are dealing with but its illogical and against all best practices to have this many keys in a single object. it's equivalent to saying having tables with 65k columns is very common.
on the other hand most database decisions are about finding the sweet spot compromise tailored toward the common use case they are aiming for, but your comment sound like you are expecting a magic trick.
by halayli