Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Can't you solve it using hash trees (or Merkle trees) ?
You tag each computation nodes with a hash of its dependencies and some constant salt, that gives you an ID which identifies the results that the computation node would produce; before running it.
You can then use those IDs to index the computations results in a cache; whenever you query a computation results, as long as you update the IDs of each leaf of the computation graph, you will only re-compute the nodes that need to be updated
by evomassiny - What you're describing is akin to a basic pull-based incremental engine, akin to salsa. The base design is straightforward, but you need some additional logic to avoid following the whole execution tree when hashing. Their downsides is that sometimes you do have to follow the whole execution tree, even if nothing changed.
Push-based designs instead "push" changes to their dependants, which can be quite efficient especially in the case where the update doesn't propagate much. However it has the downside of potentially requiring to update nodes that are no longer used, or updating nodes multiple times.
by SkiFire13 - Parallel Self-Adjusting Computation[0] is a fun entry in this genre.by erichocean
- This library is based in part on that same paper.by atombender
- Oh cool I was working on something similar awhile back and couldn’t find much precedent. The project got retired as our use case went away. But I’d love to revisit it. It’s still on npm as “data-rambler”.
Idea was a DSL that could be loaded into a JS runtime, then fed data streams. Your module would transform that data into various output streams which could then be fed to a separate reporting library. Powering dynamic reports based on a template.
Our first iteration was already pretty powerful but I had some big plans to bring it into a more JS feeling syntax to reduce complexity.
by shortercode - I have built something similar like this for my fund 7 years ago. We were doing parametric optimization on large computational graphs. I have never programmed Ocaml but my understanding is introspection is kind of a weak spot for the language. Curious language choice! I know Ocaml is fast, about 1-2x speed of C, on par with Java.by mempko
- Not trying to be pedantic but do you mean half the speed of C?by hankbond
- One of my favorite tech talks is "Seven implementations of incremental": https://www.janestreet.com/tech-talks/seven-implementations-...by lilactown
- Electric Clojure does incremental rendering that crosses the client/server boundary: http://electric.hyperfiddle.net/
The closest thing to Electric (IMO) is SolidJS, but frontend only: https://www.solidjs.com/
by pgt - If you find this interesting, also check out their UI library called Bonsai that built on top of Incremental: https://github.com/janestreet/bonsai/
Libraries like React are pretty efficient with skipping work by using Virtual Dom, but constructing this vdom still takes time. Bonsai makes the vdom incremental and it is pretty fun to work with.
I built a desktop UI library with it by targeting (now unmaintained) Revery. It is using a much older version of Bonsai however: https://github.com/ozanvos/bonsai_revery
by osener - One thing I've never fully grokked is how this differs from an observable pattern where one can publish new values to inputs, propagate that through the computation, and push newly computed values to listeners.
I guess there's probably optimizations around change detection and stopping the propagation if there's no change (though observables can do that as well). The stabilize command also makes things interesting as a way to batch changes together before recomputing (but again, doable with observables too).
Is the delta primarily coming from introspection and automatically building the compute graph? Or is there something more fundamental that I'm missing?
by RandomBK - I mean, it's fundamentally just a graph, but it's a way of correctly and efficiently computing changes in massive, dynamic graphs. Let's imagine you have a diamond shaped subgraph that fans out to hundreds of intermediary nodes before collapsing down again via and paths with different "lengths". And what if some of those paths have e.g. min(A, B) where the max side is the only one changing?
A naive observer approach will 1) compute that potentially exponential blow-up very inefficiently and 2) probably have "concurrency" issues. This library will be close to optimal and correct, even if you start dynamically changing the graph structure.
But yes, you can achieve the same thing with observers and other kinds of approaches. Most of them just a lot harder to get right while avoiding performance cliffs.
- https://www.scattered-thoughts.net/writing/an-opinionated-ma...
Can help map out the landscape.
by iamwil - Roughly, you subscribe and listen to an observable. Incrementals are more like a cache across some DAG of computation + state that lets you optimize by only recomputing what needs to be recomputed.
There's a really good talk from Ron Minsky here: https://www.janestreet.com/tech-talks/seven-implementations-...
by blovescoffee - It depends on how you define the observable pattern.
The fundamental components here are laziness and weak connections between graph nodes. Node values are getting materialized only when you observe them, and the system is flexible for live structural changes.
Usually, you don't need to materialize the entire graph when you need to observe just some nodes. Additionally, you can halt computations at any point in time leaving the graph in semi-actualized state, make extra changes to the inputs, and continue materialization of the nodes of interest. The algorithm will sort out all changes for you.
Essentially, incremental computations is just a term covering these features. You can organize the same system in terms of observers and subscribers.
Perhaps, classical Excel spreadsheets is the best illustration of the idea. Also, see my article on the topic: https://medium.com/@eliah.lakhin/salsa-algorithm-explained-c...
by Eliah_Lakhin - I was very curious about Dataflow programming years ago - I think a lot of people were coming at this problem from various angles. This specific library immediately reminded me of Javelin from Clojure [0]by djtango
- This is cool.
As far as I can tell, incremental the library aims to solve the problem of partially hydrating a computation graph when source data is altered. This approach is similar to the one pursued by (well designed) build systems and is common in the FP world. [2] This has many use cases and is very cool.
In addition, in the sphere of incremental computation, there exists Differential Dataflow, Timely Dataflow (adjacent), and DBSP. Systems like Feldera are built on DBSP. Materialize is lead by some DD guys.
Personally, I am pursuing an orthogonal approach specifically for the problem of financial data and financial workloads, There exists huge, very important problems to solve! [1]
[2] Signals And Threads episode on the subject https://signalsandthreads.com/build-systems/
- Check out https://github.com/ila/openivm which implements a very large scope of aggregations as incremental operations in an SQL-to-SQL compiler, and extension for duckdb that automatically maintains a materialized viewby hantusk
- > https://modolap.com
Redirect to a 2k USD stripe payment with no explanation when clicking on the main callout button is a pretty baller move.
by valzam - Goldman took the same approach with instrument pricing ~30 years ago. I recall long discussions about "Node Purpling" in my ~13 year tenure there.
Computer Science has evolved, and AFAICT this is not a graph approach, but things like differentiation are computationally expensive, and therefore you want to minimize the number of times you do it to as close to the theoretical minimum.
Edit: Related HN discussion https://news.ycombinator.com/item?id=36006737
by fadesibert - Yea and this created "bank python" informally a good article is here.
https://calpaterson.com/bank-python.html
The best description about how it became a problem is one of the paragraphs.
"New starters take an exceptionally long time to get up to speed - and that's if they don't resign in fit of pique as soon as they see the special, mandatory, in-house IDE (as I nearly did). Even months in, new starters are still learning quite fundamental new things: there is a lot that is different."
I think it took me til I was there around two and a half years to fully comprehend it when I was working on it. Not much modern training til they figured out they had to teach it again that was better. The worst part is to make an UI around it coding it and it wasn't approved for new projects.
- This style of reactive programming is quite popular in JavaScript UI frameworks these days under the moniker “signals”, with a proposal for standardization here: https://github.com/tc39/proposal-signals#-javascript-signals...
It’s used by frameworks Vue, SolidJS, Svelte, Ember, Angular, and there’s a few different implementations for React like Mobx and Jotai. There’s a few different algorithms for how to propagate changes and evaluate the DAG, I believe SolidJS2 uses a height-based algorithm similar to Incremental.
I’ve been fooling around with an implementation that uses an Int32Array arena to allocate nodes and link them together with linked lists without paying O(dependency edges) GC load: https://github.com/justjake/dalien-signals/tree/dalien-signa...
There are a few of these for Rust as well, Leptos is an example in UI frameworks, and Salsa is an example in general incremental computing, used in rust-analyzer.
Another way to look at this sort of thing is as a build system with automatically tracked dependencies. One such build system is tup, which instruments build jobs to detect what files they read to establish dependency relationships. Interesting reading from the author: https://gittup.org/tup/build_system_rules_and_algorithms.pdf, see also the classic Build Systems à la Carte https://www.microsoft.com/en-us/research/wp-content/uploads/...
by jitl - It looks as if there is a significant difference in treating updates to complex objects, and probably scheduling as well.by tgv
- Another notable example: JetBrains Noria ( https://blog.jetbrains.com/fleet/2023/02/fleet-below-deck-pa... ).
As the authors highlight: "Noria is not a UI framework at its core. Instead, it’s a platform for incremental computations.". But currently they happen to use it to optimize gui rendering in JetBrains Air IDE.
by avallach - You can build some lightweight dependency graphs that flush out quickly. You don’t need to describe how the signal has changed, just that it might have changed, then flush dependencies on change (some listeners might be notified of a change they don’t care about anymore) and re-register them when they come back for fresh data again.
But incremental computation isn’t exactly functional reactive programming. They are different domains in practice that often get thrown together because the problem they address can overlap. Incremental computation explicitly derives a function that can operate on deltas, FRP might just use damage and repair instead.
by seanmcdirmid