Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Great read!by yahiamost
- How is this any different from the old OLAP cubes like IRL / Express? The cube is basically an array in memory and super fast, the dashboards based on Group By as well. I was an Express consultant at Oracle, great to see old concepts come back to life if it’s what I think it is.by wlessard_ca
- I’d loved to have seen more detail on how the Parquet file was actually built — are there any good resources that cover this?by youngtaff
- On the surface (I haven't tested it) it looks like a great cost and runtime saver, but only for data sets that need a cadence above 5 or so minutes. You wouldn't be able to have a refresh button to get the "latest" data outside this window, depending on size and build time? Wondering if you can apply a hybrid approach, combining the historical parquet file with a live query.by wyck
- Interesting how noise complaints dwarf any other type of complaint in NYC.by deanebarker
- > A dashboard like this one is designed to answer a bounded set of analytical questions ~ requests per day, requests per day for one agency, all-time totals by borough. Each question can be answered by GROUP BY queries, so we can precompute them all ahead of time and save each result as its own small table, called a grouping set. Stack all of the grouping sets in one Parquet file, one section per set, and you have a data cube. A grouping set is only useful if it either enables a question to be answered, or reduces the latency of pulling the data.
What's the benefit of "data cubes" over caching?
by cosmojg - A clever repurposing of technologies but realistically only worthwhile for static datasets with range payloads small enough to fit into a web response.
> your pipeline has to rebuild each customer’s file fast enough to meet the update cadence. ... data that updates on a coarse schedule rather than in realtime
by mrbluecoat - > The bytes pass through a small Cloudflare Worker on the way, because the free r2.dev URL is rate-limited.
For a 40MB file I suggest hosting it directly on GitHub Pages - that's effectively a free CORS-enabled CDN and supports HTTP range requests, so you should be able to get that demo working without needing to involve Cloudflare Workers at all.
by simonw