Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I love this. I'm trying not to go down yet another rabbit hole, but I really want to be able to tell claude code to make a visualization 'like this but just the way you know I like it'.
I'm seeing more attempts at standardizing how AI displays data via presets. It's practical, I guess, and hopefully it will make things less error-prone.
by aitor1717 - We struggled the same, especially when deploying an agent to end users, they are even more frustrated with any mistakes from agents than us!by chenglong-hn
- This is a valuable method of closing the gap in making LLM results available in a good visual form. The idea of viewing the charting process as the last step—where the AI deals only with high-level semantic specifications as opposed to all the low-level visual details—makes so much sense. I'm interested to know what the layout optimization engine does: can the developers inject the desired limitations for some cases, or is the layout design process a black box? It's good to see you've made it open-source!by nourihab
- It's a white box! We built a demo here: https://microsoft.github.io/flint-chart/#/documentation/char...
We use a fun elastic algorithm to decide dimension etc within the developer's constraint.
by chenglong-hn - https://github.com/scicloj/kindly
but enterprise
by ietcd - I don't really understand the point of this, I feel like LLMs have been able to one-shot matplotlib since GPT 3.5. I have extensively used LLMs to do data viz and haven't run into any problems. What is a specific instance where an agent struggles to generate a visualization and Flint solves it?by jrflo
- This comes with a bit of last mile issue, if just chatting with GPT in the chat panel and keep steering it, it's generally fine (as a power user).
But when building it in a tool that serve end users, we are starting to see that a 80% success rate in generating good looking charts can become a big issue. We experienced this when building some data analysis system. So the reliability, expressiveness, and costs (in terms of time and tokens) are hard to achieve all together with directly generating matplotlib, vega-lite etc.
So we essentially designed the langauge as a trade-off across the three, by moving some decisions to the compiler to reduce generation cost while maintain good expressivenss.
by chenglong-hn - Project page: https://microsoft.github.io/flint-chart/by chenglong-hn
- >Instead of requiring verbose low-level parameters such as scales, axes, spacing, and layout.
Ok, Microsoft is conflating two different things here: LLMs don't really care about code being low level and verbose, they can read things like Assembly and SPIR-V just fine: visualization is the real issue in that LLMs have no natural understanding of spatial composition through visual comparison because they literally "see" things differently than humans, so the way to get around that is provide them with "visualization" in code form that they can easily reason about and understand, so basically anything that's not deeply nested and has hidden states that they have to reason about.
Also, Flint being stringly typed in JSON is a decision that I don't think I agree with. Looking at the actual spec, this could have just been a normal, human usable TypeScript library, and it would have been 100x better. Using their own example (excuse the formatting):
type SemanticType = "Category" | "YearMonth" | "Profit";
type ChartType = "Heatmap" | "BarChart" | "LineChart" | "ScatterPlot"; // extend as needed
interface ChartEncodings { x: string; y: string; color?: string; size?: string; tooltip?: string; }
interface ChartProperties { colorScheme: string; [key: string]: unknown; // allow other optional properties }
interface ChartSpec { chartType: ChartType; encodings: ChartEncodings; chartProperties: ChartProperties; }
type SemanticTypes = Record<string, SemanticType>;
interface ChartConfig<TData = Record<string, unknown>> { data: TData; semantic_types: SemanticTypes; chart_spec: ChartSpec; }
// The actual typed object literal: const chartConfig: ChartConfig = { data: {}, // replace with your actual data shape/type semantic_types: { game: "Category", period: "YearMonth", newUsers: "Profit", }, chart_spec: { chartType: "Heatmap", encodings: { x: "period", y: "game", color: "newUsers", }, chartProperties: { colorScheme: "redblue", }, }, };
EDIT:
Went and actually looked at the source instead of just eyeballing it from the docs, and it was a lot more complete and sophisticated than my assumed mockup already.
Core complaint (string-keyed JSON vs. a real generic authoring surface) still stands, but the specific types I posted aren't what Flint has. My bad.
by YuechenLi - sadly, I think we are stuck with JSON as the most reliable way to get data / code in and out of an LLM (could be worse, could be YAML) … I’m interested in custom DSLs that improve LLM predictability and it is quite nice to see that even the Microsoft dinosaur “gets it” … see the Contacts example at https://slangify.org/examples which does VCARD to JCARD round tripping as a way to easily roll your own DSLby librasteve
- Agreed, JSON is not the most optimal spec language choice, but is it better if different projects all started to writing their own specs?
Related: https://www.openui.com/blog/stop-making-ai-write-json
by bodash - I do find the chartType part is not quite elegant, since templates should be more extensible. We will need to fix that.
For other parts, it's quite common in visualization and diagram etc libraries to have json, since they are easily portable in different rendering contexts.
by chenglong-hn - > simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability
N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?
by nrub - we are considering also reliability, interactivity besides expressiveness. Simpler spec with good expressiveness comes handy when you want the agent to be reliably for non-expert users and with small models.by chenglong-hn
- A simpler spec can be used by a simpler agent. So, maybe that's the use-case here... use by smaller/cheaper agents that run in parallel as opposed to large models running one visualization at a time.
Or at least, maybe that's the idea?
IME, Claude and ChatGPT do just fine generating ggplot models, but extensive customization can get a bit hairy.
by mbreese - > requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler
Isnt graphviz there for the same reason?
Edit: I see it is using JSON as the declaration language, I am OK with llms being "good at json" but a syntax also consumable by humans it is not!
by theK - Absolutely. This is DOA honestly and not really better than what we had before.by simlevesque
- In fact, Json as a common language for human in visualization has been around for a while! The benefit of declarative grammar is that users can effective manipulate specs through UI (drag and drop, clicks).
Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.
by chenglong-hn - Since it isn’t mentioned on the page, I’ll mention that accessibility is a really important thing to design in when creating data visualizations.
This podcast (I only have a spotify link) has a really good short interview on this:
https://open.spotify.com/episode/18dHTAxCCeIaLOTch6tRld
The interview is with Frank Elavsky who seems to be a rock star in the field (and no, I don’t know him and am not him) and made a project called Chartability relating to this which has heuristics, principles, and guidelines for a11y audits:
by natch - Thanks a lot for the suggestion, I do think we will need to work on accessibility support in Flint, this is a perfect place to centrally handle accessibility issues.
I added an issue to track this
by chenglong-hn - Is there a specific explanation about how this is better or different than vega itself? https://vega.github.io/vega/docs/specification/
My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data.
by kveykva - I was wondering the same for vega-lite, which is relatively high level, declarative, and looks similar to their syntax.by pea
- Vega was a high-level language in the past for human, but now they can be a bit too low-level for AI agents! AI agents have to write a lot of low-level params just to make charts looking good, and the result is that programs are hard to write reliably for AI agents.
Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.
So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.
by chenglong-hn - There’s an emerging pattern in agentic systems and this project is a great example.
A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.
I feel we will be seeing this more and more in the near future.
by cpard - Programming is more alive than everby alightsoul
- Yes, this has been the pattern for agentic systems since the beginning: permissive generation, that retries over and over until it gets the right size shape through the hole, and the input validates.by alansaber
- I'm all in on this idea. Every piece of agentic coding I have done in the last month has been via an intermediate representation. Iteration is done in the IR layer mainly. It's remarkable how close you can get to a deterministic coding output using this methodology.by rightlane
- A well designed intermediary enables both validation and control over the output independent of the AI. This changes the interaction model between human and AI from delegation to collaboration.by ajrouvoet
- When I first saw Claude generating PPT decks by writing Python code instead of making the XML directly, it was sort of an "aha moment" for me. This seems to be the path for many things. It also feels slightly limiting, and like a hack LONG term, but 100% correct approach for a while.by pwarner
- "For AI agents". I understand why everything needs to be marketed in this way, but it's just ... an easy-to-generate language for expressing charts. That's impressive! That's useful.by rbalicki