Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I have been using GPT 5.5 to build a video game. Benchmark sounds about right. It generates assets and sprite good enough, if not closer to AAA level games. Will check antigravity now.by debarshri
- Would you be able to share a bit about your workflow? Have been meaning to try AI gen for game models, and would love to know how people are tackling this.by phn
- I tried Claude code designing a snap fit, vase mode printed box. Ultimately didn't work out, it couldn't get the tolerances right and kept designing features that wouldn't print in vase mode.
Scad needs unit tests. It would be powerful to asset that a profile doesn't have slope greater than 45°, that intersection of two objects is null, or specific volume.
It also needs cut away views. I got okay results using boxes to remove everything except a sliver, to view a slice and internal details. But without hash marks, texture, or outlines it can be hard to tell the forms.
- "Vase mode snap-fit box" sounds to me like "flexible concrete".by gbgarbeb
- I'm unconvinced, this is one of the most iconic historical buildings with tomes written about it and plenty of existing photographs and public models to train on.
I would be more interested in benchmarking the modeling of an anonymous structure based on provided references alone. It kind of feels like the shallow magic of watching an LLM one-shot a to-do app..
by seemaze - The fact that the model recreated the interior dome pattern is concerning and makes me think this didn't test what they think. The interior dome pattern isn't visible in either picture. So it took the picture, and the name, then either via search or training data, knew that there should be an interior pattern. So it could be getting information on the pantheons dimensions or existing models. Whereas other models might be building based purely on what is seen in the reference pictures.by ecshafer
- From the article:
> Antigravity was the only autonomous agent that implemented the Pantheon’s signature interior ceiling pattern: repeated square coffers visible through the oculus.
The article also includes a video showing the patterns visible through the roof oculus.
by thanhhaimai - Still a long way from shorting Autodesk.
As a side note Autodesk released an agentic assistant back in December for Fusion. Six months later it is still quite bad.
by dhfbshfbu4u3 - Still a long way to go, but I'm sure it will get there eventually.by shideneyu
- Have you yet tried the Fusion MCP that was launched last month? https://aps.autodesk.com/blog/bringing-fusion-claude-creativ...by blorenz
- It is almost comically bad. I've had a few simple parts to design for 3d printing in the last weeks and tried it with them (each are about 4 operations on the timeline), and it never created close to what I was trying to do even if spelled out step by step according to Fusion naming.
At this point I'm not even sure if it can properly create a simple primitive solid.
by hobofan - The only thing faster moving that AI these days are the goalposts. Three years ago we would have been amazed if models were able to produce anything, now we have the luxury of nitpicking. Even the worst entries in the benchmark are quite impressive.by ReptileMan
- Welcome to human nature.by nutjob2
- No one asked for faster horses, they still became obsolete when cars came. Nothing newby ramon156
- Things mature, and expectations grow appropriately. That is true of more than just LLM performance.by LatencyKills
- I remember getting wound up about latency and server issues playing counter-strike in the early '00s. At the same time though, it was hard to justify being angry because playing a multiplayer game with friends who were scattered all over town was something that had to be real magic.
I guess the wow!->adjust->complain->wow!->... cycle is endless as a human
by WarmWash - Using reference images is a huge step for this sort of thing. The text-only approaches I've seen before were never going to be that good even with "perfect" AI, simply because describing 3D objects in text is not something that anyone is really any good at.by alnwlsn
- Creating a single real-world object and declaring it a benchmark? No, it doesn't work that way for a robust tool. You need to do something like Iron Chef, with a Greek architecture theme and and a panel or judge that declares the winner. This is just seeing which tool subjectively makes the best looking Pantheon.by 1970-01-01
- Yeah, this is less of a benchmark and more "I like this one guys!".
Just totally subjective grading criteria of a single poorly defined example with no end use case in mind to guide how to even do evaluation.
by Eridrus - I've had such a bad time trying to do this myself. You might get a half-way decent draft on the first try and then you start to "debug" this and after a very frustrating session you realize that the model can't properly "see" the results. That is, you just can't iterate on it, at all.
I'm guessing that most harnesses/tools will resize an image before processing and in doing so will loose enough detail to make it much harder to reason about - especially wireframe images.
I'm sure I'm holding it wrong, but this test didn't really test this. It was just a one off. That breaks down pretty quickly and especially if you don't have reference pictures of what you are trying to create.
by tjoff - I've run a tons of benchmarks for OpenSCAD for all kinds of models and setups, and what I realised is:
- Models are very jagged (might excel in one type of 3d model, but not another)
- Gemini models are the least jagged in my experience and have the best image understanding
- Gemini models are also the most creative (which may be undesirable if you want precise CAD part)
- Overall this benchmark doesn't prove much because one 3d model (and one attempt) is just not enough. I am usually testing on at least a dozen models each generated 3 times, but should really do much more, but it's too pricey for a solo dev.
Still, thanks for publishing this. Will be definitely run flash 3.5 soon to see how it performs.
by ponyous - OpenSCAD doesn't do curves. It's useless. I'm not sure why it continues to get so much attention.by willis936
- > Antigravity was the only autonomous agent that implemented the Pantheon’s signature interior ceiling pattern: repeated square coffers visible through the oculus.
That is seriously really impressive. I looked at the 3D model and didn't even thing to LOOK INSIDE the building before reading this.
Here's [1] the 3D model with `show_cutaway` enabled.
[1] https://modelrift.com/models/pantheon-benchmark-antigravity-...
by jlhawn - Was just going to say.... I looked inside by accident, and it gives a better impression of intelligence and effort than the outside.by hereme888
- I can't decide whether it's good or bad that it has included outside information clearly not present in the prompts to make the model. Clearly its the right thing to do if you want "the Pantheon" but I don't think any draftsman or engineer would find this acceptable work.
- Antigravity may well Top the whatever benchmark but:
My Antigravity (forced) replacement for Gemini CLI requires me to log on via browser every time I use it, and my Antigravity IDE won't update at all, so:
If it's ok I'd prefer they just work on reaching a baseline acceptable rollout before worrying about being Top in anything.
Ps actual title:
OpenSCAD LLM Benchmark: Building the Pantheon
by mellosouls - "Pantheon" bloody hell, why is it people writing these articles are so up themselves, it's so overbearing.by stuaxo
- Wild that it doesn't cache the creds.
- I've got an AI pro plan and haven't been able to log in for months. Endless checking in with my google support guy. At least Dinesh wishes me good health every week, so that's nice.by arthurtully
- Having my workflow disrupted is the main reason I never adopted Antigravity, despite liking it. I'm glad to see G is invested, but the older I get the more protective I am of my workflow.by freedomben