GPT 5.6 Sol is the best "vision" model OpenAI ever released

Join the discussion

• Hacker News
• From our experiments it’s the best video captioning model in the world by a mile. This was not the case a year ago.

  When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.

  The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).

  Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.

  by schopra909
• Have you checked versus more recent Gemini models like 3.5 or perhaps 3.7?
  by Vacyyyy
• Vision is still embarrassingly bad.

  ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b...

  The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...

  by dllu
• The second answer is far more revealing than the first:

  OP:

  > do you think you did a good job there

  ChatGPT:

  > I spent 15 minutes, emitted several fake-sounding “tracing the puzzle” progress updates, and then gave a confident permutation without showing that I had actually followed the lines correctly. It reads much more like I guessed than solved it. The only part I did well was obeying the “no Python or tools” instruction.

  My observations:

  1) Sarcastic tone suggests pre-prompting, or frequent (and therefore stored in memories) denigration of the model in past conversations. I'm leaning the former - it sounds like it was instructed to read admission of defeat.

  2) The part about "no Python or tools" is setting the model up for failure.

  I mean, this task is, for a human, basically a game of "simulate a line following robot in your head". Pretty sure a VLM could solve that if it was allowed to do the same thing. Off the top of my head, an algorithm like:

  1. Identify start and end points

  2. Foreach start point, follow next pixel minimizing angle, until endpoint is reached.

  3. Report answer

  It's literally what every human facing this task does.

  EDIT:

  My attempt - same image, prompt altered to allow for code (but still no search/external checks), solved in 1/5th of the time, correctly, and (going by thinking trace summaries that I don't think show up in shared chats), basically the same way I'd approach it, by tracing the lines, coloring them as it goes.

  https://chatgpt.com/share/6a834f76-8240-83ed-acff-0c67af399d...

  INB4: I know this is now not a pure vision check, but it really doesn't make much sense to diss models for failing to solve tasks explicitly designed to teach humans to externalize computation that's hard to do in their heads (i.e. kids, crayons, coloring paths).

  Still, if such things are becoming a benchmark for tool-less evaluation, it's only a matter of time until the models learn - much like humans learn in school - to follow algorithms mentally, essentially emulating an ad-hoc computer in their head.

  by TeMPOraL
• In the third vision bench result, Sol is 100% correct but the expected has 1 error. Seems like an oversight.

  In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.

  by kzrdude
• Hi! I’m the author of this blog and benchmark. You’re right. I’ll fix it in the ground-truth dataset. Thanks for pointing it out.
  by SkalskiP
• Seems to be due to the detection area being not fully accurate. Green vs red shows the difference between actual and detected
  by defrim
• Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.
  by fpgaminer
• but 3.7 flash is expensive for img inputs no ?
  by zuzululu
• 3 Flash never left "preview" status and is listed as deprecated.

  https://ai.google.dev/gemini-api/docs/deprecations

  by bastawhiz
• Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.
  by mv4
• I'm sure a typical frontier model would also be happy to write that opencv script for you, and it would do it well.

  That is certainly pretty far from what was possible 25 years ago.

  by geysersam
• Basic Template matching has severe limitations around scaling, rotation, and perspective. In my experience it greatly underperforms compared to deep network object detectors. My experience- and I imagine others have different experiences- is that SIFT techniques also fail pretty badly with noisy data.
  by dekhn
• The point is that it's general. It can do this task and many other tasks and it doesn't need custom development like OpenCV does. Of course if you only want to count pills and you want it to be cheap/fast you're still better off using OpenCV.
  by lebek
• I'm assuming you mean that this tech became available in OpenCV 25 years ago, but as it turns out, the underlying tech can be traced back much further, at least as far as 1977! :)

  https://ieeexplore.ieee.org/document/1674847 G. J. Vanderbrug and A. Rosenfeld, “Two-Stage Template Matching,” IEEE Transactions on Computers, Vol. C-26, No. 4, pp. 384–393, April 1977. DOI: 10.1109/TC.1977.1674847

  by maxime_cb
• A better headline would be "Gemini 3.5 Flash is the best vision model". It tops almost every single benchmark shown in the article.
  by mherrmann
• It is funny to me seeing Sol used for what a "traditional" AI model can do already (counting pills).

  We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.

  by bearjaws
• How are we supposed to pay off all these data centers and chips if you’re not willing to burn a microwave burrito worth of electricity for each prescription? Think of the benchmarks
  by repeekad
• Agreed, this like asking a chainsaw to carve a wooden spoon. Impressive it can, but definitely not the right tech to scale.

  LLM needs to setup an image classifier to use as a tool call.

  by kooi
• Hi! I’m the author of this blog.

  I’m evaluating these VLMs to figure out which ones are good enough to auto-annotate my data, so I can fine-tune my detector.

  I wrote a bit more about this here: https://x.com/skalskip92/status/2080334344061694429?s=20

  by SkalskiP
• Dumb question: When your testing "ChatGPT 5.6 Sol" are you testing an actual LLM or some visual pre-processor stack that sits in front of it (along with a maybe a bunch of other such pre-processors) that is bundled into what's call "ChatGPT 5.6 Sol"? I.e. last I checked LLMs had a something like a 30-100K token alphabet to work with and it's hard to imagine how throw pixels arrays at one directly would work.
  by theteapot
• I'm ~95% certain that images are tokenized, just like regular text, and fed directly in; that's the 'multimodal' part of these models. Now how this tokenizing works I don't know, and there might be some level of preprocessing, but it's certainly not converting the image into text and feeding it in to a regular LLM.
  by MoltenMan
• Penny sample shown looks like failed EXIF orientation registered by the model/harness. The coins are correctly marked, it's rotated 90 degrees.
  by evrimoztamur
• Good call out, I noticed the same rotation issue but pointing at EXIF data sounds about right.
  by DustinBrett
• Hi! I’m the author of this blog. I had the same intuition, but together with the OpenAI team we figured out that the issue was image resolution. GPT-5.6 doesn’t handle large images well.
  by SkalskiP
• I wonder what about UI review, which model is the best?
  by dostick
• Anecdotal, opinion:

  Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.

  by weli
• What is a "non-normative UI block"?
  by DaiPlusPlus
• Hi! I’m the author of this blog. GPT-5.6 is much better at vision than previous GPT versions, but it’s still much weaker than Gemini 3.5 Flash or Gemini 3.7 Flash, which was released last week. One interesting approach is to use Gemini through a tool call.
  by SkalskiP
• Assessing the subjective quality of a thing is in my experience one of the worst ways to use any LLM.
  by velcrovan