Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- > All that’s left is the dumbest workflow possible: I fire up the debug viewer myself, look around for weird mistakes, then take a screenshot and tell the language model how badly it messed up this time. Eventually I just decided to do all the debugging work myself, so I would at least get to do the fun part too.
I call this "thanoscoding" for two reasons:
1. "Fine, I'll do it myself"
2. In the past I found I have to "snap away" the mess the LLM made in order to start afresh from a known good state (generally with git reset). But that was 1-2 generations ago when it comes to models. GPT6 Astra probably does things right the first time, 90% of the time.
by bitwize - Those three colored shapes at the bottom look a lot like the EPA logo, a former grocery store chain: https://de.wikipedia.org/wiki/EPA_%28Warenhaus%29?wprov=sfla...by creichenbach
- LM still can not see and understand the desktop app UI on a level that is acceptable for testing. All the advances in coding are from web dev and thanks to the nature of html UIs. Try to develop a desktop app and it’s like working with a legally blind person who can see some part of the screen is they squint in a certain way but surely will miss all minor details.by dostick
- One of the first things I tell the junior/mid-level developers I mentor is "You can't debug something just by reading the code." We all have a mental model of how our code works, and it's usually a bit wrong. Bugs are the real world manifestations of those mistakes. When you read the code it's all filtered through your model, and that makes you blind to seeing why something unexpected happened. In order to debug something you have to be able to put the system in the state where the bug happens to see why it occurred.
LLMs generally only debug systems by reading the code with whatever information you give them in a prompt. The image in the article is meta-prompt - the prompt is whatever comes from the vision model the AI happens to use to 'understand' the red circle annotation. That won't work. To successfully debug what's going on it will need much better state information. Has the 'shelf' been explained to is? Is the contrast and lack of shadows in the image messing up the vision model? Why isn't the 'lid' in the image? And so on.
LLMs are clever but they're not magical. Treat them like a naive junior dev. Give them enough data about the state of something to understand it properly.
by onion2k - I've found that LLMs are specifically bad at a certain kind of debugging, though I can't quite put my finger on what that is.
"Spot the bug in this code" when the code can be looked at and pattern-matched against bugginess is something they seem really good at.
Some parts of debugging, like "Here is this logfile, what do you think is going on?" are also surprisingly good.
It's that thing kind of in the middle -- I know it when I see it honestly is the best way I can put it into words. An example from recently, I'm receiving some bad data on a network message parser. Immediately I don't know whether it's a my-side or their-side thing, but I know if I try and just vaguely describe the behaviour to the LLM it will start churning tokens.
My current approach to problems like this is -- I need to tell the LLM what it needs to do to give itself the data it needs to solve the problem. My first reaction now isn't "It's not working, there's a bug, it's not doing X". It's "Okay, this isn't quite working properly; I need you to add some debug logging around X, Y and Z so we can figure this out". That tends to avoid spirals and get me out of the situation much more quickly.
The seeing eye dog analogy is pretty apt actually. I would love to see some transcripts from the author if they are able.
Edit to add: I think the 'thing' I'm alluding to might be -- if I have trouble expressing the buggy behaviour clearly in words, then I know it's probably going to be a fair few back-and-forths with the LLM to get something; the harder I find it to concisely describe, the more risk that it'll fall into a pit. Doubly so if I offer up a hypothesis which turns out to be wrong.
by Fr0styMatt88 - A bit off topic, but I absolutely love the little robot on the author's main project's landing page (https://rerun.io/). I normally condemn mouse hijacking, but this implementation will be allowed.by beklein
- > I used to argue with people on the internet, after about six replies, you realize that you’re speaking to someone incapable of thought
He realises the current woe of things but doesn't realise he's been arguing with bot farms and teams of people hired just for this reason - to sow doom, arguments and engagement.
Around 10 years ago I noticed this happening on trending topics of Twitter - it wasn't that the opponents were stupid because they disagreed, it was that they were simultaneously intelligent and stupid in the way they spoke in a way that I realised I'd never seen in genuine people, making me realise it was probably different people or bots under one account.
If I realised it 10 years ago it was probably happening for at least 15. He wasn't better than these people he was falling into their trap
by dwedge - I completely understand this. I’ve worked on robot hands and 5.6/Fable 5 were practically useless at helping me debug anything visually.
What I have found super useful actually is having models make a interactive 3d viewer in which I use move / highlight / paint (soft body painting directly onto the geometry for issues and different colors mean different failures). This gives a much better way to communicate the physical relationships and positions that are hard to get across in a labeled screenshot.
Its for sure still a lot of manual work so the "seeing-eye dog" description definitely holds. But I have found that after a couple of examples with the extra context the model gets much better at handling the problem and becomes useful.
by hspeiser