AI Coding Without the Vibes

Join the discussion

• Hacker News
• This article hits very close to what I have come up with myself and its good to see others thinking the same lines.

    Core modules: Coded by myself, AI reviews and AI to discover/learn.
    Stuff I don't care about Craft: API layer, CLI layer, Smoke tests, Integ tests - Dial AI heavy, and lighter human reviews accordingly
  

  Obviously takes a lot of patience and very easy to sin, but on good days, its doable.
  by conqrr
• Tests not being core seems suspect
  by Kinrany
• Never does the author say what language he based his essay on. If it was C the conclusions might be skewed.

  Also, with AI the coding process converges on developing a good spec and discussing the architecture tradeoffs first. It often doesn't see architectural opportunities to get the best performance, so that's still your own job.

  by childintime
• I think one of the best use cases of AI is as a natural language interface to programming. The syntax of a programming language is an opinionated part of its design that often adds to the complexity of learning the language.
  by docheinestages
• You can let it help with syntax, but you can't rely on it to understand precise semantics.

  SQL is the worst offender here: LLMs are very good at remembering its awful syntax, but very bad at writing nontrivial queries.

  by Kinrany
• You can't 100% spec something out in natural language without ending up with the actual code. The code is the natural language spec expressed with all the ambiguities removed.
  by _superposition_
• Everything old is new again. The reason natural language never caught on for programming, is that natural language is inherently unsuitable for the task of formulating precise and unambiguous instructions for a machine.

  Numerous attempts have been made in past 70 years, but we landed on the compromise that is high level programming languages with their (mostly) well-defined syntax and semantic rules.

  If you look outside of the world of programming you'll find that the same is true for every craft and profession that requires precision and unambiguity. Cooking, logistics, military, carpentry, mechanical engineering, law, medicine, etc. all have their own domain specific languages for describing tasks, procedures, and tools. Heck, even sufficiently large organisations have their own internal languages and terms for that reason.

  by qayxc
• I like the idea of treating the agent as an expert reviewer, and I think that having it be "read-only" is a good approach. I use it at my job this way due to professional constraints, and I'm using it this way in my own time to learn a new language.

  However, I think the points about not allowing the AI to run anything take it too far. A good code reviewer, in my mind, will check out your PR branch, run your tests, and ideally try to break them. An agent can run your code in its own sandbox, try variations to see what breaks, etc.

  by blairfrandeen
• Im currently doing something like this. The agent wakes up automatically at the end of the day and runs it's interactive checks. Using https://shellbox.dev as substrate so it is independent of my own schedule.
  by messh
• >It's a verbose coder, it overcomplicates things, and it works fast. Even if you had an inhuman level of attention, you couldn't keep up.

  I wonder if that trick about prompting it to write like a 5th grader* would help here. Keep it simple!

  *A trick which allows it to pass for human 70% of the time...

  by andai
• Tangential to this post but Peter Bloem’s Transformers from Scratch (2019) was one of the best early tutorials on transformers and self-attention. There have been many more tutorials since then but this was the one that solidified a mental picture for me.

  https://peterbloem.nl/blog/transformers

  by d4rkp4ttern
• Vivian, the commercial baker, learned a different set of skills. She might have deskilled with regards to hand-rolling dough but she gained new skills related to industrial production.

  I've made a half-dozen or so DSLs with LLM tools over the last couple of years. I've learned a lot about the architecture of parsers, semantic analysis, editor services, LSPs, DAP, etc. I have a new set of skills! I'm able to focus more on the syntax and semantics languages themselves and the overall developer experience. This would have taken literal man-years of development to do otherwise.

  For my hand-rolled craft I'd much rather sit down at the piano or pick up my guitar!

  by williamcotton
• I went from just coding manually, to using LLMs for "surgical edits", to "woah, AGI!", to "haha whoops, not even close", to just coding manually (my brain still works!), to "surgical edits" again.

  My current approach is "ask for very small diffs" + "review them very carefully".

  I'm not working a job though, I'm working on a multiplayer game.

  Main findings: The frontier models can't reliably modify Pong without breaking it, so their skill appears to be quite domain-specific. (OK, to be fair, neither can I half the time!) This is probably because they are "time blind". I had one model try to test a game by running it at 0.1 frames per second and shoving each frame in the vision API...

  If you leave any room for a misunderstanding, they will laser in on do it and do the stupidest thing possible. If you're not checking everything carefully, you will discover this later, and you will cry.

  Formal proofs, oddly enough, do not improve the situation: they will simply prove mathematically that the absurd and pointless and backwards implementation is completely without defects. (It obviously does help within an implementation, though.)

  They can't formally prove what the hell you meant when you told them to build something. That job remains frustratingly human!

  Current dissatisfaction: (1) Harnesses are designed for super bloated codebases (i.e. designed to load as little context as possible) which make them pretty clunky for small repos and small edits. (I had a Surgical Edit Tool I need to bring back...), (2) Current LLMs are anal about verifying the most trivial change, even without prompting, even if it's impossible for them to verify it because they're blind so they start measuring pixel data in Python... Both of which eat up Speed and Cost, taking the work even further from Realtime/Interactive to Tedious/Sad.

  by andai
• Insightful:

  > the term “vibe-coding” suggests a kind of laissez-faire attitude where you don’t really care about the outcome and you’re just having fun. That’s what the phrase meant when it was coined, but the world has moved on. In many companies professional programmers are using AI in such a way that it’s impossible to imagine that they are also reading the resulting code in detail. This is what modern vibe-coding is. Deferring to the AI, not worrying about the individual lines of code, and keeping an eye on whether the code passes its tests and throws up any problems in production.

  This way of working is the only one that justifies the trillion-dollar bet on the AI industry. I agree, this method should be called vibe-coding.

  by ripe
• It should depend on whether you've taken other steps to constrain the output of the LLM. I've been building some pretty elaborate test harnesses (which I test independently beforehand), to ensure that I get something that does what I ask of it.

  I don't care which words we use, but they should separate this sort of empirically-minded work (versus the rationalism of typical development) from just dropping a few sentences into the prompt and crossing your fingers.

  by __MatrixMan__
• I had the same thought and came to the exact same name: vibe crafting. Published a skill in the process, steps: https://github.com/scosman/vibe-crafting
  by scosman
• “But that would be like telling a student in the 70s to pretend that calculators or computers don’t exist.”

  If students of the 70s or today pretended they didn’t exist up to a certain point when they needed them to move forward, like bioinformatics or something, they 100% would be better off. There is plenty of research on off loading thinning providing a worse understanding of the material - eg side rules proving a better understanding than calculators.

  by conception
• I liken this to how my own final exams in school were. We had a "cheatsheet" on the front of every exam [1] with all the formulae that might (or might not) be relevant, even simpler stuff like pythagoras' formula was on there that everyone ought to know by that point. We also had access to calculators.

  This is, of course, after years of learning without cheatsheets and calculators, where you learned to intuit if a number actually made sense when spat out by the calculator, and when you learned when and how you're supposed to use a certain formula. For example when doing any kind of Physics or Chemistry you're working with numbers that don't make intuitive sense (and often are quite literally imaginary!), like things that are to the ^32 or ^-28, but you still learn to intuit whether the relative scales of the numbers make sense with what you'd expect to see given similar prior experience.

  As for the formulae, even with the cheatsheets if you didn't know why or how some of the formulae were supposed to be used, well good luck to you because just knowing the formula isn't even half the picture. I just dug up an example formula reference sheet and despite definitely knowing a lot of the stuff there, I haven't practiced a lot of the maths involved here in years so I wouldn't be able to get anywhere close to answering some of these questions despite my access to the formulae.

  AI takes all of this to an extreme. People simply plug in their question into their chat interface of choice and don't even bother reading the output even slightly. It'd be like seeing that the mass of a planet is x^-27 and not realizing you must've input something wrong because you simply trust the calculator without any understanding of what numbers you're plugging in.

  [1] - As an example of what I had - https://pmt.physicsandmathstutor.com/download/Maths/A-level/...

  by sensanaty
• I don't know if this was universal, but when I was a student we weren't allowed calculators until I believe 10th grade

  You learn stuff by heart, so that when you mess up with the calculator you have a feel that stuff has gone wrong

  by joseda-hg
• If the challenge is held constant, giving people tools is just teaching a different skill set than teaching them unaugmented. The augmented skill set is more relevant to how people do work in the real world, but working through things unaugmented likely help people build a robust internal schema more quickly. We definitely don't need to spend as much time training unaugmented abilities going forward though, kind of like how the number of assembly related classes in CS departments has gone down over time as we've moved up the stack.
  by CuriouslyC
• “You can use the electronic tuner once you learn to tune by ear” - any decent guitar teacher
  by b38tn1k
• Exactly. I always think about the ways consumers make terrible decisions with everyday purchases at the grocery store etc. If people could easily estimate unit costs in their head for example, a lot of predatory options would have to go away. But calculators keep people from having to get good with everyday math in their head, while those same calculators sit unused in many everyday scenarios where people don't go to the effort to use them.
  by gdulli
• This misses the best use of AI in my opinion, which is to gain understanding. Whst is this bit of code doing? Is there a risk of data leaking here? Are permissions enforced downstream of this function?

  Code review can go much deeper now if you use AI to aggressively attack a PR combined with your human insight. Same for planning a feature:

  Can I consolidate this logic to a shared function? Does the error surface to the user and are there any gaps? What preexisting functionality is affected by this PR? Can this query be made more efficient?

  LLMs are great with focused questions, up and down abstraction layers and across all kinds of concerns. Stack up these focused concerns into a rich understanding of what you are doing or writing.

  Understanding is the real output, code is the byproduct.

  by lubujackson
• Closely related: one of the biggest AI superpowers I've found is testing. "Please implement a comprehensive unit testing suite for this, making sure to test ..."

  You still have to check the work and shouldn't trust it blindly, but in my experience it's pretty damn good.

  by api
• Doing this may give you understanding but won’t really teach you how to understand. For students (as the article is focusing on them), the latter is equally important.
  by wonnage
• Do you need that analysis every time from an LLM? Those are things I’d expect anyone to grow out of pretty quickly once you get used to understanding code. Especially someone with Senior in their title.
  by righthand
• Wish I could upvote twice or something
  by bkircher
• Agreed — I also always append to my agents.md — "If there are any technical choices available, please surface them to me so I can understand them and choose"
  by fkozlowski
• > LLMs are great with focused questions, up and down abstraction layers and across all kinds of concerns. Stack up these focused concerns into a rich understanding of what you are doing or writing.

  I agree. This is much better advice than what the article tries to do, which is to give best practices around how to use or not use AI to code.

  by ripe
• Agree with the spirit of this, but it assumes a human is still looking at the code, and human inspection of code is vanishing faster than people are willing to admit (especially when coding in a language said human doesn’t know).
  by d4rkp4ttern