Is anyone experimenting with different ways of using LLMs for coding?

Join the discussion

• Hacker News
• I'm working an an agentic graph-based workflow execution engine/framework. The concept of the harness is completely abstracted away/generified - a 'node/agent's is a harness (cc, codex, open code, pi, etc) + model (I test different model and harness combinations). I have a set of tasks from trivial to complex - a set workflows (a workflow is a set of initial nodes and their behaviour) is defined and each one is asked to perform each task (multiplied by each harness/model combination roughly). The workflow can include agents/nodes which are able to modify the workflow graph and create nodes. Other nodes can break down tasks and send subtasks to other nodes. Mostly experimental stage at this point. I'm exploring/tracking metrics such as total wall clock time to complete a task, total cost in tokens and $, among others. This gives me a decent amount of data/insight into the abilities/performance of different harness/agents/models for different tasks, and gives me a great testing/dogfooding of my own harness (which is one of the harnesses being tested, and as of now the most efficient one).

  The main bottleneck at this point is the cost of all of the tokens in the fairly large test matrix of tasks, harnesses, models.

  I hope to release/open source all of this stuff eventually.

  by aleqs
• > The main bottleneck at this point is the cost of all of the token

  Are you using Chinese models? Quite a bit cheaper, but maybe still too expensive?

  by worik
• I rolled my own simple execution DAG program.

  It’s shockingly effective due to rooting sub-DAGs into Planner nodes which are the only mutators of the DAG. The deepest topological leaf nodes become the blockers to the next Planner node.

  The only other special node is a Human node; structurally impossible for agents to close (I rolled my own harness) and block on my attention.

  by fractorial
• My company tried to build something like this pre-TUI as a tool-AI-IO dag dispatcher. The biggest mistake I made was thinking that people would have no problem figuring out how they could translate their work or define multi-step automations, and focusing on the orchestration and sandboxing thinking that was the core, when it was really figuring out how to get the onboarding UX/complexity to not feel daunting or more trouble than it was worth.

  Eventually for my own work, I discovered that the context management and runtime was more like a stream or active service mesh than a dispatching / one-off processing problem, most others' were too. Then all my prompts would degrade across model versions or providers, and I realized that actually setting the context for the tasks and keeping track of it all was a ton of work and something I had to do everytime as an actual user, but never when I was testing or demoing it on existing data.

  Curious how you're testing your work and if you've managed to avoid the problems I ran into. I need to permute across the same set of workloads/configs you mention (and maybe more) for my next set of work so I'd be very interested in sharing or collaborating on the test infrastructure! At Google I did a lot of permutation testing using https://github.com/cloudprober/cloudprober and was going to start using it sometime in the next couple weeks. It exists basically one layer above the workload content/targets so it's probably compatible with everything except the test client/driver you're using.

  by weitendorf
• YES!

  It's still very wip, I spent a couple of weekends on it so far, but I'm working on a harness that eschews autonomy and instead aims to work as a pair programming partner. Key to that are distinct "driver" and "navigator" modes, with the capacity to flip between them rapidly.

  https://gitlab.com/philbooth/opair

  (not really usable yet, but after tomorrow's session I expect to be developing opair in opair, which is mildly exciting)

  by philbo
• Love this idea and will be following closely! I've wanted a pair programmer style interaction for a while now. Something closer to VSCode's Copilot inline conversations and FIM, but where it's continuously watching what I'm doing and ruminating on suggestions.
  by jesse_ash
• I'm building "workboxes" to work on my startup. It helps me develop features insanely fast. A workbox is a simple worktree-in-a-sandbox per feature. I have a simple front end where I can launch new workboxes: I input a prompt (a documented grilling session) and it creates a branch, a PR, and starts an opencode coding session on an e2b sandbox based on a custom template with the app's monorepo. Each workbox has a public https endpoint so I can manually test the web app after the coding session is complete. At any point I can either approve the PR, send a follow-up prompt, or connect to the opencode session for more control.

  I think my next step is to perform the grilling session inside the front end, currently I perform it in my terminal and then paste in the front end.

  by just-tom
• Is it similar to how Claude Code Web works? It generates a cloud container and clones your repo, and works on a whatever you want (preferebly something specific), and then it generates a branch and a PR.
  by LikelyLiar
• My absolute favorite modality is one I don't use all that much at the moment: Zed's edit completion.

  If you're unfamiliar, it's like tab-completion, but it has a context that includes the edits you've made in the last few seconds, and it can predict around the cursor.

  The model isn't advanced enough to understand complex tasks, but it has more the feel of the "crafting gun" in Subnautica or other survival crafting games, if that analogy makes any sense.

  Personally I hate working with a chatbot - it's low-bandwidth and rage-inducing. If I could imagine a perfect workflow, it would be something like me whispering my train of thought as I program, and then pointing a very fancy "autocomplete gun" at the code.

  by colinmarc
• That's what VSCode next edit does too. It gathers context from the surroundings and recent actions to suggest blocks of code.

  It often feels magical. But I find agentic plan->review->implement->review workflow to be a net positive in cohesion, documentation and throughput, relatively speaking.

  by bel8
• I'm in the same boat and I'm not a fan on the current way of working of agents, but I think tooling is what needs to catch up.

  So, I actually decided to try to tackle it myself and worked some months (full time) on it.

  https://beolis.com is the result of that, it's a local cli in a kanban board style with a remote server to keep the team on track (I've been using it myself for some time and actually started to ask some friends to use it just yesterday -- feedback very welcome, I still wanted to do some additional things before asking more people to use it, but oh well, I'm a fan of building in public anyways and it's probably better to have feedback sooner rather than later).

  The main point there is that you work mostly in the ticket description (your own spec) and the plan (the spec as the agent sees it, generated with a custom workflow) and then having another custom workflow to implement it (you can choose how you want it -- https://beolis.com/blog/post/custom-coding-workflows has some info on what I'm using myself).

  As a result, at least for me, I do spend more time immersed in a flow state (although I'm in that state writing the specs and reviewing code -- although in some cases it's more work to write the spec in a way the agent can work when things get more complicated vs just diving into the code, so, going into "code" mode is something I still have to do, agents are definitely not perfect).

  I guess I'm lacking in docs on how to effectively use it. I have plans to create a video next week and post it in the blog, so, if you're interested, keep track of it ;)

  by fabioz
• Yeah, flow state is not the same, and I miss it.

  I've stumbled into a couple of different ways to work with AI, each with their advantages and disadvantages:

  * Ask it to solve the issue and trust the results. You're outsourcing your thinking, and lose understanding of the code. The result might work, or it might not. Chances are it will work, but your code slowly grows messier.

  * Ask it to solve the issue and review the results. This should help you understand the resulting code, and give you a better chance of setting the AI straight when it messes up. But you're still outsourcing your thinking, and not thinking of the solution yourself still means you lose touch with the code. But more importantly, reviewing is the boring part of software development.

  * You write the code, and let the AI review it. In a way, I think this should be the sweet spot. It doesn't make you faster, but your quality should go up. AI is very good at reviewing, and often finds issues that humans skip over. This is the quality over quantity solution. More than code, I think this is particularly important for writing high-stakes non-code documents, like financial reports for customers. Quality is really important there.

  * You tell the AI how to solve the issue. The AI still writes the code, but it receives tighter guidance from you. This is what I usually end up doing. I like to think this results is better code. It certainly gives me the impression I understand better what the code does. And I do tackle much larger amounts of code, but I check what the AI does, and often push back on its suggestions and assumptions. I think this is a nice middle ground between speed and quality.

  * Full agent mode. Let the AI do everything. Let multiple agents work simultaneously doing everything. You lose control and your mental model. You're going to have to trust whatever the AI is doing. Something it will be correct, sometimes not. Let's hope you never have to personally touch that code anymore. But it sure is fast.

  by mcv
• I had this exact problem, tried opening multiple agents in different terminals but that just frays your flow state even more. There is one great workaround I’ve found.

  Walk coding. Walkoding, if you like.

  Use a harness, create a harness if you like, then load it up in telegram and off you go. I’ve been on solo hiking trips and shipped numerous features. It means you can stay concentrated on your task, while not sitting there being bored.

  It’s truly liberating, highly recommend.

  by anthonyfrisby
• I've also tried this. Very interesting, but must be used sparingly in my experience. Hiking is a place where before I used to go to think, or not think at all. Prompting an AI while doing so can feel very bad if really what I'm looking for is some disconnection or quiet thinking time.
  by tony_cannistra
• Are you pausing your hike to review code, check tests, and so on? Or do you just keep prompting blind until you get done with the hike and then batch review all the work?

  Sorry if this is a dumb question, I haven't tried any harness-based development yet so perhaps I'm imagining a deprecated workflow in the first place

  by pcthrowaway
• Not being able to enter flow state is a very interesting observation. I've felt it too to the extent that I went down a whole new rabbit hole of what it means to be in flow state. Let me know if anybody here wants to know more, happy to post some links.

  To answer your question - I discuss the approach with Claude Code (e.g., should I implement my own ACT model in JAX or PyTorch, Python or Rust or Julia, etc.). Then write the initial part of the code myself. Opening up a blank vscode is a simple joy of life I refuse to give up :-) I'll ask Claude for advice if I get stuck, it will helpfully offer to write that code for me, I obstinately decline. Eventually, I'll get bored of some minutiae or other, at which point I'll ask Claude to complete just that part of it.

  by avilay
• I'd love to have some links please :)
  by tolg
• >I've felt it too to the extent that I went down a whole new rabbit hole of what it means to be in flow state. Let me know if anybody here wants to know more, happy to post some links.

  I'm not a programmer, but I very much enter a flow state working on tickets, or playing a video game on higher difficulties when everything "clicks"

  by Scroll_Swe
• I'd be interested in the rabbit hole of flow state. Also with regards to the dopamine rewards of solving a bug as motivation.

  Sometimes using a LLM can assist these and sometimes it can feel like cheating myself out of a good thing and I'm not entirely sure where the borders are. It could also be related to a sense of ownership or pride in ones work and seeing the value in doing quality work.

  by thinkingemote
• https://en.wikipedia.org/wiki/Mihaly_Csikszentmihalyi is considered the originator of the flow concept: "a highly focused mental state conducive to creativity", it's applied to other fields such as music. Definitely worth going down the rabbit hole.

  https://en.wikipedia.org/wiki/Flow_(psychology)

  https://www.sciencedirect.com/science/article/pii/S002839322...

  by telesilla
• Ok here are the flow related links. This was about 1.5 years ago when I was trying to figure out burnout and it turned out flow (or lack thereof) was closely related.

    * https://youtu.be/VbUFMYs0kXQ?si=xiNw4ZFlla8k-p7w  The person who gives this talk (Rian Doris) has a good newsletter that I still read. I just checked their website and it has gone in full commercial mode, so YMMV.
  
    * https://www.ted.com/talks/elizabeth_gilbert_your_elusive_creative_genius
  
    * https://www.amazon.com/dp/0465074871
  
    * https://www.betterup.com/blog/meaning-of-personal-values

  by avilay
• I've been experimenting with making AI generate literate code. The goal is to have the AI produce a bunch of prose alongside the code. A lot of context for the purpose and design of what you are doing usually gets thrown away, but with literate programming you can save it. And this way I can have some hope of understanding the code being created. I've liked it so far.

  I am using a literate programming project I built (https://github.com/adam-ard/organic-markdown) and I have an AGENTS.md file that looks like this:

  "All the work we do on this project should utilize the organic-markdown literate style that you see explained/demonstrated in the `organic-markdown` project: https://github.com/adam-ard/organic-markdown

  Some guidelines for how you should utilize organic-markdown literate programming..."

  Then I list a bunch of conventions for making literate code the way I like it.

  by aard
• Nice.

  There have been some similar activities on this front. In case you didn't already know of them;

  Literate Programming in the Age of LLMs - https://news.ycombinator.com/item?id=48691666

  Natural Language Outlines for Code: Literate Programming in the LLM Era (inspired by LP) - https://dl.acm.org/doi/10.1145/3696630.3728541

  Visual Programming + Literate Programming: A Better Way to Build Business Logic With AI - https://www.leapter.com/ai-verification-problem/

  Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects - https://arxiv.org/abs/2502.17441

  by rramadass
• "I haven't been able to enter flow state like I can when I hand write code." new flow state is having 10 terminal tabs in diff worktrees and trying to remember what each bit is
  by tombot
• But it feels more exhausting and stressful.
  by UltraSane
• exactly, it's like Bobby Fischer playing 10 games of chess simultaneously
  by npollock
• i hate that there are so many worktrees now that its a real pain to find the actual file and modify it by hand.

  there are so many stupid little changes (e.g., rewording or deleting their numerous comments) that would be done better and faster by me but it's usually just less cumbersome to keep asking the machine to tweak it instead.

  by parpfish
• one option Ive liked for entering flow state for this is running a small 3b param model and using it as an alternative to stackoverflow or websearch while I code.

  I don't often do this, mostly bc I don't care too much about most of the code I write and just want it to work, and then I skip over it and see what's missing and tell the LLM to fix it

  by sometimelurker
• Here's hoping someone starts to pull these things together into an app or Web frontend so the rest of us can configure these setups more easily.

  (I was wondering yesterday, for example, if LMStudio might roll in speech-to-text capability without everyone instead having to jury-rig some custom install/config.)

  by JKCalhoun
• It sounds silly but lately I've been able to hit flow states doing exactly this.
  by codybontecou