Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I’m making an in-terminal IDE over at https://github.com/paradise-runner/toast and deep seek flash has been doing a wonderful job since the last weights release. Makes me think we’re past expensive AI to do general engineering work.
    by dividedcomet
  • i've been working on several rather large projects these past few months, and i'm trying to write as little code as possible.

    I don't think i wrote more than 10 lines of code in the largest project i'm working on. Lines of code: Java: 900_635, typescript: 725_418, C++: 180_445, Dart: 96_181.

    It's been obvious from the start that no model, as good as it is, can do large(-ish) amounts of work by its own without supervision, control, criticism, etc ... If left unsupervised, models usually do half the work, leaving stubs and todos everywhere.

    Quality comes from applying software engineering principles as much as possible, just like you would do with teams of junior devs: planning sessions and implementation sessions with adversarial critiques, specifying as much as possible upfront, planning unit/smoke/integration tests, etc ...

    Many systems rely on swarm of agents to build software but i've found it very difficult to get good results without lots of overhead/token waste because of inter agent communications mostly.

    So instead i built what is mostly a workflow engine to structure / organize processes into workflows with different agents assigned different roles. I've setup a basic landing page here https://kodfactory.com if anyone wants to follow along.

    by guybedo
  • So I can not code. I just tell an AI the concept of what I want my software to do and execute in blocks. Most operations are well defined and depend on a central mechanism and that is where everything starts. The rest is like an octopus : the body defines the concept, the arms the consequences. This way there is no need for long code since every segment has a clear beginning and end. It is also quite efficient overall. My point is that long code can be circumvented by clever architecture.
    by Terratrader
  • It really depends on how you define "complete". It's possible the answer in "not large at all" if we're looking for high-quality, reliable, user-friendly software that works near perfectly after being written, tested, deployed completely by AI with no human intervention.

    People who try to get AI to do a lot of un-manned work seem to have a bit of a bias toward viewing LLM work as "good" or "good enough" in a way that they wouldn't have accepted in the pre-AI era.

    by vcryan
  • What size project an AI can complete on its own is IMHO not an interesting question. The AIs and harnesses are getting good enough that quite large things can be built, primarily as a function of cost.

    The interesting question in my view is what is the largest system an AI can maintain over a long time horizon -- while keeping a coherent architecture that adapts and responds well to change. All the while remaining performant, scalable, and understandable to humans. I am unaware of what is the currently longest-lived vibe coded project but it cannot be more than a few years old.

    So to my mind, we don't have very much real data yet.

    by _doctor_love
  • isn't this a harness engineering problem at its core (and the human equivalent being a product org problem)?
    by htrp
  • I think reproducing existing software and building a new product are quite different tasks.

    In a benchmark, there are usually clear tests and a correct reference. In actual product development, requirements are often unclear, and we do not always know what the correct result is.

    Passing tests is also not enough to confirm security, maintainability, or operability. I would like to see a benchmark where an AI continues changing the same product for several months. It would be interesting to see whether the architecture remains understandable and safe after many changes.

    by arstructinc
  • This actually proves the hardest part of software engineering is defining requirements.
    by manoDev
  • That makes it sound like a PM problem. The hardest part is building the mental model which doesn't crumble. I have worked with humans who had the same failure modes as LLMs.
    by wannabe44
  • I'm looking at these reports and then at my sessions with Claude. In latest one, it got stuck in a loop fixing one thing and breaking another for 10 iterations, until I've noticed and gave it the insight. Or where it started continuously writing "echo", I don't know for what purpose. And I ask how, how can you not be the driver; how can you let these agents roam free and actually accomplish things.
    by eithed
  • we bomb schools and walk past beggars and devote our surplus time and money to scones and brain rot and gym memberships.

    our c code probably looks at us and says: how, how can you let these agents roam free.

    llms are not code and are incompetent and inefficient relative to code, but on the other hand you cannot roam without a little echo echo echo in your blood.

    by dnnehgf
  • I believe they can make ANY software project the largest it can be!

    This solution should look for more problems.

    by thisisauserid
  • The benchmark does not reward producing more code.
    by anothermathbozo
  • Yeah came here to say the same thing. The Hard Problem of software engineering isn't making a complex monstrosity with millions of lines of code and endless feature creep, but not doing that. This was as true 20 years as it is today.
    by marginalia_nu
  • You say that, but I asked Codex to build a Asteroids (but FPS) as a web app, and it wrote it in under 4K LOC, and a Tempest implementation in 6.5K LOC. Those seem like really reasonable line counts to me.

    I also built a caching proxy server for apt packages, that's a bigger project, and that is 30.5K lines of golang, which doesn't seem bad compared to apt-cacher-ng which is 19.3K of C++ (my project does significantly more).

    by linsomniac
  • We (Team of 3) built a retrieval engine from scratch in .NET for investigative search, with AI doing most of the backend implementation while humans handled the design and architecture and front end. It was developed using a task based, incremental approach. Includes more than 3,000 tests and extensive documentation, and is optimized for asynchronous ingestion using NATS and a disk backed WAL.

    It includes a custom WAL, structure aware semantic chunking, ingestion and indexing pipelines, hybrid (rrf), text and vector retrieval, ACL aware filtering at query level, query classifier, citation tracking, and a CrateDB backend that performs fused retrieval over a single table. total LOC: 180 K. Status: evaluation by State agencies for investigative search on legacy documents. tool used - Codex, Luna, and earlier codex 5.3, and gpt 4 mini.

    by sreekanth850
  • "we have more than 3,000 tests"...

    Great, but since no human has looked at them, how do you know that they are relevant and actually test undesired behavior?

    Also, would you be able to point at which tests should be added in the future?

    by harrouet
  • How long did it take you to review things for correctness? Like an incremental query engine is a few thousand lines of reasonably difficult code on its own, a reliable WAL is another few thousand, etc, and each of those might take me days or weeks of thinking to have any serious confidence in them. I guess you can get provenance and ACL tracking "for free" from a query engine, but it still seems like a very ambitious project.
    by AlotOfReading
  • I'm experimenting with large-ish software projects using Claude Fable. The results are surprising to me.

    - Project portfolio management software. Good results so far. This is unguided experimentation: Fable researches feature ideas and plans them, then Opus 5 implements them, without me in the loop. The features include things such as an auth server, full-text search, audit logging, resource leveling, HA/DR, IaC, and more.

    - FHIR & SNOMED medical software experimentation in demo-only environments. This has mixed results so far. This is guided experimentation: the FHIR specifications and SNOMED specifications are thorough and I'm in the loop. Fable plans OK, but Opus 5 is thrashing a lot, going down many rabbit holes.

    by jph
  • "Reproduce an existing piece of software" doesn't seem like a problem that generalizes well to "develop a new piece of software," for a fair number of reasons.
    by gipp
  • I agree...if there is a manual that documents well specified software including edge/corner cases i would kinda consider it a waste of tokens unless the aim is to build a better version of the software
    by dougSF70
  • I have been using Claude to build a clone of Bash in Rust (https://github.com/jdstanhope/huck). It has been goin going on now for 81 days, 2600 commits and I have covered a good chunk of the features such that I can source by .bashrc with a number of extensions. It has been doing pretty good but it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests.

    That being said, I am still guiding it on a regular basis and only recently let it run open ended trying fixing small to medium size bugs. The next stage will be try let it run in a loop trying to fix the 100 issues it has identified as well as getting most of the bash test suite to pass.

    by Jdstanhope
  • I find most of the time it would be fine on its own, but every now and again you avoid a really bad degradation through guidance. Some kinds of big change definitely problematic.
    by justincormack
  • > it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests.

    you ever worked with human engineers before? :D

    by rco8786
  • Very cool, but I'm skeptical. Over the last year, every Cursor/Claude/Codex project I've worked on devolves into shitty, grafted-on, duct-tape and bandaid'ed architecture, with "fixes" just working around broken code, and tons of duplication that lead to whack-a-mole. The system works at first, but then bugs start piling up and eventually take over. Everyone I talk to shares the same experience. Not to mention that if I, human overseer, don't course correct it many times a day, it goes off into Neverland.

    Yet I see comments and articles like above every week, where someone says "Yeah, I just left my agent run for a few nights, and then it made this awesome tool."

    There is a huge gap here, but I can't pinpoint it yet. Either I (and everyone I talk to) is Doing It Wrong, or your code is actually flawed at its core and bugs will take exponentially longer to fix and never converge, or... I just need to let my agent run for 3 months??

    by khazhoux