Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Thanks for the really nice in-depth post! Hoping for a future one about:

    "Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post."

    by erwincoumans
  • Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp).

    If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools.

    The article is a valuable resource, thanks to the author.

    by mark_l_watson
  • Thanks for the post. For https://Document.bot (Kinda Cursor IDE for knowledge workers), im already trying to improve the harness (besides spotting bugs) using hillclimb experiments. More and more i'm using a AI harness engineering skill in Codex to further improve the app. This blog post helped me to improve the skill a bit.
    by cahaya
  • I did something similar. It started off as a "self-improving agent" project, inspired by autoresearch, then later on I reframed it as "harness training" (discrete program search) borrowing the mental model from ML training.

    I "trained" the harness on a subset of Terminal-Bench 2.0 tasks while keeping the LLM (local Qwen3.6-35B A3B) frozen. Making LLM inference and the task environments fully deterministic was necessary for clean credit assignment. I learned this the hard way after spending the initial 1 month on experiment noise.

    My final results showed that on the full 89-task Terminal-Bench 2.0 suite, the trained harness matched or beat the official Terminus 2 harness for four LLMs that it never collaborated with during training (e.g. GPT-OSS-120B score increased from 18.7% to 36%, while using 55% fewer input tokens per solve). A harness trained only on SWE-bench improved Terminal-Bench scores too. Here's the write-up: https://www.henrypan.com/blog/2026-07-18-harness-training/

    I packaged the training loop as a PyTorch-style framework. https://github.com/workofart/harness-training

    by megadragon9
  • Really good points Lillian. Agreed on keeping the evaluator outside the loop that evolves the harness.

    One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.

    After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.

    by gopalraja
  • The quest for Torment Nexus continues
    by Kinrany
  • IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations.

    I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors. Some day the llm prose will improve.

    I have even a preliminary theory on what is needed for the positive feedback loop: https://zby.github.io/commonplace/articles/reflective-self-i... - (this is not stable yet - but I think you can give it to your agent to read :).

    by zby
  • I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.

    The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.

    I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).

    by bisonbear

Explore Birbla archives