Better Models: Worse Tools

Join the discussion

• Hacker News
• > In case you are curious about Fable: I intentionally did not test it because I was not sure if the classifiers they are running might downgrade me to Opus silently.

  Is this still a thing? I thought Anthropic walked back the silent downgrades so now all the different domains downgrade non-silently.

  by sestep
• Claude Code downgrades loudly but I'm not sure what happens over API or with other harnesses, OpenRouter, etc.
  by resonious
• Pi is my daily driver. I noticed the same phenomenon, and had Claude analyze all my past transcripts for classes of 'edit' error. Built an extension which patches the edit tool to self-heal on the majority of those kinds of calls. It's not 100%, but it cuts down on the rejections quite a bit and saves a few round trips.

  EDIT: It's still quite fascinating seeing the kinds of things the models keep trying to do. It almost seems like when a human has slightly off with their nervous system. The conscious brain wants to do one thing, but for some reason the signals aren't getting to the hands correctly.

  by pugio
• Surprised models still output tools as text when for ages we’ve been able to constrain the output at the inference engine level and constrain the model what tools, parameters etc are available

  Edit: found it, it’s called Grammar-Constrained Decoding (GCD)

  by aetherspawn
• constrained decoding tends to make models dumber - this is why it's rarely used
  by CompleteSkeptic
• Some model providers when using json_schema: true (eg. with_structured_output), it does constrain the output.
  by miketery
• I imagine the challenge comes from recognizing that your model is trying to call a tool before it actually has and only constraining output then. Running a separate pass for an optionally-empty list of tools afterwards may work, but maybe constraining its output like that causes many spurious tool calls.
  by jdiff
• Once models get better, we could avoid paying for a cache read on edit or write calls, and have the model assume they succeeded and not interrupt the stream to get output. We can then just parse the output and once we encounter such a silent toolcall execute it. With high probability its correct (glm in pi for me had 95% tool call success rate) and we can continue, else rewind. As a workaround, you dont want to use the provider feature that interrupts the stream after a tool call, but instead parse the reasoning. I tried this in pi and it kind of worked, but the model got confused about whether edits had been applied and in several runs either double checked or used the bash tool instead, negating any possible benefits.
  by bazodedo
• There's a spectrum of possible explanation, from "this is a model training artifact which for now they correct via the harness" through to "this is deliberate, and creates a constantly moving target to make third-party harnesses less efficient for lock-in purposes".

  I'd not discount the adversarial end of the spectrum.

  by mft_
• Suggestion for Pi: capitalize tool names for the Sonnet/Opus models (edit -> Edit, bash -> Bash, ...).

  The rationale: Anthropic's own harness (Claude Code) uses PascalCase tool names — Bash, Edit, Read, Write, Glob, Grep. Since the models are post-trained/aligned against that harness, those naming conventions are effectively baked into the model. Matching your harness's tool names to the same casing puts your inputs closer to the training distribution, which lines up with the more reliable tool use I've seen in evaling.

  A related pattern that fits the same distribution: for long outputs, have the model reserve placeholders first and complete the work across multiple steps.

  Reference: https://github.com/evotai/evot/commit/765151796c43965964a9da...

  by BohuTANG
• > Suggestion for Pi: capitalize tool names for the Sonnet/Opus models (edit -> Edit, bash -> Bash, ...).

  Pi is doing that.

  by the_mitsuhiko
• It sounds like harnesses might have to start to have model by model system prompts, though retrying works, I guess. It reminds me of the ancient times when browsers all read HTML and CSS differently, and differently on different devices. In that sense, this is nothing new. I was going to say, at least we don't have different device types, but then, the model still has to output the right variant of `grep` as well.
  by lukasco
• The flip side of this is training models to better understand harness interaction, I suppose, which (if I understand it properly and I am in no way sure I do) appears to be what the Qwen AgentWorld model is doing?
  by dofm
• The problem with hyper targeting harnesses to models is that you end up locking yourself quite quickly into special behaviors of models, and you make your sessions non transferrable. That can be an acceptable trade-off and I know people who do that.
  by the_mitsuhiko
• It's not the failed call that worries me. The call itself was correct, and the only thing off was a couple of invented fields. That makes the runtime feel like part of the model's interface rather than just an implementation detail. Train a model in a forgiving environment and other runtimes end up inheriting its habits.
  by aberrahmane_b
• > You can ask the model to produce valid JSON

  Doesn't always work, for better performance you can kneel and start begging

  by wseqyrku
• rubberhose cryptography makes another comeback!
  by nashadelic
• Humiliation-assisted prompting. it's the future.
  by automatic6131
• As critical as I am about articles endlessly concerned with the weaknesses of closed-source cloud LLMs, this one is pretty great, and not just because it concerns interactions with Pi, which looks to me like it's going to end up a sort of quasi-reference implementation of an open source harness, and because it has so much useful technical detail.

  But:

  "Now I’m somewhat worried about the track we’re on here. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology."

  Only implicitly?

  --

  Many decades ago when I was working on research related to using MOOs as a learning environment, you would add "tool calls" into the stream of text that a MOO object might generate, so your rich client would e.g. show a picture, load a web page in a frame, move you on a map, trigger a change in an on-screen representation of an object.

  Everyone who tried this in MUD/MUSH/MOO clients ran into more or less the same problems that LLM clients do: any attempt to shoehorn control sequences into in-band content was riddled with security risks, objects accidentally triggering the wrong interface etc.; you could never truly communicate out-of-band.

  The more I read about how agentic harnesses work, the less embarrassed I feel about the code twenty-something-year-old me wrote in a MOO client.

  by dofm
• When building agent integration for my serverless backend https://saasufy.com/, I decided to not use MCP but to put curl commands inside skill markdown files instead: https://github.com/Saasufy/skills

  The curl command is extremely popular so models seem to be really good at using it.

  Also I like that curl uses a bash syntax and my platform requires JSON payloads; it makes the separation clear to the agent. I find it to be very reliable.

  by socketcluster
• I really like this simplification
  by nashadelic
• The skills are very readable too, so you win a nice documentation for free. At the very least it's human readable machine instructions.
  by gchamonlive
• This is easily solved with good error messages.

  Claude always gets the syntax wrong on my tool calls.

  So I did a revolutionary thing and made the error output print helpful guidance on how to correctly call the tool.

  The agent tries again and always gets it right. Total time “wasted”: 1-2 seconds. It happens every session, but it only happens once per context window. After that the agent holds on to the lesson.

  To do this for your own tool calls, imagine what you’d do in the agent’s place - what info you’d need so you can correct your mistake. Assume the agent wants to achieve the goal so it’ll try again. These are probabilistic systems, so we need to give them an extra loop to get the deterministic bits right.

  by cadamsdotcom
• LSPs and linters serve the same purpose. I use the latter in git hooks.
  by esafak
• So, is this part of the tool definition, or did you create your own coder agent?
  by Shorel
• This maneuver requires you to anticipate all the edge cases or error messages beforehand which is practically not possible in many situations. The moment something unanticipated happens or the model changes its processing logic, the tool call system stops working just like any other deterministic program or tool.
  by pyeri
• I've built a library that makes creating rich feedback systems easier, check this out:

  https://tool2agent.org/

  by klntsky
• So, are you saying that skills are not such a good tool for agents to learn, they still need tool-trial-and-error dance after injecting them? (I'm assuming each tool comes with its own skill.)
  by siwatanejo
• Pi already emits good errors messages; I always see Claude Opus 4.8 correct itself in its next attempt when it gets a tool call wrong.
  by euiq
• This will cause an extra round trip to the LLM. Which means more $ spent.
  by psadri