How did we make DeepSeek outperform Opus

Join the discussion

• Hacker News
• > he trigger was watching deepseek-flash fail on the simplest /review run, every shellCommand and readFile call bouncing back with a raw zod issues blob, the model unable to recover because the error wasn't in a form it could read. by the end deepseek v4 pro was beating opus 4.7 6/10 times on our internal evals.

  I think this is why Xiaomi forked OpenCode and created their own agent hardness, to reduce friction between model and harness:

  https://github.com/XiaomiMiMo/MiMo-Code

  by bel8
• probably yep, i doubt they'll be able to maintain that well. we did a lot of work improving MiMo's cache thrashing, it was pretty bad early on. Now we're seeing 97% cache rate on their models after lots of prefix work.
  by ahmadawais
• hey HN, sharing harness engineering deep dive on tool calling repairs for open models. i've been thinking about why "open model bad at tool calling" is almost always a harness problem, not a model problem. spent time looking at billions of tokens from DeepSeek (and other open models) in our coding agent. ended up building a tool-input repair layer on top of Zod. by the end, DeepSeek V4 Pro was beating Opus 4.7 in 6/10 of our internal evals. the main things that helped:

  - most failures came from a small set of recurring schema mistakes - switched from preprocess-then-validate to validate-then-repair - handled some weird cases like markdown auto-links leaking into file paths

  full writeup: https://x.com/MrAhmadAwais/status/2050956678502420612

  video version (more detailed): https://www.youtube.com/watch?v=f61DCDwvFis

  by ahmadawais
• Thanks! Nice findings.

  Is the output, that others can use, the research and findings, or is there a tool or something that came out of this that I can plug in to one of the common harnesses?

  I guess that brief note in the twit is that it'll be part of that harness you're opening up in the future?

  by kadoban