Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- this seems really low?by chalmovsky
- Is this a proprietary harness? I feel like harnesses have a huge influence on how models behaveby klooney
- This benchmark expects models to not only do the happy path, but also edge cases, even though the ticket they give to the model doesn’t specify if. This to align more to real world tickets. Usually with benchmarks it’s the other way around: only implement what’s asked. So I welcome this type of benchmark because this is how I use the models.by andhuman