Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- TypeSafe's new model, Jev, doesn't write text. You give it a situation and a question, and one quick call returns a probability for each answer.
The test: 770 posts from Reddit's r/AmItheAsshole. Each model had to predict the verdict Reddit actually gave.
Jev came second of seven setups. Sonnet 5 was a bit more accurate, though the lead is borderline once you account for how many comparisons were made.
Jev's median call was 6.3× faster than Sonnet's. Fast, but not 40–200×. That was measured on one laptop in one evening, so treat the exact number with some caution.
I have no affiliation with TypeSafe. Code, logs and method are open, so you can rerun it yourself.
- Too much text. The only question I have and want to see at the top is whether it's more human aligned than LLMs (or, potentially, overfit)by chaoz_
- Using Reddit as the ground truth here feels like evaluating on training databy maplet
- It’s definitely not in the community’s interest to have auto posting bots also commenting, and I am not advocating for that
But in this case I appreciated the summary so i didn’t have to waste any time on the article.
Hopefully next time, my agent will read it all for me and make this comment.
by edmundsauto