Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I put Decisions API against classic statistics experiments: (1) loaded coin, and (2) marble selection from a jar, with replacement. I tested both predicate and choice questions. I ran thousand trials against each experiment, and I also did an experiment where I change the order of choices, to see if it matters. Summary:
* Using predicate questions gave nearly perfect/expected probability outcomes
* Asking it to choose an outcome behaved differently from drawing randomly - if the true probability of a red marble draw was 50%, using Decisions API produced 86%, i.e. it picked the right marble but gave a significantly more biased weight on its choice
* Changing the choice order changes the probabilities! Moving the red marble from first to last choice changed its probability estimate from 86% to 73%
by armcat - Just going to drop this here: https://jeffyclassify.com/
Open source classifier models you can run and train locally on CPU
by nico - Note that, being that this is gpt-6-luna under the hood, this offers you 1m token input window, and multi-modal (image) input. In my testing so far, I'm seeing 160-175ms end to end. Worst 5% 285ms, worst so far was 743ms.by agentdev001
- > gpt-6-luna is the only model currently available.
Clearly this was a rush job to respond to the competition. I am more curious about how the dedicated model will perform after they've had time to do it the right way. The probabilities I am seeing so far do not correspond with figures the business would find very agreeable.
The hidden danger with this could be demonstrating how thin the veil actually is. We may wind up reducing confidence in decisions simply by making their probabilities visible. Some kinds of information are quite hazardous.
by bob1029 - Boy am I glad we're already dropping the "noul" term for a binary decisionby isoprophlex
- Ran my decisions evals (still rudimentary, less than 600 calls (UI component selection, chat charting, tag selection, PKM stuff)) on this via OpenRouter against Jev and Mercury Decide. Jev because it has replaced my mt0 efforts by sheer force of affordability (more importantly, the limits running on a MacBook Neo bring even after vocab pruning and quant insanity) and Mercury Decide because I do like dLLM efforts (and I'd like to use fewer model providers if possible).
Preliminary of course, but seems to be slower than Jev and similar to Mercury Decides latency, though not in growing linearly with the amount of input (346ms p50 and 860ms p95, (Mercury Decide also had some extremes up to 1,3s that were around 800ms today, likely preview related, it scaled far more consistently with size)), less "confidence" concerning my ambiguous UI component and response shape specific tasks (have very specific use cases for these models which Luna often fails to meet at 0.6 and lower), lead to a few failed calls which neither competitor had (4 vs 0 for both) and measured more expensive than Jev to boot by a factor of 3,1 times on average (Mercury Decide pricing I think is still unknown so no numbers there).
Basically slower, more expensive and less capable than Jev, roughly on par with Mercury Decide (provided, in my insane set of use cases and requirements that are a PKM focused Firefox fork with multiple infinite canvas using decision models to improve information synthesis from multiple sources).
Seems a bit undercooked overall and I'd rather frontier-labs don't jump on bandwagons until they can offer something competitive in price, performance or both. In fairness, though, I have yet to test image input, maybe that makes all the difference. Also, again, mine is unlikely to reflect everyones use case, so interested in seeing others results.
Didn't comment at the time, but having read up on Devday after the fact, there seems to have been a lot of that going around. Notion and GDocs, Jev, Muse, most seems to have been cloned from existing competitors (and despite infinite, ultrafast, ultra code tokens with unsandboxed Mega Astra not that amazing to boot).
Prefer less announcements, but focused and at a higher quality. Considering ChatGPT Atlas (their Chromium based browser) and its insanely fast death, I'd be skeptical to put much into any of these even if they were in some way an improvement over what is out there. Maybe focus on a fresh pre-train and some sandboxing improvements.
by Topfi - The response to Jev should be the nail in the coffin over whether or not the AI business is a commodity market.
Out of no where Jev appeared as the next round of the price wars. Jev showed the value of System One models. A fast yes/no/confidence score not only is cheaper but also often all people want. Open source versions flood hugging face and now the big players are giving up a potentially big driver of output tokens to keep customers and race to the bottom price wise.
If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
by TSiege
Returned:curl https://api.openai.com/v1/decisions \ -H "Authorization: Bearer $(llm keys get openai)" \ -H "Content-Type: application/json" \ --data ' { "model": "gpt-6-luna", "input": [{ "role": "user", "content": [ {"type": "input_text", "text": "I am angry about the new product feature"} ] }], "questions": [{ "type": "predicate", "name": "complaint", "instructions": "Is this a complaint?" }, { "type": "predicate", "name": "compliment", "instructions": "Is this a compliment?" }] }'
That https://api.openai.com/v1/decisions endpoint is notable because usually when OpenAI define an endpoint like that it ends up as a defecto standard for other providers.{ "model": "gpt-6-luna", "answers": [ { "type": "predicate", "name": "complaint", "probability": 0.91 }, { "type": "predicate", "name": "compliment", "probability": 0.06 } ], "usage": { "input_tokens": 310, "input_tokens_details": { "cached_tokens": 0, "cache_write_tokens": 0 }, "output_tokens": 0, "output_tokens_details": { "reasoning_tokens": 0 }, "total_tokens": 310 } }(I turned this all into a new llm plugin: https://github.com/simonw/llm-openai-decisions)
by simonw