Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Smart trick, but assumes the “dumb” llm is smart enough not to derail into an article about the lives of South American red ants. Obvious exaggeration, the point being outcomes should stay strictly within topic, avoid unrelated bloat and hit the target.by claudiosf1
- If you use structured outputs they’ll usually stick to the program. Not to completely constrain the categories like TFA was saying, but something like
Where you don’t really care about the rationale but you’re using it as a pseudo thinking for models that don’t support it.{ rationale, categories }Luna is surprising capable and cheap, and I haven’t done this type of thing since before GPT 5 so might not be such a useful trick now
by phoghed - No? This is just giving up and hoping.by iandanforth
- No change from regular chatbot coding, then.by chrisjj
- Is there a solution you are using to solve this that is more accurate and cost effective? I'm working through it now so would be curiousby motoxpro
- This is basically HyDE (Hypothetical Document Embeddings), no? I had tried this approach in the past, worked with limited success.
- https://arxiv.org/abs/2212.10496 for others like me hearing about HyDE for the first timeby kaycebasques
- We have this running in production. Can get pretty expensive and slow. We are trying to replace this with cheaper and faster methods that don’t hammer our LLM and elastic search endpoints as much.by memjay
- I would propose the following, query vector store for 10 closest categories based on a query, feed it to an LLM, in the prompt ask it to produce a single digit 0-9 representing the number of the most appropriate choice. Use plain text prompt, dont inflate token count with JSON. There you go, you just drastically reduced the output pricing.
Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.
by piterrro - good this yeahby cesargstn
- Or press 9 to hear these options againby jddj
- New embedding models support queries, so you don’t need to hallucinate a document before finding the nearest neighbor. Curious how it compares to this approach since you’d get to skip the LLM altogether.by ed
- What does it mean that an "embedding model supports queries"? An embedding model maps text to embedding vectors. You can always perform queries with such embedding vectors against a stored set of embeddings.by bonoboTP
- Prompt expansion of input to extra categories makes sense if your embedding isn’t working well. But on its own, why use the LLM at all? I think you could have demonstrated the original step first and then shown that it’s useful.by arjie
- I did something similar 10 years ago, but instead of llms I used word2vec to calculate a embeddings of product descriptions and map those to existing categories. The LLM approach is very likely better, but I'm curious what the cost difference is.by cimi_
- Did you validate this by running a A/B test? Main question is were you able to classify back into your known categories correctly all the time, or did the errors compound from the llm hallucination plus embedding searchby eka1
- Using a Nano model, a tad worse than shipping a vocabulary to a larger OpenAI model. (And it’s an huge improvement on not classifying the queries at all).
But no classification is perfect. In search in particular, you will also want to have places for manual intervention for high priority queries.
by softwaredoug - > In the notebook, I compute a MiniLM embedding of every real Wayfair classification. I compute the embedding of the fake, hypothetical embedding from the LLM. I then dot product the fake embedding into the real ones to find the most similar. Producing: [the right answer]
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
by Majromax - Yes what you're describing is a classic way of doing query understanding.
I've found, though, getting it in the language of the vocabulary has generally improved performance.
Further, when searching for "blue shoes" you want to separate the color from the item type. So its useful to have a dumb LLM do this for you. And with the LLM in the loop, its further useful to get it into the language of the taxonomy to improve embedding retrieval accuracy.
There are of course many ways to skin the cat here :)
by softwaredoug - Nice trick. Couldn't you embed the query though, compare it to the embedding of the categories, then ship only categories that are close to it in the prompt to a smaller model?by pu_pe
- It might be a little worse, but it will definitely be way cheaper.by liampulles
- Yeah I had the same question. What's the point of the intermediate step?by tantalor
- Agreed that you almost certainly can just embed the original with most modern embedding models.by vessenes
- Yes absolutely that's another good trick.
Even better is to search the corpus first with like naive BM25 / embedding search, aggregate over top N to get most representative categories, then have the LLM categorize in that set.
by softwaredoug - It's basically a variation of HyDE (Hypothetical Document Embeddings), and the rationale is that the embedding of the query is not necessarily close to the embedding of the answer. If you generate a hallucinated answer, it can line up with the actual document better (in the embedding space, via BM25, or hybrid).
But honestly, it only works for common knowledge that's already in the LLM. If the target document contains very niche or private information, then the hallucinated answer's embedding can be even farther away than the query's.
by kgeist - A common case I have is when you don't have classifications to begin with. For example, you need to find what users complain about most. I take embeddings of all records, then cluster the embeddings into semantic groups, then ask an LLM to take a random sample from each clustered group and create a classification for that group.
This method is sensitive to the thresholds (what is the maximum distance between embeddings for them to be still considered part of the same semantic group), so I run it all in an agentic loop where an agent tries different thresholds and clustering algorithms until it's satisfied with the result, plus it may deduplicate some groups.
I run it all on self-hosted hardware, so it costs nothing to leave it running for, like, a night, and as a bonus, none of the corporate data leaves the office. I think a rigid set of manually created classifications may not capture all the possible classifications that can exist. Needs a review by a human, though.
by kgeist - The idea of distance thresholding models that are trained to satisfy an ordering constraint is a bit strange. The reason it's hard is that there isn't a threshold!
You can slice and dice it a ton of different ways, but the significance of groups is incidental.
It's a good starting point, but having done this a few times for a few companies it always seems like it needs substantial human review.
by nostrebored - I worked on spam classification for litigation targeting in the early days of CANSPAM [0] enforcement.
We had a similar problem where you can literally millions of email that we were pretty sure came from only a limited set of bad actors.
We first started classifying emails into buckets by From, mailserver relay chains etc as that's all we had to to go on.
Over time, those buckets got linked to spammer signatures and then we narrowed down from there.
Fascinating to see this happening nowadays with LLMs.
by alexpotato - I can’t believe programming is now at the stage where advice like "first have the computer give you totally wrong answers, then just find a function that maps the wrong answers to the correct ones!" is a thing.by Sharlin
- the trick is that it's not totally wrong to start withby agos