Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Highly suspect of content marketing.

    Ends with referring to a product, and saying "this is a parody post", after pretending to make a serious point.

    by jorisw
  • How to write Jev in 25 lines of Python:

      1. draw a circle
  2. import the rest of the owl
    by zeroq
  • I also built one, but mine uses embeddings. It classifies concepts defined by a collection of positive and negative examples. The classifier model is trained in <1 second using ridge regression. The model itself is exactly the same shape as the embedding, so it works as a concept embedding. Since I already have a dataset, I can use it to do conformal prediction in order to calibrate confidence scores. Jev, on the other hand, has a generic model, not trained on in-domain examples, so its confidence scores are uncalibrated for any non-generic task.

    So you might ask: how do I obtain the training examples? Just collect samples and use a coding agent to classify them as match or no match. From time to time you can add more examples to the dataset to have your concept adapt to changes in input distribution. It's all automated, but it only uses LLMs to train concept vectors, after that it works like a regular embedding model with a calibrated classifier on top. It's also 20-30x faster than Jev, free, and runs on CPU.

    An illustration of how it defines a concept as opposed to simple cosine similarity: https://github.com/horiacristescu/semlabel/raw/main/images/c...

    by visarga
  • > "25 lines of python" > "import Solution" ok
    by philipbk
  • Beyond the missing latency and compute comparisons that Heaney commenter mentioned, also nothing about its error rate compared to Jev (nor if it even always outputs in a format the app can parse, not sure how solved that is).

    But then at the end it says it’s parody. Maybe HN title should say it’s a joke.

    by no-name-here
  • The number of - “I did/invented Jev last year”, or, “here’s a version of Jev I vibed up last night” is getting a bit ridiculous.

    Especially ridiculous is how the hacker news crowd seems to be taking these at face value…

    There was one the other day with a compelling demo. But when you looked closely at it, it was feeding in the options with the word “best” on the option to pick and a fine tuned model designed to recognise that word…

    by iamflimflam1
  • Because of masked attention in LLMs, if you put the options before the body (the email to analyze), the transformer already knows what it needs to look for, and can use more tokens to create state to address that specific task (BERT has no mask in the attention, so tokens attend also to next tokens). You could also do a few examples in the system prompt to improve calibration.

    Another trick that works is to repeat the question two times: "I'm repeating the task and labels for clarity: ..."

    by antirez
  • Going directly for the logprobs is always icky when you use a chat model as base, because they are trained to write prose as output. So your "choice" tokens and thus their probabilities might get diluted in whatever else it wanted to say. If you have to do it in the same way as this post, at least add clear system instructions and a carefully worded beginning to the assistant output section of the prompt to lower the chances of it wandering off immediately.

    I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).

    by sigmoid10

Explore Birbla archives