Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?by breadislove
- (founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
the page shows the exact trace of all the models we are comparing against and the aggregate scores
we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)
- Keeping track of any AI progress is becoming harder by the day, because there's ambiguity around common/clear/consistent benchmarks. Everything is constantly skewed into favourable directions.by krm01
- I (and I imagine many others) would love to use something like this, but can't, because my data is too sensitive to be uploaded to a cloud of which I have no gaurantees of privacy/security.
Is there any way we do this using rented GPUs and open-source software stacks? Paying for the service isn't the issue, I don't care if it's free or if a cut is taken in some capacity, I just don't want the provider to have access to my data.
by nullbio - In OpenRouter there are Zero Retention options. And if you use a EU provider you can be somewhat sure that your privacy is given.
Other than that there is not really a difference to renting a GPU since the GPU provider can also just steal your data.
Local GPU(s) are always an option if you have the possibility. It is also not that difficult to run with stuff like “LocalAI”
by i5heu - castform founder here. unfortunately, we are cloud-hosted at this point.
but some easy options on the open-source side include huggingface's trl & unsloth.
you can run our data-generation scripts here: https://github.com/castform-ai/benchmax and then hook it up trl/unsloth for training. should be able to do all of this on your own compute.
by kumama - One thing that plagues [insert current FAANG] is the large amount of corpus knowledge that is outdated/misleading or just plain wrong. I'm curious how this addresses that if it's deriving the reward function from the corpus itself.by richwater
- (founder of castform here!) - having worked at FAANG / big tech, i totally get this. our example was on gitlab's open source company handbook but i think a real company's corpus is way more messy and has many sources of truth.
a few ideas i have yet to validate are: - prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others) - actually including contradicting documents that talks about the exact same topic might be a good training example - ideally the model should surface all the relevant info it can find, and explain what it has found. (usually contradiction comes from the fact that the later document is the updated stance) - you could also mine high quality Q&A from public slack / communication channels where questions were asked and someone else in the team linked some docs / answer. those are strongly validated "ground truth" answers
- Hmmmm, this is a problem I have been facing recently.
Basic embeddings give decent-ish results (in the top say 20 chunks). Basic agentic retrieval gives slightly better results so long as the agent part of it doesn't go down the wrong track.
I like the idea of what's discussed in the link, however atm we are on Bedrock KBs and so locked in to a very basic implementation of RAG, because Amazon doesn't have the foresight to make things flexible enough - including making it an absolute pita to use their hybrid search. But, I guess they "work" reliably.
One of our core issues centers around a 1300 page document all about the same overall topic but with minor various for specific procedures/situations. Typical embeddings waters this down so that each chunk really just represents the common theme and therefore lacks a lot of contrast.
But now that luna's (and others) price has been cut, perhaps I'll start experimenting with giving it free rein to explore the data a little in the same way that I do a web search.
One thing that definitely helped was providing a separate index of each section where I had another model summarise the primary unique topics in each section to act as a guide for the agent. I think either we should be chucking the entire doc at a model (400k tokens...so not really ideal at this time) or improving RAG accuracy. For the latter I think even with embeddings, meaning of words and semantic connections are not enough at all - attention is KV so it is 2 dimensional and once I started getting into it I've kind of realised that 2 dimensions aren't really enough to represent the logic that exists between tokens (i.e. sections of documents that refer to a sequence of actions dependent on some logic that references "variables" from another section, i.e. "if x, y has happened then refer to z sequence). There's much deeper meaning to human language than I think basic embeddings covers.
I think it's becoming clear to me that in the same way that embeddings encode the web of semantic meaning of a chunk of text, I need something similar to a hybrid of the author's model + reranker + super-embeddings that encodes as much of the entire meaning of a text as possible and not just semantic.
by fennecfoxy - Nice, but there's no mention of how Luna or DSFlash perform on the same task? (Being 25x and 50x cheaper respectively.)
Nor of how much faster their custom model performs?
by andai - we actually have the test benchmark against luna but no deepseek flash (we haven't added DSFlash into our benchmarking model pipeline)
you can check out the full comparison against all the other models here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
- founder of castform
- Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?by linux_devil
- (founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)by kumama
- It's a matter of cost. Did you see the 100x cheaper?
If you have a workload that is going to be very heavy, incurring a large training cost to make a cheaper model work well with the dataset will be dramatic cost reduction. Most large AI workloads can't afford, or truly need, the expense or capability of GPT 5.6 Sol when cheaper models can do.
Of course you could skip that and just use GPT-5.6 Sol everywhere instead. If you're running a fast food restaurant you could hire Michelin star chefs to make your burger and fries without further training. Or you could have a training program for teenagers, a sourcing program, etc. to scale up to your chain to still get consistent quality without needing that level of cost in each store, but replacing it with a centralized repeatable process.
by jmalicki - I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.by JCharante
- Any data or public links you can share? That surprises me
- we actually have the test benchmark against luna too! it's just not in our title but you can see it in the first diagram below the title. luna does pretty well tbh but sol is just a tad bit better. but luna is way cheaper.
if you want to dive down into the various traces of the benchmark, you can check this out: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
- founder of castform
- founder of castform here again - slightly unrelated to retrieval but on the topic that folks are discussing here, i was actually collecting benchmarking various coding traces for the purpose of training a model router and surprisingly, luna held up very well against sol and terra. it was able to solve close to >95% the that sol can handle at a fraction of the cost. have not benchmarked the OSS models yet but will add the popular ones to the list like Deepseek Flash and Kimi k3 to see how they fare.
will share the full results soon!
- Have been feeling the same. There's a sweet spot that threads the needle between "too dumb to search the right thing / relay the correct results" and "too smart to just stop overthinking and just report the damn thing"by jorl17
- Just an anecdote but thats why Deepseek v4 flash 0731 is my current favorite model. It's really not very "eager" and stays on the task at hand.by hankbond
- Anecdotally, it feels like Opus, Fable, and Sol "get distracted" when you use them for writing code. Great at reasoning and coordination but they will go off on a tangent and refactor half the code base. I only use them for reasoning (of course) and coordinating subagents.by barake
- This feels like the database equivalent of "use the right data structure." We've spent two years assuming the biggest general-purpose model should do everything. It makes more sense for retrieval, reranking, reasoning, and generation to each have their own optimized model if the routing cost is negligible.
- founder of castform here, we believe that as agent deployment moves from experimentation phase where cost doesn't matter as much to deployment (what's the margin of serving the request), there will be a rise in interest in optimized models.
- Yes, and it's only logical things move in this direction because there's massive hardware incentive to do so. If frontier models can be broken down into small models, networks of smaller-GPUs can be utilized. Right now the smaller GPUs are basically paper weights for frontier intelligence.by nullbio
- There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.by aliljet
- <founder of castform here> tldr: we generated synthetic training questions from the gitlab product handbook.
totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet
the training run link is also a little buried but here, you can see the comparison against the various models and their exact traces: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
- I use detailed project files. It has data regarding the project and subtasks as well as task status. It doesn’t depend on agent context and it’s managed to keep the agent on track. Feature creep with the new models is a very real issue. Capturing principles and how to reconcile tasks helps too. Even today it brought up a source of truth issue it had detected. There were multiple authorities born out of a patch and it used that principle to highlight and resolve the problem.by BikiniPrince
- I'm curious about this too as I've been working with a 1300 page document on procedures for [industry]. Where every single section is primarily about [industry] with minor differences in verbs actions etc for the procedures.
I've found with traditional embeddings that obviously you're getting an average of the content of the chunk even with the semantic awareness magic. And our (or I guess the) core problem seems to be a lack of enough contrast between chunks with makes one-shot pure embedding based RAG extremely difficult and low quality.
by fennecfoxy - Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that.
Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).
by Foobar8568 - The big lab models are academically interesting but business wise they seem toast long term. There’s no way for these model companies to compete when the models are becoming a pure commodity and others offering options that are orders of magnitude cheaper.
It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from higher priced tokens or they’ll implode.
by cmiles8