Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- As somebody who is forced to deal with lazy attorneys, I am thrilled that LLMs are starting to push in this space.
First it is going to suck, like it did with coding but after a while it is going to be good enough for 90% of legal work. And finally when attorneys will feel the competition of AI doing their work for 10 cents of tokebs, they are going to be less arrogant and more motivated to finish a task instead of client babysitting them through the process.
by Geof25 - Looks like this is not an open model. I'm curious how the sentiment toward chinese models will shift as labs trend toward keeping them proprietary.
Also, I wonder if this is at all related to the recent reports that Harvey has been getting squeezed on margins by the US labs.
by postalcoder - Not sure how good the Harvey benchmark is. I recently looked into traces from an agent solving their firm knowledge task subset and found multiple problematic tasks and rubrics.
Here are 3 examples (there are more): https://github.com/harveyai/harvey-labs/issues/191
by mathis-l - I might be incredibly naive here, but I really think LLMs can make a difference in _some_ areas of law.
Take, for example, India. The courts are backlogged like crazy. If you're not willing to bribe a judge or the opposite side's lawyer, you'r looking at multiple decades to settle simple cases.
I wish I could waive a wand and have some LLM just go through all of the 58 million pending court cases, identify the clear cut ones and free up the judges' time so they can work on the more difficult ones.
by mlmonkey - As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.by otterley
- I’m not sure if this is the right solution, but we clearly need better tools for the legal space. Anthropic's recent policy update makes this abundantly clear:
“When our products are used in ways that can affect someone's health, legal rights, finances, livelihood, or access to essential services, we require there to be a qualified ‘human in the loop’ (that is, someone who has the authority to review and, if necessary, change Claude’s recommendations), and for the individual affected to be told that AI was used.”
Something as common as a family law case has the potential to affect all of the above. Now imagine being a petitioner, respondent, paralegal, or other 3rd party using Claude to review a draft document only to find out that Anthropic has sent your entire file to opposing counsel.
It sounds egregious, but Anthropic is clearly stating they will use their own judgment to control what people can do with its product, and that they will treat customer data as if it's theirs to disseminate. So there's a massive opportunity for an unbiased tool willing to fill this void.
by victor9000 - Seems like people forgot Z.ai was snapshotting the entire workspace before every prompt. https://blog.ferstar.org/en/posts/zcode-silent-workspace-sna...
How could we trust legal product from a company who was ignoring privacy protection law like that?
- I would love to see these benchmarks split up across "generic" tasks and novel tasks.
For example, redlining an OEM agreement with my vendor. They pretty much all use the same template, and the objections are always the same. I tried using Claude first, and even my lawyer was impressed that it found things they missed. Then I used Claude to redline that type of contract and the other side said "your lawyer was surprisingly thorough". I had my lawyer do the final check, but saved a bunch of money doing the first three rounds with Claude instead. And my lawyer basically just said "looks good to me" by the time they reviewed it.
Or even writing a simple will or estate document.
It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.
by jedberg