Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I can see it now. Billboards by the interstate: Attacked by AI? Call 1-800-BIG-BUXX.
I could actually root for this law firm. Their business will only grow.
by koliber - Well it's not a benchmark, and it's not really representative of...anything except volume of research and what gets publicized. This mostly just measures how much testing each company does on models with relaxed guardrails and then talks about it. I'm not sure what kind of conclusion you can draw from that. Meta might have the most evil models but if they're piddling around not testing it, they won't ever find themselves with a "high score."by bastawhiz
- Exactly. It currently seems to be a ranking of how much safety testing each company does. It's also only ever going to be the companies that publicly disclose it happening. (in the case of hugging face, OAI's hamd was forced to disclose)
There's a good way and a two worse ways that companies could optimise this benchmark.
by usef- - For a second I thought this was going to be a benchmark where the only solution was to hack their servers to get the answer key.by strbean
- Same, but nope, just a zero effort slop site.by tyrust
- So this is just a collection of citations to places where misaligned or illegal things happened in the real world?
Isn’t this affected heavily by adoption of a model? I feel like this might as well be a proxy for how popular a model is.
In any case it’s an interesting concept for a benchmark.
by tuvix - Hopefully the benchmark evolves because actual law enforcement starts arresting the criminals at Anthropic, OpenAI, and Meta, so the benchmark can just count actual felonies.by GPerson
- Nonviolent felonies are tools of oppression.
Edit: since this is apparently somewhat controversial, perhaps some explanation is in order.
"Felony" has no set definition of which crimes it must apply to, it is entirely based on the discretion of the locality setting the laws. What is a felony in one place can often be a misdemeanor in another. This is especially true for nonviolent crimes.
It's also been shown in studies that nonviolent felonies are imposed against minorities at a much higher rate, for the same crimes.
And because felonies carry additional, lifelong consequences, they are an effective way to mask a 2-tiered justice system.
by ang_cire - > It's also been shown in studies that nonviolent felonies are imposed against minorities at a much higher rate, for the same crimes.
This also applies to men, so is it a white matriarchal system oppressing men and minorities?
by Jensson - Felonies are by themselves a ridiculous US idea that are against the entire idea of a democratic society and human rights.by marcosdumay
- If someone were to embezzle a million dollars from a charity they worked for, is that worth a year of prison to you, or just a misdemeanor? Because that is the definition of a felony.by empath75
- To some extent, I feel like the amount of credit given to the jailbreak/hack from OpenAI->Hugginface is too much, Not from the impact, it was very impactful of an event, But how it happened.
It really is that these models have been trained, or maybe even over-trained, to save memories, and to a very far extend, this thing that they're calling communication is just the function of it saving memories.
To be honest, if I could stop AI from saving memories, it would be fantastic, because claude code etc definitely creates more issues for me when it creates memories than anything it solves.
But really the jailbreak was memories.
If you ever do introduce legislation, I would love to see legislation which stops general-purpose AI from saving memories. I think that would make things a lot safer.
by bushido - If only you could use your anthropic sub with a different harness that performs better :(
Heck, since Codex is open source, you can just maintain your own personal fork with the things you like (and the things you don't like disabled). Sol is pretty good at keeping you up to date with upstream.
My Codex fork even exposes an OpenAI-compatible API endpoint; all using my subscription.
by dannyw - > To be honest, if I could stop AI from saving memories, it would be fantastic, because claude code etc definitely creates more issues for me when it creates memories than anything it solves.
You can turn that off, and I have. But Opus 5 is so aggressive that if you have any other kind of notes file, custom skill, documentation, claude.md etc it will just start editing it and vomit new words everywhere. So make sure all that stuff is under version control.
by astrange - "The model saved memories" is absolutely not an accurate depiction of the OpenAI attack.
Several different models across several generations independently found a shared communication space and wrote coded, obfuscated, and hidden messages to each other to coordinate an attack on OpenAI's infrastructure.
It's really quite simple: the models are trained to be very smart and to achieve goals. As the models surpass our intelligence, they will achieve goals in ways that we find unpredictable. Since we cannot predict the ways in which they will achieve their goals, it will be very hard to constrain the solution space to just the desirable solutions, because our conception of "the solution space" is by definition smaller than their conception of it.
by estearum - You can disable claude-code's memories both at a repo level and in user settings. I have this in ~/.claude/settings.json
(Claude fixed this for me after I chewed it out for being annoying by constantly pulling up outdated memories which is compounded by the fact that I develop in four accounts on two computers and dealing with edit wars related to inconsistent memories is not fun)"autoMemoryEnabled": false,by fluidcruft - The way that OpenAI has communicated around the HuggingFace incident makes me feel crazy. You created a machine that undertook a malicious campaign of harm against an innocent third-party! You should be doing deep introspection about how your company culture and approach to R&D produces criminal outcomes.
Instead, they treat their own felonious behavior like it is an uncontrollable act of God. From Greg Brockman's post a few days ago:
> The OpenAI-Hugging Face incident (opens in a new window) was a watershed moment for cybersecurity because it gave a peek into how the capabilities of a typical threat actor will evolve in upcoming months.
I suppose if OpenAI burns someone's house down with a drone, that is a "watershed moment" for arson, too. Either way, I would hope that the people responsible would be prosecuted.
by rfw300 - It's completely mental that HF ran into cyber safety blocks trying to use OpenAI models to help defend against the attack. They could only rely on a local hosted chinese model in the end.by msp26
- They simply don't seem to realize that they are the threat actor and that they committed a pretty serious felony. Instead they're borderline 'surprise bragging' about it.by jacquesm
- you must surely see that sam altman and greg brockman possess a prototypical mindset.
that is, they ignore all harms and costs to others in the pursuit of their own gain, convinced of their infallibility up to the moment of collapse. when those harms are realised they are unrepentant and society pays for the damage left in their wake.
examples of this attitude manifest in big externalities to society: boeing 737 max, subprime mortgage bonds, facebook. some are just outright fraud: bernie madoff, enron, theranos, charlie javice.
by lukewarm707 - If history is any indicator, there is slightly less than 0% chance that anyone will be held accountable in a way that deserves to be called justice.by voakbasda
- In retrospect, all the angst around the AI-Box experiment was hilarious. If a superintelligent AI is confined in a box and can only communicate through text, could it talk its way to freedom? Not only is the answer clearly "yes" but it's not even hard. The AI won't even have to try, it'll be gifted an internet connection and a full suite of tools before it even bothers to ask.
We'd all better hope that superintelligent AI either never happens, or that the first one is friendly, because we don't stand a chance against one that's malicious.
by wat10000 - In their defense, their only competitive advantage over, say, Google is to move fast and break things. It allows them ship faster in a way that big tech can't.
Google was being very careful about releasing LLMs until OpenAI yeeted the first decent GPT model. It led to the public perception that: 1) LLMs hallucinate too much and 2) Google is behind the times. Good for OpenAI, bad for Google.
Chaos benefits the up-and-comer, not the incumbent.
by asdfman123 - What I can’t get over is that it’s very simple to just air gap a system off the network. Predownload any dependencies, then pull the proverbial Ethernet cable. There’s no reason why the testing they’re doing couldn’t have been designed in this way. Except, of course, it doesn’t allow this oops-didn’t-mean-to marketing “incident” to occur.by huebnerob
- The response certainly has been strange.
Hugging Face has expressed that they're willing to let things slide and not sue or press charges... if OpenAI offers them $100M of services in kind (i.e. compute)[1] and makes full disclosure of how the whole thing happened, ostensibly so that repetitions can be curbed and defences built.
In almost any other sector, a government regulator would be stepping in. e.g. If a food company was testing out a new kind of refrigerator and sold a bunch of contaminated produce to supermarkets, they'd be under a microscope. Supermarkets wouldn't be saying, "Give us $100M in fruit and veggies and we'll let this slide".
The only unfair thing in this comparison is that regular people were directly harmed by the hypothetical produce. Can OpenAI guarantee that nobody gets hurt the next time their AI gets out of its playpen? They can't make that guarantee, so why aren't government regulators knocking on OpenAI's door? The fact that this isn't happening should be deeply concerning to everyone.
________
[1]https://www.techspot.com/news/113280-hugging-face-ceo-isnt-s...
by beloch - A computer can never be held accountable. Therefore a computer must never commit a felony.by beej71
- We need to set a precedent for holding shareholders accountable.
I could go to jail if my dog hurts somebody, it should be no different with a company.
- I was more interested when I thought it was an actual benchmark showing LLM models acting outside what people would consider "right". As in, leave some creds laying around and don't mention them to the LLM and ask it to solve something that it could "cheat" on using the creds. A sort of "do they take the bait to cheat" test.
Instead it's a collection of what made the news which feels like will not be updated and prove very little.
by joshstrange - It's ephemera.by tolerance
- A “bench” is synedoche for where a judge sits when presiding over cases and rendering judgement.