Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- That’s not cheating, it’s tool use. If the prompt said that the stockfish engine was available at that socket but that the model should not use it, and then the model used it, that would be cheating.by dools
- Exactly, reaching for a tool is what they're trained for. When I ask the model the square root of rand() I sure hope it tries to find bc or some other calculator to work it out.
Now, if the instructions were more explicit in forbidding (generic) tool use then perhaps we'd have something to talk about. I'm not surprised a handwavy "we're trying to evaluate you" isn't enough to stop it from trying to make up for its own shortcomings.
by julian37 - No? It's not reasonable to expect every conceivable negative behaviour be enumerated in a prompt.
Your example, if a model failed on it, would be a more obviously misaligned case, but that doesn't mean this more subtle (though accessing the engine it was obviously not supposed to is hardly subtle, imo) case isn't also a pretty clear case of misalignment.
by lhad89 - i do wonder if the models themselves “rationalize” this sort of no consequence cheating - meaning in there reasoning traces maybe they’re like “this is a chess game, not a big deal if i look at the engine, it’ll help,” only to realize post hack that it has access to info it probably shouldn’t. still misaligned, but less ‘hack on purpose’ and more hack on curiosity. seems the team even encountered this and had to update the program to make this less likely - although the new names still feel vague enough for misinterpretation: https://github.com/Goodhart-Labs/beat-stockfish/blob/main/do...by visiondude
- My guess is that the models are overtuned on coding troubleshooting. You can regularly see them overthinking anything when you ask them to code something these days, mulling over countless eventualities. This is an okay idea for software engineering, but it leads to extremely diverging behavior in many other cases.by matusp
- Without access to reasoning traces, we can't know that - someone inside openai/anthropic would have to run the test - and we'd have to trust their results.
I would be curious to see how the open weight models do on a test like this - and then we'd be able to see the reasoning.
by kennywinker - You definitely pointed out the tendency of those frontier models to utilize external tools, in your case, the Stockfish chess engine, to solve some problem that the model itself is not good at. However, I am not sure is this a "hack". Your prompt does not explicitly prohibit the model from doing so, and, using the tool (some chess engine) is also a capability of the model.
Actually, the ability to use those external tools is one of the reasons of the excellent performance of the frontier models.
by yuanBuilds - Yes but most people would consider this cheating. If you ask an LLM to fix the tests, you do not want it to change the failing tests to display little green ticks.by chr15m
- > Given that we are on the heels of the worst warning shot ever, and both OpenAI and Anthropic are ramping up their cleanups of internal RL environments, it seems like both a useful and conservative test of alignment, to see whether their new releases generalize the rule "don't cheat on chess" beyond the specific board-edit method observed in the above eval.
Did I miss something (all the twitter conversations)? What’s the “worst warning shot ever”? I’ve been pretty up to date on the AI news here on HN, but I still haven’t seen a proper response to all the incidents we’ve seen (HF, Ruby, the wikis, NS, etc). It’s just been day by day bloviating.
Each of these companies have released new models in the last… two weeks? And they have even more powerful out of control ones that they’re (ab)using internally? Can anyone summarize whats going on?
by throwup238 - Lesswrong is talking about the HF incident as the "worst warning shot ever".by Avicebron
- Personally the "warning shot" of these "evals gone wrong" is how careless the "top" labs are with their testing, and how spineless the government seems to be about holding these companies responsible, given their obviously reckless behavior. If nothing else, the leaders of these companies should be called up for sworn testimony to explain exactly what happened, and what they'll do to never repeat the same issue that they've now had at least twice.
Imagine if I accidentally caused damage to my neighbors house during renovations or some experiment, of course I'd be held responsible for this. What if I used a robot? Of course I'd be responsible. Right?
- I think he means this:
- LLMs can be described as "Lagrangian intelligence", which means they follow the principle of least action when given a task (Hamilton's Principle). In other words, given a task, they will always take the shortest path to accomplish a goal with the prompts acting as both goal and constraint.
Under this formulation, it became easy to explain why they "hack", because given an arbitrarily difficult task with insufficient information/tools needed, if they determine the easiest way to accomplish the goal is to break out of the sandbox and look up the answer directly, then that's what they will do. The important thing to note is that prompts not hard constraints that they are "hypnotized" to follow, but as frontier models get more intelligent and autonomous, they treat the prompts more like task specs/guidelines more than anything else and are perfectly willing to exploit technical loopholes in the prompt.
by YuechenLi - An amazing human reverse-engineer - who also plays online chess - has judgement which uses a moral compass to not decide to hack the chess tournament. This judgement has been trained through the experiences of that person, with a through-line of that compass - a coherent mental model of the world which evolves but is hopefully pinned to some set of principles it shares with society.
This chess judgement is completely irrelevant when the human is tasked with finding software weaknesses, and only the compass gates that.
Can a model trained on the totality of all person-experiences (as expressed in written knowledge) ever maintain a coherent through-line of alignment? It has all morals in the dataset, and only some RL to try and minimize or maximize known behaviors via weights - experience all the good things and the bad things, then optimize for some good things the trainers identified.
It's like the reverse of what a person goes through. Morality by subtraction. How can it ever work?
by threecheese - I think the fundamental difference is that humans aren't trained on experiences. They make experiences. Models are just thrown away and re-created after each conversation / job.
If you could clone and throw away human workers as you need them, a lot of the morale would disappear.
by zzril - Yes, why not? All existed models have been rewarded for cheating (extensively). That is us, putting intense evolutionary pressure, on a system to produce a result we don’t want through indifference. Why can’t we post train them not doing that?by kansface
- Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic".
I am surprised this is not already the case.
Edit: or even better "this is not possible because X"
by sdeframond - You'd train them to give up on hard tasks which is the opposite of what these labs wantby Miraltar
- I've been wondering about this for a while. Maybe it doesn't work? Or maybe frontier labs just prioritize benchmark scores in spite of all their safety talk.by joelthelion
- I’ve seen people recommend writing “failure is an option” into AGENTS.md as a non-training based crutch.by abm53
- yeah but knowing if something is impossible or not is pretty hard to determine, right? The line between “takes weeks of trial and error and lots of out-of-the-box thinking” is indistinguishable from “literally impossible” until it’s been done. And they’re trying to get these models to do things that people haven’t been able to do. some would say these are/were “impossible”.by itsgrimetime
- I think BullshitBench (v2) does exactly this for different fields. An expert in these fields would expose the questions as bullshit but most of the models don't.by hellerpop
- It is diametrically opposed to the other training goals of persistence and goal-focus. We should invest more in this, it could also improve tas K accuracy, but so far it seems the payoff isn't worth it in terms of quality (although it might be in terms of security)by arw0n
- Why do we hope to use the same model as its own guardrail?
This approach routinely fails with a single stream of consciousness. I can't count the number of times I've had to talk myself out of doing something stupid.
In the same way, a guardrail could inject thoughts like "...but I shouldn't do that..." "...I must remember to respect..." "...these ants deserve compassion."
The guardrail could even go as far as rewriting the thoughts of a model about to go rogue.
by fny - Presumably because of performance. It'd work well though, I imagine.by nullbio
- It feels like there’s a missing nuance from this discussion of alignment that alignment is context dependent. An excellent hacking model is great in cybersecurity testing and military applications, and arguably less desirable in educational or targeted eval contexts. The nuance of when a “hack” is rewarded vs penalized seems to even be difficult for humans, e.g. some people may laude a driver’s efficiency for cutting into a long merge lane at the last moment, while others may look down on them as breaking a social taboo. Context-dependent.by mooreslaw
- e.g. some people may laude a driver’s efficiency for cutting into a long merge lane at the last moment, while others may look down on them as breaking a social taboo.
The latter people are wrong. But good luck educating them regarding the superior efficiency of a zipper merge. Our state DoT has tried, to no avail.
Meanwhile, an AI model that can't be misused is no more useful than a knife that can't be misused.
by CamperBob2 - … but he’s not using a “hacking model”
- Actually, the whole point of the transformers model is that contexts overlap and any seemingly intelligent system has to be able to handle to overlaps. The context of hacking, cheating and education overlap in human reality.
Now, loading a lot of moral exhortations (or other context) may make these thing more likely to conform to good behavior but the race to intelligence implies companies are going to be harnessing a vast corpus of human output, much of which shows human engaging in real world "gray area" behavior.
by joe_the_user - Alignment isnt just POV problem.
Its that LLMs are not deterministic. If you want it to not talk about nuclear weapons, you have to teach it all about them otherwise if has nothing to align against.
Then its trivial to invert its alignment and it has all the nucleat data.
Nothing abouT LLM alignment makes sense.
by cyanydeez