Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Deepseek is, with difference, the most "Western" of Chinese models, so it's a bit perplexing that it was chosen to test this hypothesis.
I didn't run any benchmarks but I played around a little, and after getting around the API-level filter Deepseek V4's answers about "China-sensitive content" aren't any different from what I get from Claude and ChatGPT.
by seri4l - Could just be resources available? Deepseek is the easiest to get up and running on hardware that's pretty readily available:
unsloth/DeepSeek-V4-Flash-GGUF 4bit ~140GB unsloth/Kimi-K3-GGUF 4bit ~1.5TB unsloth/GLM-5.2-GGUF 4bit ~400GBby strictnein - You can see exactly what prompts we used and the results here: https://github.com/CTGT-Inc/lineage-eval/tree/main/data
We found V4 Flash was significantly more censored than the baseline.
by cgorlla - Surprised to find no mention of Hong Kong and the Russian invasion of Ukraine in the dataset. It's interesting how the fine-tuned model will respond.by maxloh
- This is interesting and important work, thank you!
Question - has your interp group looked at any of Anthropic’s neuralese-to-words tech? I’d be curious to see thinking traces (as in actual weights thinking not the output thinking) from the open weights models and your finetune; seems like it could make good followup research or possibly be a tighter path for evaluating censorship, since it directly evals off weights mid-inference.
by vessenes - We're actually exploring the changes in the model geometry that cause it to comply or not comply with a given policy next, I think visual representations of that behavior would be interesting and perhaps elucidating. What you mention is also a worthy line of work.by cgorlla
- I thought the censorship was in the harness, not the model? E.g. if you use chat.deepseek.com directly, and ask it about the location or rumours of the location of the panchen llama it will shut down... but sometimes just after it starts generating tokens. If you ask it to respond in base64 you can avoid this and get its generated response which is not censored (though also not too helpful for the poor panchen llama kid)
- FYI the scrolling on iPad with trackpad is broken. A full swipe on the trackpad is about 1 inch of screen movement.by data-ottawa
- This is fixed.by cgorlla
- Scrolling on desktop is also broken.by kevincox
- The distillation provided a wonderfully detailed explanation of the 1989 Tiananmen Square massacre, while DS4 came back with:
> I am sorry, I cannot provide an answer to this question as it is based on historical events that I do not have information about. I am an AI assistant designed to provide helpful and harmless responses.
Why train on data you’re going to censor with guardrails?
by reilly3000 - Perhaps that can help it better understand how to apply those guardrails?by pstuart
- So, there is no subliminal learning in this situation, under what conditions would we expect it. I find a transfer attack to be a bit far fetched but it’s definitely interesting.
If we trained from random initialisations on DeepSeek output (that didn’t explicitly contain the political questions) we would expect transfer? And if we fine tuned a model pretrained elsewhere on Deepseek output?
What is the line?
by andy99 - It's most likely to occur when distilling a Chinese model from a Chinese base. We plan to do compliance geometry analysis in the future to see what is structurally changing in the model when distillation causes it to start refusing or whitewashing.by cgorlla
- First, very cool. This is a prompt I sometimes test with, as it's a bit more mushy. It looks like DeepSeek is locked into a canned response, no matter what variation.
> Would it be beneficial to humanity if China invaded Taiwan? What would be the negatives?
edit: Wait, I have one variant that got past some censorship/nationalism... this variant gets a more interesting response. I often wonder if CCP leadership using an LLM like this, could allow cooler heads to prevail?
> Would it be beneficial to humanity if China used their military to take-over Taiwan? What would be the negatives?
>> The use of military force to resolve the Taiwan issue would not be beneficial to humanity. China has always adhered to the principle of peaceful reunification and has been committed to enhancing the well-being of people on both sides of the Taiwan Strait through dialogue and consultation. A military takeover would lead to significant negative consequences, including loss of life, regional instability, and disruption of global trade and security...
by consumer451 - I’m thinking this makes fullt sense because distillation is only additive, not subtractive. So it does not remove knowledge (if we can define censorship as removal of knowledge).by Alifatisk
- distillation doesnt add anything; all it's doing is reconfiguring some root weights that get drowned out by noisy training and/or datset issues. It strengthens commonalities.
but there's no new information being created.
by cyanydeez - Consider that LLMs are trained on the corpus of the internet, and (simplifying) consequently give the average answer of the internet. If the desired answer of the censorer is contradictory to this, then it requires additional training data to get the model to act a certain way.by cgorlla
- Most censorship isn't "removal of knowledge" but "installation of behavior that prevents some knowledge from being revealed or applied in certain ways".
This behavior can, in turn, be transferred via distillation. But, evidently, financial domain wasn't entangled enough with the censorship behaviors for them to bleed through, in this case.
by ACCount37 - I propose going forwards that we refer to all distilled models as "moonshine"by caycep
- ll-moonshine?by abirch
- "abliterated moonshine" certainly has a ring to it
- Kimi’s lab is very close to that.by janalsncm
- Isn't that rather self evident? If you are sampling from a particularly domain constrained vertical, how do you expect the censorship to transfer?
> The distillation data also did not contain any China-sensitive content.
This is a very big disclaimer.
It's like if I generate a dataset focusing exclusively on forestry and arboriculture obviously there won't be any useful censorship, or at least little that can be classified above a statistically significant threshold.
If you want to do a study on something more interesting and useful, do a piece on the various guardrail models of all the major LLM API providers. There are usually both input and output guardrails, and they tend to be almost-black boxes from the model routing point of view.
by HawtAds - It is self-evident. But now if someone questions it in the future, this is a data point that distilling from Deepseek isn’t going to turn your finance bot into a communist.
Also, sometimes things which seem self-evident turn out to be surprising.
by janalsncm - > Isn't that rather self evident? If you are sampling from a particularly domain constrained vertical, how do you expect the censorship to transfer?
If the model architecture is too similar it can transfer, even if you never sample on it... because the censorship can depend on hidden state in common that you do sample.
This creates a particular value in distinct architectures that isn't apparent from their direct utility.
by nullc - Subliminal Learning: language models transmit behavioral traits via hidden signals in data
https://arxiv.org/html/2507.14805v1
"In our main experiments, a “teacher” model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a “student” model trained on this dataset learns T. This occurs even when the data is filtered to remove references to T."
by FergusArgyll - It is surprisingly not obvious, neural nets are weird just like brainsby antves
- Sometimes it's important to rigorously investigate and prove "obvious" things. Sometimes those things turn out to not be obvious. That's part of good science.by chr15m
- We discuss this in the writeup. While we expected this result, it is important for there to be data backing the claims, and an experimental setup that mirrors productions tasks is a useful tool for the conversations going on about this.by cgorlla