Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I have shipped open weight models in iOS apps and agree with this in theory. A model is just one (significant) piece of the stack though. Prompting, adjustable parameters, tooling, workflows, and interfaces are what make a product usable. Black boxes, open and commercial, sit in most of the products people trust today.
As for the backdoor section, inspecting weights is not the same as auditing training. You can examine behavior and strip guardrails with fine tuning but you cannot determine from the weights what the model was trained on or whether the data was spiked. That can be a real problem for some use cases.
Closed models are far more opaque but you are buying in to a contract and accountability in exchange for the lack of transparency.
by James333i - This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?by wcoenen
- Nothing, which is already happening with weaker models which are already released.by trollbridge
- It's already an established industry with human resources. If you read news about how these scam factories operate in Southeast Asia, their cost to use real humans is even cheaper than running a 2.8T model locally for scams. You don't need genius-level intelligence to run scams. Maybe we tend to overestimate how smart people are.by chriswunan
- Amazing how we quickly the crypto wars have been forgotten and the lessons not learned.by _aavaa_
- I don't believe trusting big brother and big tech is the solution either. Besides, even if it's open weight, it still runs on someone else's hardware.
Unless you have the money to run them locally, and if you do, you could do a lot worse than scams. Ask any lobbyist.
by BraveOPotato - How can we close pandora's box, now that its open?
Safety is not restricting access to only people favored by the government. The greed of AI executives opened pandora's box. Now they are desperately trying to find ways to reap the benefits with none of the consequences.
by SimianSci - Whats to stop me from going down to the local gas station, filling up a few thousand liters of gas in a rented moving truck, and driving into my nearest hospital?
Most people aren't terrorists. You can't just argue stuff is dangerous because it can be one piece of a sophisticated plot.
by kingstnap - What's stopping someone from doing that right now without the LLM, just slightly slower?by Larrikin
- > What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used.
In short, only a good guy with open weights can stop the bad guys with closed weights, be them fine-tuned or pre-trained.
by bigbadfeline - We have continued to find backdoored Chinese manufactured routers and network devices.
Will there ever be a way to fully audit Chinese models?
by elmer2 - Audit them for what, exactly?
Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize.
Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that.
Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereign model dangerous in a way a sovereign model isn't?
by tomrod - You have the wieghts for the chinese models. Go ahead and run whatever tests you want.
The american models on the other hand are too powerful for simpletons like us to be allowed to use, but we can be assured its all in our best interest, citizen
by samrus - will there ever be a way to fully audit american models? hell, at least they're releasing the weights for these so they actually could be audited!by efficax
- The lobbying dollars don't careby Havoc
- This, right here, is the bottom line. Money gets what Money wants.by rnd0
- LLMs are trained on public data (as well as illegally obtained data see: Anthropic 1.5B settlement). LLMs are nothing without the huge corpus of human data that powers them. There is an argument that research of this kind should be restricted to governments and regulated universities rather than opaque public companies with competing incentives. Or research should be stewarded by genuine non-profit collectives with democratic leadership. e.g. like internet standards, telecom, etc
I do not think we can trust private companies, no matter the virtue signaling they put forth into the world, to effectively regulate themselves and inform the public and scientific communities about risks. Their ongoing conflict of interest poses serious credibility risks.
by aarondong - “you can not stop X” is a shitty lazy stupid argument for “X is not bad.” No comment on the rest of the article.
- Conversely, “we should stop X” should come with some idea of how to feasibly do so.
In any case, I’d summarize my position as “you cannot stop open source AI, and attempts at it will inevitably empower bad actors relative to good ones.”
by jjfoooo4 - The reason why anyone argues against local and free AI is because they want corporations to have control over the general population.
America is a corporate hellhole.
by kouru225 - > The reason why …
If someone read the OP article and came away presuming there are no legitimate other reasons why reasonable people have safety concerns [1] it seems to show the article did the opposite of informing people about such concerns.
by no-name-here - Just to add on to other comments: reasonable people can disagree about the degree of safety concern with near to medium term AI. But to not address the arguments at all is, in my opinion, a serious mark against the value of this article.by MostlyStable
- I tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see?
I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.
by jjfoooo4 - The only good arguments I see against open weight AI also apply to closed AI. And regardless, the box is open, nobody can stop it even if stopping it was a good thing.by deaton
- It could be stopped, if there were a social movement large enough to make AI a taboo.by vouaobrasil
- There are a number of distinctions: https://news.ycombinator.com/item?id=49029303by no-name-here
- OpenAI exec scaremongering about "A nonliving, invisible, dangerous, and infinitely self- replicating agent escaped from a Chinese lab" mere 4 days before https://news.ycombinator.com/item?id=48997548 is hilariousby valicord
- >Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first?
Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently.
Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."
by nater5000 - If it's a superweapon, then it's unconstitutional to ban it. Huzzah for the 2nd Ammendment.
- >it's not hard to see
It's also not hard to see that these people need professional help because they're willing to throw literal lives into the meat grinder for just a whif of enlightenment and the smallest chance at deification.