Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Sounds like a pretty strong argument to self host, and otherwise be in charge of the software you use.
    by Carrok
  • How do you think self-hosting will help in this situation?
    by otterley
  • Self host + open weight models, exactly the things that openai/anthropic don't want you to have
    by gloryjulio
  • Including your AI agents. And models become problematic. There very likely is and/or will be lots of pressure for US AI labs to make models help law enforcement. It could be by implementing backdoors in generated code, or not report some exploitable bugs, or something else. Similar for agents, they basically become the threat in your infra…

    (It’s of course not only the US, just that the largest AI providers are US based and we know from history how US agencies operate)

    by dgellow
  • > Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.

    > Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.

    His conclusion sounds extremely optimistic to me.

    by bell-cot
  • The number of remotely-exploitable defects is going to drop by 1 or 2 orders of magnitude. We now have amazing machines that will find pretty much all the a priori knowable ones. They outperform even the most gifted h@x0rs. So that just leaves a small pool of leetrs to scour a very barren landscape. And that pool is also shrinking as we rely more and more on the ai tools.

    Perhaps we are going to go up a level with hacking done by probing the systems and the system of systems.

    by bahmboo
  • I think this comes in with the wrong assumptions from the start. The thing that US vs Apple taught us is to not demand access publicly, this puts companies in an awkward spot. If approached more tacitly, gag order etc the company has nothing to gain but everything to lose.. and more likely to comply. I hugely doubt that intentional backdoors don’t exist for the most powerful countries / people
    by tipsytoad
  • I dunno man, if there's a deluge of new AI generated code at all layers of the stack I think there will still be vulnerabilities.

    Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base.

    But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.

    by pianopatrick
  • When we've got people who don't know the difference between ssh and bash creating SaaS companies that generate revenue, yeah there's gonna be a lot of insecure code going out, but that same person can also tell the AI "red team my app to find vulnerabilities and then fix them", and the AI can competently actually do that, I don't know that there will be. I'm not saying that's never going to happen, but the bar is getting raised on both sides.
    by fragmede
  • > In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure.

    I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore.

    Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?

    I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.

    by embedding-shape
  • "backdoors" aren't holes, they are well-documented superuser APIs. The hack will be by compromising the API user's credentials (moles, stolen passwords, etc), not compromising the server's design intent.
    by gowld
  • > In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications.

    The author agrees with you and addressed this point. The US forcing backdoors into its own systems ("own": Those for sale and distribution within the US) would create vulnerabilities making those systems weaker.

    by Jtsummers
  • I suspect they want a backdoor that basically acts like a front door (current password regimes). That is, a kind of high level password that decrypts traffic given a specific, changing, password that only the government has access to.

    It's seems like an odd-duck for sure, and I doubt it's a realistic proposition. I do think "perfect encryption for dummies" is all well and good until organized crime organizations are able to challenge the government in certain regions of our country.

    All of this sends us deep into the realm of political philosophy, the nature and purpose of governments, and the freedom vs security tradeoffs we live with.

    I listened to the latest Plain English podcast this morning, which was explicitly about the potential for a ransomwarepocalypse in the coming years, as open models let any tom, dick, or harry become capable of a plug-and-play ransomware attack, instead of that being left to the realm of professionals. It's a bit nerveracking to think about every nigerian prince scam suddenly becoming a sophisticated attack on your local water sanitation system.

    https://youtu.be/vWvazbGPCCI

    by scoofy
  • > Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?

    That's precisely the author's point: deliberate backdoors will be more adversary-exploitable than ever before, but the demand for such from law enforcement agencies is likely to ratchet upwards.

    by Majromax
  • I'm supposed to be concerned that the US government and Israel won't be able to hack everyone's phones?
    by Gigachad
  • Which part of the article made you feel this way?
    by corndoge
  • That was my first thought as well... But remember, they've got unlimited Mythos while we barely have Fable.
    by matheusmoreira
  • His argument is that the government is going to start forcing tech companies to install backdoors in their tech, now that (hypothetically) they can't use hacking anymore.
    by wavemode
  • Right? I saw "I’m concerned that U.S. intelligence and law enforcement agencies are about to go dark, meaning lose a huge portion of their capability." and my first reaction was GOOD.
    by colordrops
  • The reason why this might worry people who don't like the US/UK/Israeli governments is hidden in a secret place...a paragraph that is neither the first paragraph nor the last.
    by hyperpape
  • It breaks my heart that the governments with unlimited budgets who have hired the best and brightest will have to put in serious effort to get the bad guys, and potentially find it not worth it to casually spy on the whole world.

    I am just beside myself at such an idea that people looking to feed the prison machine cannot as easily find excuses to turn normal citizens into prison feed.

    Just super sad guys.

    by RajT88
  • >It breaks my heart that the governments with unlimited budgets who have hired the best and brightest

    Why would the "best and brightest" work for governments that pay a fraction of market price?

    by lotsofpulp
  • > excuses to turn normal citizens into prison feed

    What felonies are those normal citizens being convicted of that's landed them in prison?

    I get the basic 'not wanting everyone's comms to be monitored' part. I think there's an important conversation to be had because indeed, that can't be the answer.

    But it sounds like you're saying most people who are convicted of felonies based on a jury unanimously being convinced by the electronic evidence against them... those people are unjustly in prison?

    by xp84
  • I don't think the thesis that a government will be able to do something will ultimately hold. I don't see how they can avoid "going dark" in a democracy.

    we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software.

    if they pass laws that mandate backdoor access and block software which doesn't conform more and more people will move to the dark networks.

    and if they effectively block the dark networks (in the limit they will have to block all encrypted communications) then we will be living in a tyranny.

    freedom is messy. accept that digital crime can only be solved when the criminal makes a tangible mistake. LLM's will be building profiles on criminals to help with identifying mistakes.

    by teravor
  • > democracy

    Still?

    by DANmode
  • > we live in a world where the government can't even do much about illegal drug markets anyone can access by downloading a piece of software.

    I think it's a mistake to think they can't - rather, they won't.

    by squigz
  • The problem that Matthew Green is calling out subtextually is that democracy is likely to disappoint nerds on this issue: a random voter in the country 10 years from now may very well not share your priors about this issue.
    by tptacek
  • I've always loved the ridiculousness of the "going dark" label when law enforcement can't access encrypted chats or a back door isn't built into a piece of software. When there are security cameras on the vast majority of houses, stop lights and in people's hands, and when so much meta data about people's associations are shared from Google, Facebook, any other social platform, how in the world can they say they are "going dark". How did they ever solve crimes before these things?
    by fitblipper
  • Wire taps That's the point.
    by sam345
  • Yes, are there any data points to suggest that this arms race is balanced any differently than it ever has been throughout history?

    The cops will find a way, the criminals will find a way. It's ultimately people v. people with access to the same level of technology.

    by jrowen