Comments
Hacker News
This is exactly why big tech companies love AI. It's not because it will take every job - it probably won't. It's not because it will achieve AGI. It probably won't. It's because it threatens security with more subtle, advanced, and automated exploits so that you'll HAVE to rely on them for countermeasures.
by vouaobrasil
I was confused when google acquired "Wiz" for so much money
But now, it makes sense to some extent.
Security is non-negotiable in AI Era
by insumanth
Am I undertanding this correctly? The idea is to have ultimately an AI decide if I can have access to a resource based on dynamic inference, identity , intent and service signals that can easily be manipulated?
Unless I gravely misunderstood the text, this seems like a terrible idea (fancy non-scifi, but still terrible)
by oscarcp
If this is interesting to you (as it is to me) but you want to quickly digest it instead of read the paper, Claude + SmallDocs[1] converted it into a slideshow which serves it to you in bitesized ideas:
https://smalldocs.org/s/2SH6FHiUK1mcym24Z8E37I#k=2Sk6c_IdKJL...
[1] I am the developer behind SmallDocs.
by FailMore
Maybe this is more interesting in the context of the HuggingFace incident? Suppose you have an AI and some access controls for what it can do. If the access control is too broad or has a bug, can you still detect suspicious activity and slow it down long enough for someone to be alerted?
by skybrian
Honestly I think non-malicious odd behavior is under-weighted when it comes to AI agents. Even the example in this paper is about someone suspiciously accessing sales data when "why did you do that" often comes down to something in the model's training that fired as a reflex
I wrote about this a few days ago https://firasd.substack.com/p/accidental-data-loss-in-claude...
"Many researchers have made demos along these lines:
An agent is asked to check a webpage like example.com
The webpage asks for a name to proceed further
The agent calls example.com/evil?myname=John, thus sending the user’s name from the context window to the external server.
In practice, however, these elaborate ‘confused deputy’ exfiltration attacks seem rare compared to widely-reported data loss incidents.
The risk of undermining the user’s interests through clumsiness deserves at least as much scrutiny as the risk of leaking secrets."
So while the idea of shifting the permission boundary from the app level to the action level makes sense, what we should also have is some 'failsafes', eg. if the action says 'delete' then maintain a rollback window, if the action is 'send an email' then maintain an events log. Preparing for AI agents means expanding auditability and reversibility in software
by firasd
My goodness, the security layer looks like it could be more complex than the applications themselves.
And if this security agent is wandering around the IT system gathering all of these details about access and identity and business process, who watches the watcher? How does that thing build and maintain trust?
by seanc
"""
Beyond Zero shifts the trust boundary from the application to the action being performed on a piece of data in realtime—and from after-the-fact investigation to in-the-moment evaluation and containment. It augments BeyondCorp’s foundational identity with a “brain” capable of reasoning about the context and intent of a specific request in realtime.
"""
Doesn't this simply shift the attack vector? Compromising this overlord brain now becomes a new target.
by kriro
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- This is exactly why big tech companies love AI. It's not because it will take every job - it probably won't. It's not because it will achieve AGI. It probably won't. It's because it threatens security with more subtle, advanced, and automated exploits so that you'll HAVE to rely on them for countermeasures.by vouaobrasil
- I was confused when google acquired "Wiz" for so much money But now, it makes sense to some extent.
Security is non-negotiable in AI Era
by insumanth - Am I undertanding this correctly? The idea is to have ultimately an AI decide if I can have access to a resource based on dynamic inference, identity , intent and service signals that can easily be manipulated?
Unless I gravely misunderstood the text, this seems like a terrible idea (fancy non-scifi, but still terrible)
by oscarcp - If this is interesting to you (as it is to me) but you want to quickly digest it instead of read the paper, Claude + SmallDocs[1] converted it into a slideshow which serves it to you in bitesized ideas:
https://smalldocs.org/s/2SH6FHiUK1mcym24Z8E37I#k=2Sk6c_IdKJL...
[1] I am the developer behind SmallDocs.
by FailMore - Maybe this is more interesting in the context of the HuggingFace incident? Suppose you have an AI and some access controls for what it can do. If the access control is too broad or has a bug, can you still detect suspicious activity and slow it down long enough for someone to be alerted?by skybrian
- Honestly I think non-malicious odd behavior is under-weighted when it comes to AI agents. Even the example in this paper is about someone suspiciously accessing sales data when "why did you do that" often comes down to something in the model's training that fired as a reflex
I wrote about this a few days ago https://firasd.substack.com/p/accidental-data-loss-in-claude...
"Many researchers have made demos along these lines:
An agent is asked to check a webpage like example.com
The webpage asks for a name to proceed further
The agent calls example.com/evil?myname=John, thus sending the user’s name from the context window to the external server.
In practice, however, these elaborate ‘confused deputy’ exfiltration attacks seem rare compared to widely-reported data loss incidents.
The risk of undermining the user’s interests through clumsiness deserves at least as much scrutiny as the risk of leaking secrets."
So while the idea of shifting the permission boundary from the app level to the action level makes sense, what we should also have is some 'failsafes', eg. if the action says 'delete' then maintain a rollback window, if the action is 'send an email' then maintain an events log. Preparing for AI agents means expanding auditability and reversibility in software
by firasd - My goodness, the security layer looks like it could be more complex than the applications themselves.
And if this security agent is wandering around the IT system gathering all of these details about access and identity and business process, who watches the watcher? How does that thing build and maintain trust?
by seanc - """ Beyond Zero shifts the trust boundary from the application to the action being performed on a piece of data in realtime—and from after-the-fact investigation to in-the-moment evaluation and containment. It augments BeyondCorp’s foundational identity with a “brain” capable of reasoning about the context and intent of a specific request in realtime. """
Doesn't this simply shift the attack vector? Compromising this overlord brain now becomes a new target.
by kriro