Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It really is a shame that funding cuts at NIST has led directly to this. I don't know if there is an incentive for a non-profit to come out of the woods to do the validation and enrichment process, but boy do we need it.
Like gortok has said, LLMs are just probabilistic text-prediction engines. They cannot automate the vulnerability validation lifecycle because they cannot account for improbable edge cases. They may not even get the version of the software right...
by umarcyber - Funding cuts at NIST did not lead directly to this. You must be new here or haven’t spent a lot of time directly interfaced with NIST on cybersecurity in decades past.
And NIST’s role with CVE’s has always been purely ministerial/clerical. It has never been their job to do even a cursory investigation into the vulnerability itself.
by wildfireday2 - I think this is expected with the increased usage of AI on both fronts (finding vulns and automating their remediation). Yes, there are and will be companies that help other companies validate CVEs before they raise any alarms, but that noise/signal battle has always been there.
I work as a sec eng at a biotech firm and have a tool I built that pulls our ITIL and runs a search on NVD, GHSA, etc. for enrichment and classification. But the number of CVEs I see for products we house is astronomical compared to the actual version of that product we are running (and that's only one value). But that's also a challenge when working with so many sources and having a small team try to tackle them.
The age-old validate, remediate, verify proves itself useful once again!
by ghostnshells - They couldn't even be bothered to use a non-AI image.by BigTTYGothGF
- that's fine, I'm not sure they bothered to actually write the article eitherby lexicality
- Could be just me but I would be quite careful running these PoC's, especially since the CVE's mostly seem nonsensical. I suppose these CVE's could be targeting some kind of PoC execution instead of actual bug reporting? (I didn't look at the code in depth).
Why is the repo even mixing CVE's for "schreibfaul1 ESP32-audioI2S" and "SQLite"? Is mixing CVE's for different products in one repo common practice?
by trashb - There is no specific practice for a repo full of bug reports. Having a repo full of bug reports is already unusual. Normally you report a bug against the software that has a bug, not by creating a file in your own repo.by inigyou
- > The cited code didn't even exist in those versions or referenced unrelated logic. > When testing the PoC payloads they didn’t work (not triggering any crash).
I think the future is pretty obvious, if this isn't being done on projects already: you need to automate these checks and reject automatically
by gste - Great idea! We can use LLM to do this.
LLM: I ran the check and it repro'd
> Did you really?
LLM: You're absolutely right. I didn't actually run the check. Good catch! One sec let me do that now... yep I ran the check and it definitely repro'd this time.
> I hate my job
by tantalor - > Because no step in today's system actually requires a proof-of-concept or bug reproduction, a plausible-sounding fake advisory can slide right through the pipeline and end up in GHSA, downstream databases, and enterprise scanners.
I have no experience in the realm of security, but why isn't the submission process similar to any normal software shop (big or small) in terms of required bug repro steps?
by rib3ye - Somewhat related: https://sqlite.org/cves.htmlby Spide_r
- Damn, dunno why I've always assumed that if something gets a CVE, then some authority has reproduced and verified it.
Shouldn't it be CNA's job? Why would anything get a number if it hadn't been verified?
by oxydite - It's really just a number allocation and that's all. You can get a number. That doesn't mean the number refers to something useful.by inigyou
- The current system is radically overwhelmed and has been seeing cuts over the last few years.
> then some authority has reproduced and verified it.
No, the system puts a lot of trust into the submitter.
by insanitybit - I almost feel that we have a new generation of the "script-kiddie". People without much (or any?) knowledge of software engineering using outside tooling to do "stuff" they wouldn't be able to do themselves.
Maybe it's a not a perfect analogy - in this case the intentions seemed to be laudable - but we will see much more of this, including from malicious actors.
by linuxhansl - Not validating submissions seems like avenue for massive attack. Flood the whole system with endless false reports. Thus making it significantly less reliable.by Ekaros
- It's been a problem for awhile. Daniel Stenberg has talked about it numerous times on his/curl's blog for the last 4 years. They became their own CNA to try and control it, they opened a hackerone with rewards, but now removed the rewards because it got flooded with AI generated slop daily.
https://daniel.haxx.se/blog/2023/08/26/cve-2020-19909-is-eve... https://daniel.haxx.se/blog/2024/01/16/curl-is-a-cna/ https://daniel.haxx.se/blog/2025/04/24/how-the-cna-thing-is-...
by delfinom - This is what the Linux kernel is currently attempting since becoming a CNA. edit: "This" being to sabotage the CVE system via DoS, to be clear.by insanitybit
- That is exactly why many big projects are migrating to becoming CNA, so that randos can’t get assigned unqualified CVEs which nobody has looked at or validated.
Apparently RedHat is a CNA of last resort, so it might be possible to get your project under Redhat’s scope and go through them without having to be a CNA yourself.
by masklinn - The problem with this kind of thing, is that it reduces the S/N (Signal-to-Noise) ratio, so weeding out the legit CVEs becomes a lot more difficult.
But, on the other hand, I do know that LLMs have been discovering a lot of legit CVEs, and I will lay odds that the blackhats are leveraging them to the max.
- This is an issue with CVEs not LLMsby zuzululu
- I think in both cases, the LLM use needs to be handled by someone experienced with the domain. We are seeing this over and over with product development, writing, images, etc. Without good people driving this it's just noiseby gedy
- > I do know that LLMs have been discovering a lot of legit CVEs,
And that's great! Now the human in the loop can verify the accuracy of each finding and take responsibility for moving it forward.
> and I will lay odds that the blackhats are leveraging them to the max.
We just need more humans in the loop to do that more quickly, each putting their own reputation on the line with each CVE they determine to be legit.
AI doesn't stand for "accountability irrelevant", even though most people act like it does.
- Basically nothing is real nowadays unless you verify it for yourself.by hoppp
- CVEs sucked even before LLMs. CVSS is an ass-covering exercise, not an importance signal. Everything is "critical" all the time.by pornel
- Which is precisely why I do not celebrate when Microsoft or other companies say they released "their biggest set of security vulnerability fixes ever" because there's a good chance that most of them are hallucinations or non-critical issues erroneously considered zero-days or critical.
What actually needs to be done is human review, vetting, and approval of those. Otherwise, it's just unknown/unnecessary/potentially dangerous changesets in high quantities.
- I'm sending all my condolences out to everyone who is in a corporate environment using tools like Veracode or Snyk to automatically assess for CVEs and block merges or deployments based on "findings".
My experience in such environments leads me to believe this is going to be a rough ride for those heavily locked-down enterprises, because depending on the environment, an exception of "this CVE was hallucinated by AI" is probably going to be difficult to get accepted, and when it does, starts to become its own avenue for exploitation and adds even more noise and confusion to the mix.
by jodacola - The vast majority of CVEs are not exploitable, basically noise. I suspect that the overwhelming majority of the CVEs being generated by LLMs are either noise of the sort in the linked article or noise of the sort that is not exploitable.by flerchin
- We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are.
LLM-based “AI” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your LLM-based “AI” gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence. What bothers me about this entire situation is that presumably the folks that relied on the LLM-based “AI” to generate these vulnerabilities knew (or should have known) enough about their tool to know this would happen, but did not.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.
by gortok - Wow you just got us a complete nostalgia moment to the good old times when the computer who always beats us at chess became `not artificial intelligence`...by polotics