Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed.
If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then severity may come into place - all 6 are rated low impact)
by bluGill - I guess this would also require models trained on pre-2023 data - or not trained on later curl code, changelogs, blog posts discussing curl security fixes, etc.by thih9
- "How many total vulnerabilities can your tool alone identify?" and "How many unique vulnerabilities can your tool identify?" are both valid comparisons to make IMO.by zamadatix
- I think you might be misunderstanding this? This is, from my understanding, what went down:
1. curl was scanned by many different things, including AISLE, and many bugs were fixed <- all this was in the past 2. curl a week ago was scanned again my Mythos and Codex Security, and both of them said: 0 issues found 3. the same curl was scanned by AISLE a day later, resulting in ~29 reports (based on the blog post and mastodon posts from Daniel Stenberg) 4. of these 29, 6 cleared the bar and got CVEs in curl 5. these 6 CVEs were just announced as fixed in curl 8.22.0 today, together with 4 more CVEs that were detected by other people prior to point 2. of this list
so imho it was head-to-head, the very same codebase => it's a legit comparison
by goobreee - The gauntlet has been thrown. Will Anthropic or OpenAI pick it up?by jmartrican
- I like the looks of Aisle and what they stand for...
That being said you cannot compare a model with a specialised harness. These are two completely different things.
Am I missing something?
by _pdp_ - i think a key missing part is that an LLM on its own can't find vulnerabilities, so it's always an AI + harness. even mythos, when used for finding zero-days, is using an actually surprisingly heavy handed and expensive scaffold. they literally make it run in parallel on ±all files and ask "what's wrong with this?". here from the mythos technical blog post [1]:
> To increase efficiency, instead of processing literally every file for each software project that we evaluate, we first ask Claude to rank how likely each file in the project is to have interesting bugs on a scale of 1 to 5. A file ranked “1” has nothing at all that could contain a vulnerability (for instance, it might just define some constants). Conversely, a file ranked “5” might take raw data from the Internet and parse it, or it might handle user authentication. We start Claude on the files most likely to have bugs and go down the list in order of priority.
So they process it in parallel, but AI-rank them based on vuln-likelihood first = exhaustive search with a heuristic filtering first
by goobreee - Given enough AIballs all bugs are shallowby dec0dedab0de
- I also have some secret recipe finding one class of bugs: https://github.com/tmux/tmux/issues?q=is%3Apr%20author%3Ablm...
curious they're willing to run AISLE on tmux to find more than mine.
by blmarket - Good marketing and definitive proof that local (read: on-prem & air-gapped) models with correct context and tools are good enough to perform on par and above SOTA cloud hosted solutions.
We have seen this point many times before with different technologies. The first computers at university were big and expensive, same as this machine. Give it a few years and this functionality will be a commodity.
by TechTechTech - Very unrelated to the content of the article, but that is a pretty weird ft ligature in the heading. It looks a letter from another alphabet. Which maybe makes this pretty cool after all.by janaagaard
- It's a very unfortunate ligature imho because there is no distinction between f and t. Looks funky
PS: the image in the background looks like a nod to David vs Goliath, love that detail
by RamblingCTO - That's bragging rights correctly earned, i think! As marketing-y as this post is, definitely something to keep an eye on.by anilgulecha
- Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)
- As far as I understand it, the other efforts have not reported most of their findings to upstream developers, focusing on critical findings only.
This is understandable because upstream interactions at scale are difficult.
by fweimer - i don't think this is correct. if you look at this article by the curl founder daniel stenberg (https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-v...), he talks about how he previously ran Mythos on curl and that it found 5 issues: 1 turned out to be a low severity CVE, 3 were false positives, and 1 just a bug. So a) Mythos detects low severity CVEs too, and b) it is fairly noisyby goobreee
- Curl seems to becoming one of the favourite things to demo AI finding vulns.
Curl is going to end up incredibly secure.
by graemep - Anyone care to guesstimate how much effort there would be in creating an actually-secure curl and openssl? Using something like Common Lisp or Lean, instead of C.by floxy
- Thank goodness because curl is a load bearing structure to the backend of the internet.by pixl97
- Curl has a well earned reputation for high quality code. If you find something there it means you are good. There is a lot of software where finding a vulnerability mostly means you bothered to look and are not completely stupid. Nobody is going to be impressed if you find an issue with something that everybody already knows is poorly coded.by bluGill
- We had a few AISLE-generated security reports, and the signal to noise was reasonably good.
The most notable bug/exploit their scanner found was: https://gitlab.com/nbdkit/libnbd/-/commit/e50bbd2681117c2dd8...
The tool basically had to chain two exploits together to reach this. It also came up with a patch to fix which was fairly sensible (but I ended up editing it further for clarity).
by rwmj - Wow, this announcement is good content marketing.
Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.
It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.
> We then ran AISLE's autonomous AI system against curl.
They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.
by melvinroest - It defaults to gpt5.4 nano
https://github.com/weareaisle/nano-analyzer/blob/main/scan.p...
by vorticalbox - > what models the use under the hood
Presumably their own, wouldn’t they?
by drdrd - >All six are rated Low severity
This says it all. Nothing important was missing. This is marketing hype.
by 1970-01-01 - > Wow, this announcement is good content marketing.
Why do you think companies hire PMMs?
by alephnerd - It wouldn’t surprise me if AISLE uses many different providers’ models, and what’s holding back OpenAI and Anthropic is only using first-party models. Just because OpenAI and Anthropic have arguably the strongest models overall doesn’t mean their models are the strongest at finding any given class of vulnerability or lead to follow.by wky
- Their system can run with various models, they go into more details in this article.
https://aisle.com/blog/system-over-model-zero-day-discovery-...
by whizzter - The homepage says something about AI guided fuzzing based on libfuzzer or AFL. Looks like they have the LLMs identify a bunch of interesting functions to test, generate some test harnesses, and then sort through the fuzzer findings at a high level, which sounds like a pretty good idea.by tux3