I'm curious to know what people think about Anthropic's claim that Claude found 500 high-severity vulnerabilities in open-source packages.
-
-
I'm curious to know what people think about Anthropic's claim that Claude found 500 high-severity vulnerabilities in open-source packages. Has anyone confirmed that these vulns were indeed high-severity and hadn't been discovered before? Is this development as big a deal as Anthropic says? Any other critiques?
@dangoodin Daniel Steinberg mentioned on FOSDEM 2026 - full covered test suite is the wall none of "AI" could climb. I guess npm may provide even more vulnerable packages 987654321
-
@GossiTheDog @dangoodin
For #3 there are a bunch of recent commits to the lzw code.These really seem like bugs that existing scanners should have found, especially strcat use (#2).
-
I'm curious to know what people think about Anthropic's claim that Claude found 500 high-severity vulnerabilities in open-source packages. Has anyone confirmed that these vulns were indeed high-severity and hadn't been discovered before? Is this development as big a deal as Anthropic says? Any other critiques?
@dangoodin Gonna go out on a limb here and posit that Claude Code is creating vulns at a much faster rate
-
I'm curious to know what people think about Anthropic's claim that Claude found 500 high-severity vulnerabilities in open-source packages. Has anyone confirmed that these vulns were indeed high-severity and hadn't been discovered before? Is this development as big a deal as Anthropic says? Any other critiques?
@dangoodin I said it elsewhere, but what's missing in my view is the false positive rate. Ok, it found 500. Did it flag 500? 5,000? 5,000,000? That's an important data point.
-
@dangoodin if "this commit changed a thing to fix a bug" is the metric, well fuck, I've found over 100,000 'vulnerabilities' in the past year.
That's not what Antropic said. Antropic said the vulns were high-severity.
-
That's not what Antropic said. Antropic said the vulns were high-severity.
@dangoodin that is EXACTLY what Anthropic said. LITERALLY it is the FIRST "vulnerability" they bogusly claim to have found.
> Neither of these methods yielded any significant findings. Eventually, however, Claude took a different approach: reading the Git commit history. Claude quickly found a security-relevant commit, and commented:
-
I'm curious to know what people think about Anthropic's claim that Claude found 500 high-severity vulnerabilities in open-source packages. Has anyone confirmed that these vulns were indeed high-severity and hadn't been discovered before? Is this development as big a deal as Anthropic says? Any other critiques?
Thanks for all the responses. So far, projects I understand to have received reports include: Ghostscript, OpenSC, lzw, and CGIF. Are others known? Links to commits that fix the vulns also appreciated.
-
@dangoodin that is EXACTLY what Anthropic said. LITERALLY it is the FIRST "vulnerability" they bogusly claim to have found.
> Neither of these methods yielded any significant findings. Eventually, however, Claude took a different approach: reading the Git commit history. Claude quickly found a security-relevant commit, and commented:
Right, but the post doesn't say merely that the reports of the 500 vulns resulted in commits. It says all 500 were high-severity. If true, that would be significant, no?
-
@dangoodin that is EXACTLY what Anthropic said. LITERALLY it is the FIRST "vulnerability" they bogusly claim to have found.
> Neither of these methods yielded any significant findings. Eventually, however, Claude took a different approach: reading the Git commit history. Claude quickly found a security-relevant commit, and commented:
@dangoodin to which I said "hang the fuck on" and read a bit more. And hey look, it's in fonts... bounds checking...
https://security.snyk.io/vuln/SNYK-CENTOS10-GHOSTSCRIPTTOOLSFONTS-10299121
-
@dangoodin to which I said "hang the fuck on" and read a bit more. And hey look, it's in fonts... bounds checking...
https://security.snyk.io/vuln/SNYK-CENTOS10-GHOSTSCRIPTTOOLSFONTS-10299121
CVSS is 7.8, which is high, no? That would seem to support the Anthropic's claim. What's the significance of the vulns being in fonts . . . bounds checking?
-
I'm curious to know what people think about Anthropic's claim that Claude found 500 high-severity vulnerabilities in open-source packages. Has anyone confirmed that these vulns were indeed high-severity and hadn't been discovered before? Is this development as big a deal as Anthropic says? Any other critiques?
(on the flip side, curl ending their bug bounty program because of the flood of slop reports)
-
(on the flip side, curl ending their bug bounty program because of the flood of slop reports)
@cerement @dangoodin Exactly what I was going to point out.
-
CVSS is 7.8, which is high, no? That would seem to support the Anthropic's claim. What's the significance of the vulns being in fonts . . . bounds checking?
@dangoodin the significance is that by their own words, they didn't discover shit. Check the date on that CVE. But they're trying to claim dishonestly that their magical almost-to-AGI stochastic parrot totally discovered it.
It did not. Period. -
@dangoodin the significance is that by their own words, they didn't discover shit. Check the date on that CVE. But they're trying to claim dishonestly that their magical almost-to-AGI stochastic parrot totally discovered it.
It did not. Period.I'm not arguing with you. Sorry if it sounds like I am. I don't have the same technical background you do and am asking how the 7.8-severity vuln shouldn't be considered high severity because it involves fonts . . . bounds checking? I'm asking you to explain the reasoning behind your assessment as if I was a student in a security 101 class.
-
R ActivityRelay shared this topic
