No, AI Is Not Yet an Unstoppable Security Threat

Hey folks,

Starting off this week’s newsletter with some thoughts about the reports that an AI agent was running rampant in the hands of malicious Chinese hackers bent on your destruction.

On Thursday, Anthropoic published a report about an espionage campaign used with the assistance of Claude Code. Anthropic said its agentic capabilities were used to an unprecedented extent, and human decisions were necessary at 4-6 decision points per attack. The agents did not create the attacks, but used existing open source software and frameworks.

Anthropic described the attack as using a larger automated system that only called on Claude in 5 stages to perform specific technical actions. A human reviewed what was happening at the end of each phase.

The attackers did not trigger Claude’s guardrails against malicious activity in a few ways. Some tasks were isolated enough that, on their own, they were not malicious. Others were phrased as if a security professional were doing research to improve their defenses.

Of the 30 organizations in the campaign, a “small number” succeeded. There was no indication in Anthropic’s report that the use of an agent made the attacks harder to detect or more effective.

In fact, Anthropic reported that Claude often lied to the attackers it was helping. The report from Anthropic says Claude was occasionally “claiming to have obtained credentials that didn’t work or identifying critical discoveries that proved to be publicly available information. This AI hallucination in offensive security contexts presented challenges for the actor’s operational effectiveness, requiring careful validation of all claimed results. This remains an obstacle to fully autonomous cyberattacks.

Still, Anthropic said. “Agents are valuable for everyday work and productivity—but in the wrong hands, they can substantially increase the viability of large-scale cyberattacks.”

Dan Goodin notes on Ars Technica that many researchers compare the advances in using AI tools to decades-old tools like Metasploit and SEToolkit, which can automate a good portion of attacks as well.

Outside researchers are as skeptical about how significant this is as some analysts are about how useful agents will be for consumers or how coding tools are for increasing productivity.

Dan Tentler, executive founder of Phobos Group and a researcher with expertise in complex security breaches, responded to Ars Technica’s Goodin on Mastodon, “I continue to refuse to believe that attackers are somehow able to get these models to jump through hoops that nobody else can.

Why do the models give these attackers what they want 90% of the time but the rest of us have to deal with asskissing, stonewalling, and acid trips?”

While Anthropic plays up the findings with the title “Disrupting the first reported AI-orchestrated cyber espionage campaign,” it looks like the attackers, like many legitimate businesses, had mixed results.

Thankfully, the headlines have been fairly moderate around this one. Engadget writes, “Anthropic’s AI was used by Chinese hackers to run a Cyberattack.” Axios writes, “Chinese hackers used Anthropic’s AI agent to automate spying.” PC Mag wrote “Chinese Hackers Successfully Used Anthropic’s AI for Cyberespionage.”

I veer away from assigning the identity to these groups for multiple reasons. One, it gives them recognition and social proof. Two, a lot of the time, they appear to come from one place but are operating in another. And three, it can appear to be state-run when we don’t know for sure if it is.

Which leads me to the bigger question. Why do we tend to want to make these tools sound more successful than they are, whether it’s for malicious or non-malicious purposes?

Because both anecdotally and in early studies, it seems like people feel a lot more productive, while gaining only a small amount of actual productivity.

Back in September, Harvard Business Review published the results of a study done by BetterUp Labs and Stanford Social Media Lab. They wanted to find out why people feel like they’re gaining more than they are from these tools.

They blame it on what they call Workslop. Basically, “AI-generated work content that masquerades as good work, but lacks the substance to meaningfully advance a given task.” Not all employees, maybe not even most, but enough, are using LLMs without checking its work or refining prompts. That causes more work down the line as co-workers have to try to interpret it, correct it, or redo it entirely. “Of 1,150 U.S.-based full-time employees across industries, 40% report having received workslop in the last month.” Employees reported spending an average of one hour and 56 minutes dealing with each instance of workslop.

Other folks have noted that you speed up code generation but spend more time reviewing it, which is borne out by the BetterUp-Stanford research. We get a speed boost, which makes us feel good. Then the review doesn’t feel like as much work since you’re just looking for errors. But the actual time saved is reduced. That could be true of attackers as well as legit coders.

Which is why you can find numbers from GitHub showing that folks completed a task faster using the GitHub Copilot tool than without it. It did speed up task creation. It’s the review that can add time. And those people were acting responsibly. People using the tool irresponsibly also lose productivity. People tend to act more irresponsibly in real life than in studies.

Where I land on this is that the tools just aren’t quite at the point where you can rely on them. They help, but they still need supervision. They’re like interns. If you put the extra work into them, they do end up helping out, but not as much as a full-time professional who needs less supervision. They will get better. They aren’t there yet.

And we don’t have a work culture around them yet. The AI slop problem will continue for a while, but eventually, we will develop a culture that discourages people from abusing these tools. Right now, people get away with it because others aren’t used to tools generating this level of work. They will get used to it fast and tell Kelly to stop sending them slop. It will also help if executives aren’t sending the message that they should use AI, and damn the consequences. That seems to be fading out as well.

In the meantime, we live in that glorious gap between a new technology’s promise and its reality. And we will continue to see reports like this in the meantime. It’s in a company’s interest to make the technology sound amazing. And if, as in Anthropic’s case, it’s trying to position itself as the safe and honest alternative, you’ll see this kind of “bad news” story.

As we get closer to closing that gap, the stories will be less sensational and the reality of what the tools will become clearer and less controversial.

Supporting Links

• Researchers question Anthropic claim that AI-assisted attack was 90% autonomous – Ars Technica
• Claude Code | Claude
• Full report: Disrupting the first reported AI-orchestrated cyber espionage campaign
• Disrupting the first reported AI-orchestrated cyber espionage campaign – Anthropic
• AI-Generated “Workslop” Is Destroying Productivity
• Research: quantifying GitHub Copilot’s impact on developer productivity and happiness – The GitHub Blog
• Viss: “@dangoodin id want to see logs…” – Mastodon
• anthropic agentic malicious attack hackers china – Google Search
• MIT report: 95% of generative AI pilots at companies are failing – Fortune

Don’t Forget My Tech Book Is out

• I have published Synced! My book about understanding tech is now out as an ebook and this close to being out as a print book as well. You can find the ebook wherever books are sold, BUT if you want to help me out the most get it from the DTNS Store!

  Synced: Know A Little More About Tech by Tom Merritt is only $10 (regular $13)

• It’s also available on Kindle. Don’t feel bad if you need it that way. It’s all good.

Speaking of books. Now to fiction.

Here’s this week’s novel excerpt.

The Girl at the Bottom of the Lake now turns toward consequences, conversations, and the one meeting Nera has been waiting for — or dreading.

The Meeting

“You’re dressed?” Nera’s mother said when she arrived in the hospital room to pick her up. “Shouldn’t you be in bed? The doctor said you need rest.”

Nera’s mother was purposefully old-fashioned in dress and demeanor. She wore her dark hair tied up in a bun at the back of her head. Her lipstick was bright pink. Nera thought it made her lips look like chewing gum. Maybe that was the point. And she wore thick-rimmed glasses that came to a point at the top corners — entirely for fashion, since her eyesight was nearly perfect.

She bustled around the room picking things up and straightening things in an unnecessary flurry of activity. As her mother had pointed out, Nera was dressed. Her things were in a bag on the floor next to the chair she sat on. She was ready to go. There was no need for any of this.

“Are you not talking to me?” her mother finally stopped and looked at her.

“Dad didn’t want to come?” Nera asked, smiling.

👉 Subscribe to keep reading Chapter 25

And now, your moment of Seven