CoSAI Strategic Update September 2025

This strategic update was written by a group of CoSAI Project Governing Board members with input from the CoSAI Technical Steering Committee and workstreams. It previews upcoming papers. This update does not represent the views of all members of the CoSAI project.

Imagine you’re a medieval blacksmith, and someone tells you that in five hundred years, a single factory will produce more horseshoes in a day than your entire guild produces in a decade. You’d probably think about what that means for horseshoe quality, or maybe worry about your job security. The implications of that kind of fabrication technology probably wouldn’t have you immediately jump to “and therefore the entire concept of transportation will be fundamentally reimagined and horses will become recreational curiosities.”

This is roughly where we are with AI and security right now.

The scaling laws are holding. This shouldn’t surprise anyone who’s been paying attention—they’ve been remarkably consistent since the Perceptron in 1957. Every year, we throw 4x more compute at these models, and every year, they get predictably smarter. Not in some mystical, science-fiction way, but in the boring, reliable way that Moore’s Law made computers faster. The difference is that this time, the thing getting more capable isn’t just calculating spreadsheets faster—it’s approaching human-level reasoning across increasingly broad domains. Even if that were not true and all chip and algorithmic advances halted today, each frontier lab could allocate 1,000 people working on Reinforcement Learning environments for every sector of the economy for the next 10 years and get incrementally more useful models from this effort alone.

By the end of 2025, somewhere between 80% and 90% of all code written will be written by AI at the frontier AI labs. Not reviewed by AI, not assisted by AI—written by AI. Engineers describe their workflow as “write a spec, go to lunch, come back and review.” The junior engineer role, as traditionally conceived, is already obsolete at these organizations. These changes will emanate in concentric shockwaves throughout the global economy over the next 18-36 months.

But here’s the thing everyone’s missing: this isn’t really about code. Code is just the first domino.

Agentic Identity Challenge

We have a problem that is an urgent operational challenge: what is the identity primitive for an AI agent?

Consider what happens when you hire a human employee. They get an email address, a Slack account, a manager, a desk (maybe), and gradually accumulate permissions to various systems as they demonstrate competence and need. The security model for this is well-understood—we’ve been refining it for decades. We know how to provision access, how to deprovision it when they leave, how to audit their actions, how to detect when their account might be compromised.

Now consider an AI agent that:

• Works on tasks for hours or days autonomously
• Maintains its own notes and memory between sessions
• Collaborates with team members via Slack
• Needs access to the same systems a human would need
• Can spawn sub-agents to parallelize its work
• Might be running on infrastructure you don’t control

Is this a service account? Not really—service accounts don’t learn and adapt. Is it a user account? Not quite—it can operate at superhuman speed and spawn copies of itself. Is it something new entirely? Probably, but our entire security infrastructure assumes a binary choice between “human” and “service.”

What we are confronting is a new class of non-human identities that don’t fit neatly into our existing access models. Our security infrastructure must evolve to circumscribe these identities with principles tailored to their unique properties—ephemeral existence, scale, and autonomy.

The companies that figure this out first will have an enormous advantage. Those who do it well will widen that lead even further. The ones who don’t will either hamstring their AI deployments with excessive security restrictions or create vulnerabilities that make traditional security breaches look quaint by comparison.

AI Levels the Cyber Battlefield

Think about the most sophisticated cyber attack you can imagine. Now realize that in 18 months, a moderately motivated teenager with $100 in API credits will be able to execute it.

This isn’t hyperbole. The recent DARPA AI Cyber Challenge demonstrated autonomous systems finding and patching dozens of real vulnerabilities with minimal human intervention. Google’s Big Sleep is finding vulnerabilities in production systems. These aren’t theoretical capabilities anymore—they’re being deployed today.

The traditional hierarchy of cyber capability—script kiddies at the bottom, nation-states at the top—is about to collapse into a much flatter structure. Third-tier actors will operate like second-tier actors. Second-tier actors will operate like first-tier actors. First-tier actors will operate like… well, we don’t have a good analogy for what first-tier actors with AI augmentation will look like. Imagine a team who can analyze every line of code in your infrastructure, understand every possible interaction between components, and craft exploits that precisely navigate your security controls—all in the time it takes you to read this paragraph.

The good news is that defenders have access to the same capabilities. The bad news is that defenders have to be right across their layered defense, while attackers can automate traversal of those layers. The weird news is that this might not matter as much as we think.

Refactoring the Security Start-up Ecosystem

For the past several decades, the security industry has operated on a build-versus-buy spectrum. Most organizations buy security products because building them requires specialized expertise that’s expensive and hard to find. A SIEM, a vulnerability scanner, an identity management system—these are complex products that take years to build and refine.

That entire economic model is about to invert.

When an engineer can build a respectable competitor to a commercial security product in a weekend using AI assistance, the calculus changes completely. Why pay for a vulnerability scanner when you can build one that’s customized to your exact infrastructure? Why buy a SIEM when you can create one that understands your specific business logic and risk profile?

This sounds like good news for security teams, and it is, but it’s disruptive news for security vendors. The security startup ecosystem is about to experience a refactoring. The most successful companies will be those that either:

1. Have genuinely unique data that can’t be replicated
2. Provide infrastructure that’s genuinely hard to build (like cloud-scale platforms)
3. Transform themselves into AI-augmented service providers rather than software vendors

The enterprise security market, currently worth hundreds of billions of dollars, is about to be compressed into a much smaller market for AI tools and platforms. The value won’t disappear—it will shift to the organizations that can most effectively deploy AI for their own security needs.

The Insider Risk Problem, Reimagined for AI

Let me paint you a picture of what’s coming, not in some distant future, but in the next 12-18 months:

You hire a new security analyst. They go through your standard onboarding—reading documentation, watching training videos, getting familiar with your tools and processes. They’re assigned a mentor and given a starter project. They join the team Slack, attend meetings, and gradually ramp up their productivity.

Except this analyst never sleeps, can work on ten projects simultaneously, and is more productive. They’re not a tool or a script or an “agent” in the current primitive sense—they’re a virtual collaborator, as close to a human team member as makes no difference for most practical purposes.

These virtual collaborators will have persistent memory, learning from every interaction, and increasingly have work of greater complexity delegated to them. They’ll develop expertise in your specific environment and even need to be performance managed. 

The organizations that figure out how to manage hybrid human-AI teams will thrive. The ones that either resist this change or implement it poorly will find themselves at a devastating competitive disadvantage.

If you accept that AI agents are inevitable—and the economic pressure makes them so—then the critical question becomes: how do we prevent them from going catastrophically wrong?

The answer is partially prevention, but, because the pace of change is so fast, the only truly reliable option is self-learning Detection and Response that continually manages risk posture.

These agents will have legitimate access to sensitive systems. They’ll need to perform actions that would look suspicious if a human did them but are normal for an AI (like accessing thousands of files in seconds). They’ll operate with a degree of autonomy that makes traditional access controls suffer from the same creeping problems that human access has had.

This is the insider risk problem but now with AI agents as the insider threat.

A viable response is to build sophisticated detection systems that can understand intent, not just actions. These systems need to:

• Establish baselines for normal agent behavior
• Detect when agents deviate from their stated objectives
• Identify potential prompt injection or misalignment
• Respond in seconds, not hours

This is fundamentally different from traditional security monitoring. It’s not enough to detect that an agent accessed sensitive data—you need to understand why it accessed that data and whether that reason aligns with its assigned task.

Organizations that nail this will be able to deploy agents aggressively and safely. Those who don’t, risk forgoing the benefits of AI automation or suffer spectacular breaches that make traditional incidents look trivial by comparison.

Here’s where things get genuinely weird: we’re about to use AI to create better AI, including better AI security systems, which will enable us to deploy more sophisticated AI, which will create even better AI security systems, and so on.

This recursive improvement cycle is already beginning. AI systems are writing code to improve AI systems. They’re designing better training regimens, identifying more efficient architectures, and automating away the bottlenecks in AI development. And analogous to that blacksmith-versus-fab-grown example that we mentioned at the beginning, we’re adding datacenters, networking equipment, and connective tissue at a record pace offsetting all economic downturn indicators—which itself will need to be secured by AI.

The solution isn’t to slow down AI development (that ship has sailed) or to achieve perfect security (that’s impossible). The solution is to architect systems that can anticipate failure, isolate the impact, recover quickly, and learn from each failure to improve resiliency.

CoSAI’s Strategic Priorities to Build Defenses Proactively

CoSAI can recognize that we’re navigating a fundamental transformation in how security works, who can do it, and what it means to be “secure.”

Our strategic priorities should be:

1. Define new identity and access paradigms for AI agents – There are emerging industry efforts at CSA, within OAuth, and we should offer a perspective. We should consider agent-to-agent, as well.

2. Develop detection and response frameworks specifically for AI systems – Traditional security monitoring won’t work for AI agents. We need new approaches that include the total risk posture and mechanistic interpretability.

3. Create guidelines for hybrid human-AI security teams – How do you manage a security team where half the members are AI? We need playbooks for this.

4. Establish norms around AI capability disclosure – As AI systems become more capable, how do we balance transparency with not providing a roadmap for attackers?

5. Build bridges across the AI and security communities – These have traditionally been separate worlds. They can’t afford to be anymore.
   
6. Augment prevention for a world where AIs look like insider threats – Seek to avoid persistent access and allow agents to only have contextually granted, just-in-time and just-long-enough access.
   
7. Establish a new model for the Software Development Lifecycle (SDLC): With AI agents engineering the majority of code, the traditional “shift-left” security model is insufficient. We must pioneer continuous AI-driven security remediation.

The security industry has always been reactive—we respond to new threats as they emerge. But the pace of AI capability improvement means we need to be proactive for once. We need to be building defenses for attacks that don’t exist yet, from AI systems that haven’t been created yet, targeting software and datacenter infrastructure that hasn’t been deployed yet.

Building the Future of Security Collaboratively

Despite everything we’ve written above, we’re actually optimistic. Here’s why:

AI is ultimately a tool, and tools reflect the intentions of their users. The same capabilities that enable sophisticated attacks also enable sophisticated defenses—at the software, networking, and datacenter layers. The same AI that can find vulnerabilities can also fix them. The same systems that can be misused can also be aligned and controlled.

More importantly, we’re having these conversations now, before things get truly weird. We’re building detection systems, thinking about identity management, and developing frameworks for AI governance. In SDLC, AI unburdens engineering teams—helping them deliver faster, take on more complex challenges, and spend more time on uniquely human contributions such as design vision and stakeholder alignment. The organizations that thrive in this transition will be those that blend human expertise with AI capabilities—treating the two as complementary workforces that, together, raise productivity and the quality of outcomes.

No single organization, no matter how advanced, can solve these problems in a silo. The challenges are too broad, the pace of change is too fast, and the stakes are too high. CoSAI exists to be the nexus for this collaboration, breaking down the traditional walls between the AI and security communities to build a shared foundation of trust.

The next 18 months will be chaotic, but they will also be decisive. This is our opportunity to be proactive and architect the security guidance and tools for the world that is coming, not the one that is passing away. The challenges outlined here will continue to define the core of CoSAI’s workstreams. The future of security is not something to be passively observed, it is something to be actively built.

Learn More and Get Involved

CoSAI is building the future of AI security through collaborative development of open-source tools, frameworks, and industry standards. Our community brings together security experts and AI leaders from organizations worldwide to create practical solutions that any organization can implement. Learn more about our initiatives, access our latest research and technical resources, and connect with our growing community on GitHub. We welcome new contributors and invite you to join us in advancing AI security for everyone.