Using open-source AI, sophisticated cyber ops will proliferate

Open-source AI models are on track to disrupt the cyber security paradigm. With the proliferation of such models—those whose parameters are freely accessible—sophisticated cyber operations will become available to a broader pool of hostile actors.

AI insiders and Australian policymakers have a starkly different sense of urgency around advancing AI capabilities. AI leaders like Dario Amodei, chief executive of Anthropic, and Sam Altman, chief executive of OpenAI, forecast that AI systems that surpass Nobel laureate-level expertise across multiple domains could emerge as early as 2026.

On the other hand, Australia’s Cyber Security Strategy, intended to guide us through to 2030, mentions AI only briefly, says innovation is ‘near impossible to predict’, and focuses on economic benefits over security risks.

Experts are alarmed because AI capability has been subject to scaling laws—the idea that capability climbs steadily and predictably, just as in Moore’s Law for semiconductors. Billions of dollars are pouring into leading labs. More talented engineers are writing ever-better code. Larger data centres are running more and faster chips to train new models with larger datasets.

The emergence of reasoning models, such as OpenAI’s o1, shows that giving a model time to think in operation, maybe for a minute or two, increases performance in complex tasks, and giving models more time to think increases performance further. Even if the chief executives’ timelines are optimistic, capability growth will likely be dramatic and expecting transformative AI this decade is reasonable.

Detractors of AI capabilities downplay concern, arguing, for example, that high-quality data may run out before we reach risky capabilities or that developers will prevent powerful models falling into the wrong hands. Yet these arguments don’t stand up to scrutiny. Data bottlenecks are a real problem, but the best estimates place them relatively far in the future. The availability of open-source models, the weak cyber security of labs and the ease of jailbreaks (removing software restrictions) make it almost inevitable that powerful models will proliferate.

Some also argue we shouldn’t be concerned because powerful AI will help cyber-defenders just as much as attackers. But defenders will benefit only if they appreciate the magnitude of the problem and act accordingly. If we want that to happen, contrary to the Cyber Security Strategy, we must make reasonable predictions about AI capabilities and move urgently to keep ahead of the risks.

In the cyber security context, near-future AI models will be able to continuously probe systems for vulnerabilities, generate and test exploit code, adapt attacks based on defensive responses and automate social engineering at scale. That is, AI models will soon be able to do automatically and at scale many of the tasks currently performed by the top-talent that security agencies are keen to recruit.

Previously, sophisticated cyber weapons, such as Stuxnet, were developed by large teams of specialists working across multiple agencies over months or years. Attacks required detailed knowledge of complex systems and judgement about human factors. With a powerful open-source model, a bad actor could spin-up thousands of AI instances with PhD-equivalent capabilities across multiple domains, working continuously at machine speed. Operations of Stuxnet-level sophistication could be developed and deployed in days.

Today’s cyber strategic balance—based on limited availability of skilled human labour—would evaporate.

The good news is that the open-source AI models that partially drive these risks also create opportunities. Specifically, they give security researchers and Australia’s growing AI safety community access to tools that would otherwise be locked away in leading labs. The ability to fine-tune open-source models fosters innovation but also empowers bad actors.

The open-source ecosystem is just months behind the commercial frontier. Meta’s release of the open-source Llama 3.1 405B in July 2024 demonstrated capabilities matching GPT-4. Chinese startup DeepSeek released R1-Lite-Preview in late November 2024, two months after OpenAI’s release of o1-preview, and will open-source it shortly.

Assuming we can do nothing to stop the proliferation of highly capable models, the best path forward is to use them.

Australia’s growing AI safety community is a powerful, untapped resource. Both the AI safety and national security communities are trying to answer the same questions: how do you reliably direct AI capabilities, when you don’t understand how the systems work and you are unable to verify claims about how they were produced? These communities could cooperate in developing automated tools that serve both security and safety research, with goals such as testing models, generating adversarial examples and monitoring for signs of compromise.

Australia should take two immediate steps: tap into Australia’s AI safety community and establish an AI safety institute.

First, the national security community should reach out to Australia’s top AI safety technical talent in academia and civil society organisations, such as the Gradient Institute and Timaeus, as well as experts in open-source models such as Answer.AI and Harmony Intelligence. Working together can develop a work program that builds on the best open-source models to understand frontier AI capabilities, assess their risk and use those models to our national advantage.

Second, Australia needs to establish an AI safety institute as a mechanism for government, industry and academic collaboration. An open-source framing could give Australia a unique value proposition that builds domestic capability and gives us something valuable to offer our allies