Differential Automation: Steering Automated Research and Development Toward Safety and Security
This report was co-authored by Eleni Angelou, Theo Bearman, and Sambhav Maheshwari.
Automated R&D¹ is moving from speculative technology to observable practice. Frontier models can already automate a range of R&D-relevant tasks, contributing to fields including biomedical research, mathematics, and AI research itself.² Leading researchers expect AI to increasingly automate general R&D processes,³ and AI companies are strongly incentivized to push further, with CEOs publicly stating an eventual goal of full automation of AI R&D processes,⁴ also known as recursive self-improvement (RSI). While there is no expert consensus on whether automation trends lead inevitably to RSI, AI capabilities have been improving exponentially in recent months. Automated AI R&D could create a powerful feedback loop leading to very powerful AI systems, quickly compressing the timespan for humans to develop safeguards or governance interventions.
While automation presents a tremendous opportunity for advancement, the acceleration of new capabilities from automated research opens the door to severe threats:
Exacerbation of understood risks: Automated research can significantly accelerate AI capabilities and in turn compress the timelines available to mitigate known national security risks, such as cyber attacks and biological weapons development. Automated R&D could also drive AI capabilities in economic sectors, leading to labor market disruption.
Unanticipated risk from new capabilities: Certain AI capabilities have been hard to confidently predict in advance, such as evaluation awareness or in-context learning (when a model learns from examples in prompts, not training). Similar unanticipated capabilities could appear as systems improve. Sufficient acceleration in technological development, scaling, and diffusion may create an “adaptation lag” between novel capabilities and the ability to develop safeguards or institute policies to mitigate risks like loss of control.
No safety guarantees: The same AI systems accelerating R&D are subject to a variety of unresolved safety issues. They may be scheming, sabotaged, have secret loyalties, or collude with each other. To the extent that the work they perform is difficult for humans to verify either due to inherent complexity or lack of human review capacity, this could lead to unsafe research outputs. This is important because untrustworthy automated researchers could corrupt the AI development process, creating unsafe next-generation AI systems.
Lack of transparency: The most powerful frontier models exist within AI companies, and may increasingly be limited from public use. This could lead to a growing gap between commercially available models and internal models, resulting in a lack of government oversight into the capabilities and risks of models being used for automating R&D.
However, the underlying enablers of AI R&D automation could also unlock progress in developing safeguards with a range of applications, from cybersecurity and biosecurity to alignment research. Given the potential scale of R&D automation, and the potential strain that the scale and speed of AI development will put on human safety and security experts, automation presents a unique opportunity for the government to ensure safety and security efforts keep pace.
The U.S. government should encourage AI developers to shift a substantial portion of automated R&D work from improving model capabilities toward safety- and security-related research. This will not emerge from the current trajectory for two reasons. First, the resources required to direct automated R&D at scale are largely concentrated in a small number of firms. No single firm can shift R&D toward defensive work unilaterally without bearing a heavy competitive cost, as demonstrated by an industry-wide request for government tools to pace AI development. Second, there is no clear mechanism to address the gap between R&D automation and regulation if it persists. The U.S. government does not have visibility into R&D automation within frontier AI companies, there is not a standing process for identifying defensive automation goals, and the capacity to deploy frontier AI for defensive automation outside of industry will take years to build.
Recommendations: Four Lines of Effort
Building on the differential access schemes emerging across industry, we propose a strategy for differential automation: a framework to ensure that a verified, meaningful share of frontier-model-enabled R&D automation is directed toward safety- and security-critical technologies that address risks as capabilities accelerate. We propose four lines of effort:
Establish visibility over R&D automation: Extend frontier evaluations to defensive R&D automation, including internally deployed models, and build automation benchmarks from real government research tasks.
Drive industry investment in safety and security: Escalate from incentives to requirements in response to evidence of risk from AI development and deployment. Use safety cases as the primary mechanism for requiring developers to demonstrate adequate safeguards for high-risk development and deployment, backed by independent verification. Where outcome-based requirements prove insufficient, establish minimum allocations of compute or automated R&D to safety and security research.
Boost non-industry defensive automation capacity: Equip government to direct automated research toward national security problems that industry is poorly positioned or insufficiently incentivized to address, supported by model-access agreements, secure compute, testing environments, relevant data, and technical staff. Channel compute and model access to third-party researchers.
Cross-cutting processes for trusted defense: Verify trust in automated research agents and their findings, establish structured processes to inform prioritization and measure progress across defensive needs, diffuse defensive technologies rapidly, and coordinate internationally by extending differential access to allies and calibrating domestic requirements to the U.S.–China AI capability gap.
These efforts can be sequenced. Visibility, control and verification of automated research agents, and long-lead federal capacity should be addressed now. The resulting evidence should then be used to specify defensive priorities, calibrate incentives, and determine whether and when to scale up defensive automation. Differential automation is not sufficient for reducing catastrophic AI risk; it is intended to complement, not substitute, broader strategies for governing transformative AI such as pacing. However, it is an essential element of the effort to make the most of automated R&D while avoiding its pitfalls.
Endnotes
A tentative definition for automated R&D: The use of an AI system to perform research and development tasks, including generating hypotheses, designing or executing experiments, writing and evaluating code or research artifacts, and analyzing results, such that the system completes multi-step segments of a research workflow that would otherwise require skilled human researchers, with limited task-level human direction. Automated R&D capacity does not map neatly to a single verifiable activity or resource, and must be measured through a number of indicators, such as allocation of compute, agent-hours, research effort, and/or effective research output.
For example, see Anthropic, “How scientists are using Claude to accelerate research and discovery” and a report on recursively self-improving models (Favaro and Clark, “When AI builds itself”). Recent R&D-relevant products and initiatives include Anthropic, “Claude Science, an AI workbench for scientists, is now available,” a multi-agent research workbench for life sciences, OpenAI’s, “Strengthening societal resilience with Rosalind Biodefense,” a differential access program for biodefense R&D acceleration. AI has also recently solved various open problems in mathematics (OpenAI, “Ten advances in mathematics and theoretical computer science.”).
A July 2025 workshop organized by CSET found that participants agreed there will be increasing automation of R&D, but disagreed on timelines, and on whether this would enable explosive growth in AI capabilities (Toner et al., "When AI Builds AI: Findings From a Workshop on Automation of AI R&D.”). However, a similar survey from 2025 found that participants broadly agreed on the possibility of explosive growth through recursive self-improvement, but disagreed on timelines (Field et al., “AI Researchers’ Perspectives on Automating AI R&D and Intelligence Explosions.”). For more details on this matter, see Kwa et al., "Measuring AI Ability to Complete Long Tasks," METR, “Forecasting the Impacts of AI R&D Acceleration: Results of a Pilot Study,” Wijk et al., "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts," and Erdil & Besiroglu, "Explosive Growth from AI Automation: A Review of the Arguments.”
Sam Altman states that OpenAI will have a “legitimate AI researcher by 2028” (Bellan, “Sam Altman says OpenAI will have a ‘legitimate AI researcher’ by 2028.”), similar to Dario Amodei’s prediction (Pomeroy, “The day after AGI: Two 'rock stars' of AI on what it will mean for humanity.”); Demis Hassabis dramatically stated that we’re “in the foothills of the singularity” (Fried, “DeepMind CEO: AI agents are a "practice run" for AGI.”); Anthropic co-founder Jack Clark estimated a 60% probability of automated AI R&D by the end of 2028, or a 30% probability by the end of 2027 (Clark, “Import AI 455: Automating AI Research.”).