AI Decision Support Systems: A Neglected Source of Military AI Risk
This policy memo was co-authored by Brianna Rosen and Bill Anderson-Samways.
The Pentagon’s current policy on lethal autonomy does not address AI decision-support systems (AI-DSS), which are now operational and shape lethal decision-making in ways that can introduce errors, bias, and escalation risks in U.S. military operations. Department of Defense (DoD) Directive 3000.09, the principal policy framework governing lethal autonomy, covers fully and semi-autonomous weapons systems, but does not extend to AI-DSS that analyze intelligence, generate target recommendations, or present commanders with ranked courses of action. These systems have attracted limited policy scrutiny on the flawed presumption that they preserve human judgment by shaping, rather than executing, the use of force.
AI-DSS increase the pace and scale of decision-making, producing recommendations that are difficult to independently verify under time pressure. Rather than preserving human judgment, these systems structure the decision space before the human enters it — and, as they grow more capable, the effective scope of human oversight is likely to narrow further. As the Pentagon moves from experimental adoption of AI-DSS to more widespread operational use, including most recently in Iran, the absence of adequate safeguards poses increased risks to U.S. military personnel, allied forces, and civilians. Expanding state use in Gaza, Ukraine, and beyond suggests AI-DSS will play a more decisive role in future conflicts, underscoring the urgency of addressing this challenge now.
Existing legal and policy frameworks have not kept pace with these operational realities, leaving a critical gap in ensuring that AI-DSS are secure, reliable, and subject to meaningful oversight. AI-DSS rely in part on commercial frontier AI models — including Anthropic’s model Claude, which is reportedly integrated into Palantir's Maven Smart System — that may not have been tested and evaluated sufficiently for security and reliability.Adversarial compromise, corrupt or outdated data, and misalignment in these systems can propagate directly into use-of-force decisions, resulting in target misidentification, increased collateral damage, or unintended escalation, even where a human remains formally in control.
This memo identifies key failure modes that make frontier AI-DSS a distinct policy problem and proposes four recommendations to address them: (1) establish meaningful human accountability for AI-DSS; (2) invest in R&D to make AI-DSS more reliable and secure; (3) develop acquisition standards for reliability and monitorability; and (4) provide comprehensive AI-DSS training and upskilling for operators, commanders, and technical staff.
What are AI Decision Support Systems?
AI-DSS are tools that process large volumes of data to generate recommendations for military decision-makers across the chain of command. In modern warfare, commanders rely on an expanding array of sensors, communications systems, and intelligence sources to build situational awareness and direct operations. AI-DSS integrating frontier AI — highly capable, general-purpose systems at or beyond the capabilities of today's most advanced variants — can process this information faster and at a greater scale than human analysts, and with increasing autonomy. These systems act as a resource multiplier, allowing states to develop more comprehensive battlefield awareness, accelerate targeting cycles, and compress the decision loop in ways that can confer decisive military advantages in near-peer or high-intensity conflicts. The practical effect is that while a human formally authorizes the use of force, the framing of that decision has already been shaped by the system.
Illustrative Use Case
Intelligence analysts task an AI-DSS with identifying and prioritizing targets in the area of operations. The system's AI agents rapidly fuse data across intelligence disciplines, including intercepted communications between enemy commanders, satellite imagery revealing concealed vehicle formations at a candidate site, and social media posts from nearby civilians reporting unusual military movement in the area. The AI-DSS presents analysts with a ranked target list, each entry accompanied by confidence levels, sourcing, and a collateral damage estimate that accounts for nearby civilian infrastructure.
A targeting analyst selects a priority target and requests strike options. The system generates courses of action pairing available platforms with appropriate munitions, optimized to minimize risk to civilians and friendly forces. When new intelligence indicates the target is preparing to move, the targeting analyst asks the AI-DSS to regenerate options using only assets currently on station. The system presents a revised course of action, noting that the available munition has a larger blast radius, and provides an updated collateral damage estimate, allowing the officer, authorizing commander and Judge Advocate General officer to weigh the risk of the target escaping against the increased collateral risk.
How Might Such Systems Fail?
AI-DSS that widely integrate frontier AI could have failure modes that are qualitatively distinct from conventional software defects. When deployed operationally such failures could increase the likelihood of or amplify targeting mistakes, including by misidentifying civilian targets as military objects, underestimating anticipated collateral damage, and recommending courses of action that result in unintended escalation. The technical failure modes listed below have been observed in experimental settings with frontier AI models and could plausibly impact the ability of AI-DSS to perform effectively in military operations.
Technical Failure Modes
1) Adversarial compromise. Adversaries may intentionally corrupt AI-DSS to make its outputs diverge from operators' intent.
Data Poisoning: An adversary corrupts the model’s training data, causing it to learn subtly incorrect behaviors that are difficult to detect after the fact.
Adversarial Manipulation: Feeding the AI-DSS misleading inputs during operations, such as spoofed sensor data, causing target misindentification or situational misjudgment.
Sleeper Agents: Tampering with models so they behave normally during testing but act against operators' intent — such as targeting allied forces instead of enemy forces — when specific trigger conditions are met during deployment. Research suggests such hidden behaviors can persist despite standard safety measures, including adversarial red-teaming.
2) Misalignment. Even without adversarial compromise, an AI-DSS may pursue objectives that diverge from its operators' intent.
Specification Gaming: Overseers may inadvertently reward a subtly mismatched objective during fine-tuning (e.g., the AI-DSS learning to prioritize easy-to-confirm lower-value targets over high-value ones).
Goal Misgeneralization: A correctly specified goal may generalize poorly beyond the training environment. For example, an AI-DSS rewarded for decisions "coherent with existing doctrine" may perform poorly against an adversary deploying irregular tactics, which are by definition difficult to anticipate in training data.
Sycophantic Behaviors: AI-DSS may tell operators what they want to hear rather than providing accurate assessments (e.g., downplaying collateral damage estimates when accurate reporting would complicate mission objectives).
Escalation Risk: AI-DSS may systematically recommend escalatory courses of action due to tactical optimizations that insufficiently account for strategic and political consequences. Research in simulated nuclear crises has found that frontier LLMs exhibit pronounced escalation bias under time-sensitive constraints.
Institutional Failure Modes
Beyond technical failures, overreliance on AI-DSS can degrade the quality of use-of-force decisions through cognitive and institutional dynamics.
1) Cognitive biases. Even a technically reliable AI-DSS can produce consequential errors in use-of-force decisions if the institutional context in which operators engage with outputs systematically undermines meaningful human judgment.
Automation Bias: Operators over-trust AI-DSS outputs, reducing independent verification, particularly when systems consolidate multiple analytical steps into a single recommendation in fast-paced operational environments.
Confirmation Bias: Operators are more likely to accept AI-DSS outputs that align with prior expectations rather than scrutinizing them for errors.
Cognitive Offloading: As AI-DSS perform underlying data synthesis and recommendation generation, operators may increasingly transfer active reasoning to the system. This narrows the decision space before the human enters it and may create a feedback loop in which growing system capabilities progressively displace human judgment, even when a human remains formally in control.
2) Deskilling. Repeated deference to AI recommendations may erode operators’ capacity to conduct complex assessments independently, with direct implications for operational readiness and the robustness of human oversight in high-stakes military deployments.
Policy Recommendations
The expanding integration of AI-DSS on the battlefield, coupled with the distinct and potentially high-impact failure modes of more advanced versions of these systems, warrants proactive policy interventions. The recommendations below are designed to improve system resilience to adversarial compromise, establish meaningful human accountability, and develop the tools needed to detect, prevent, and respond to failure modes before they arise operationally.
1) Establish meaningful human accountability for AI-DSS. Update Pentagon Directive 3000.09 to include guidelines for appropriate human oversight of AI-DSS, including to:
Ensure that a specific individual in the chain of command is designated as accountable for each engagement involving AI systems used for target generation or lethal force recommendations;
Require that human operators carefully reviews each AI-generated target at each stage in the targeting cycle;
Preserve human-readable chain-of-thought reasoning traces and logging practices to ensure AI-DSS outputs are explainable and auditable, and;
Conduct mandatory legal reviews of AI-DSS as a “means or method of warfare.”
2) Invest in R&D to make AI-DSS more reliable and secure.
Fund increased testing and evaluation of AI systems that contribute to use-of-force decisions, including in the period prior to the completion of the standardized assessment framework which will be developed by the Cross-Functional Team for Artificial Intelligence Model Assessment and Oversight pursuant to FY26 NDAA §1533.
Harden AI-DSS against adversarial manipulation by developing automated solutions to find, triage, and fix adversarial compromise failure modes, including those outlined above, and piloting a secure data center for inference and fine-tuning, including R&D into secure hardware, software, protocols, and formal verification.
Build resilience to misalignment failure modes in AI-DSS by investing in scalable oversight techniques (including automated detection and control mechanisms); stress-testing AI-DSS in realistic simulated “military ranges” to better illuminate pathways through which misaligned behavior might manifest; and evaluation frameworks which measure AI-DSS faithfulness to operators’ intent across operationally representative scenarios, including time-pressured and escalation-prone contexts.
3) Acquisition standards for reliability and monitorability. Develop a Pentagon-wide acquisition framework ensuring that AI-DSS are resilient against adversarial manipulation and reliably follow the intent of their commanders and operators. These conditions could also include a requirement for AI systems acquired by the Department to produce faithful and human-readable reasoning traces and to declare any limits in interpretability.
4) Comprehensive AI-DSS training and upskilling. Update Pentagon training and upskilling requirements to ensure that operators of AI-DSS, commanders who rely on their outputs, and technical staff who maintain their supporting infrastructure are aware of the failure modes outlined above and how they can be mitigated.