Now
America.gov opens its doors, then revises its answers €30bn+EU call for up to seven AI gigafactories Europe’s AI Act: what applies now, and what waits until December 2027 554deepfakes logged in Brazil’s first round; only 371 labelled California signs 13 AI bills, including the “No Robo Bosses Act” 30+AI firms sent the first AI Act information requests Washington and Beijing open an AI incident channel 164AI-made ads in the US midterms, by one count Medicare’s AI pilot: thousands of denials, and an 83-day wait 42signatories to Canada’s voluntary data-centre principles The midterms’ machine-made ads 174MWCape Town data centre now under appeal Pew: global publics trust China more than the US or EU to regulate AI 83 dayswait reported under Medicare’s AI pilot, against a 72-hour standard Indonesia’s AI Rules Are Stuck on the President’s Desk 20home-grown foundation models backed by the IndiaAI Mission Pakistan Writes Rules for the Algorithmic State 2%of Turkish public investment budgets earmarked for AI Cape Town’s Data Centre Fight Puts a Price on AI’s Thirst Turkey Governs AI by Circular, Not Statute ChatGPT Now Answers to Brussels Twice Over Mexico Wants One AI Law. Its States Got There First Washington’s $1 AI Era Is Over. Now Agencies Get the Meter India’s Sovereign AI Bet Meets the Price of Silicon
AI&GPP
Sign in Subscribe

Guardrails for Hire: When Defence Contracts Rewrite AI Rules

Two US courts have given opposite answers in the dispute between the Pentagon and Anthropic. Beneath the litigation lies a larger question: should a vendor's usage policy or a government's contract decide how the military uses AI?

Security Desk
The Pentagon from the air, May 2023
Photo: Staff Sgt. John Wright, US Air Force / DoD, CC BY 4.0 · source

In the space of a month, two American courts reached opposite conclusions about the same dispute. On 27 August a federal judge in California found that the Defense Department had acted unlawfully in punishing Anthropic, the developer of the Claude models, CNN reported. On 25 September the US Court of Appeals for the D.C. Circuit ruled 2–1 that the department may designate Anthropic a “supply-chain risk” and remove Claude from its work, according to the Associated Press. Further review is possible.

The case is being litigated as a question of administrative and procurement law. Its significance is broader. It is the clearest test yet of whether a government can use its purchasing power to override the conditions an AI company attaches to its own products, and of who should decide what military AI may and may not be used for.

How the dispute arose

According to the reporting, the disagreement dates to February 2026, when Anthropic resisted dropping restrictions in its usage terms on surveillance and autonomous-weapons uses. The Pentagon subsequently moved to label the company a supply-chain risk, a designation that allows the department to exclude a supplier from its systems.

The two rulings reflect the two sides’ positions. The California district court, as reported by CNN and TechCrunch, accepted that the department’s action amounted to unlawful punishment of the company, a first court win for Anthropic. The appeals court majority, as reported by AP, accepted that the department has the authority to make the designation and act on it. The split decision at the appellate level, and the conflict with the lower court, mean the legal question is unlikely to be settled soon.

This publication takes no view on which court is right in law. The structural question is the one that will outlast the case.

Guardrails at the negotiating table

The dispute does not stand alone. On 1 September Fortune reported that the Pentagon’s GenAI.mil platform had added OpenAI’s ChatGPT and xAI’s Grok, cleared for controlled unclassified information at Impact Level 5, alongside Google’s Gemini. About 1.7m of the department’s 3m personnel use the platform.

A week later The Intercept published documents obtained under freedom-of-information law showing that the Pentagon had asked OpenAI for a militarised model with “minimal refusal rates”. OpenAI says it never agreed to that language.

Model guardrails used to be a design choice; they are now a line item in a defence contract.

Taken together, the episodes show the same pattern. The behaviour of an AI model, what it will refuse, what uses its maker permits, is no longer settled only in a developer’s policy document. It is negotiated, and sometimes contested, between buyer and seller.

The case for the buyer

There are serious arguments that elected governments, not private companies, should set the terms of military use. Defence policy is subject to democratic control, statutory limits, the laws of armed conflict and congressional oversight. A company’s usage policy is subject to none of these: it can be changed by a board, may reflect commercial or reputational considerations, and is not accountable to voters. A military that depends on a tool whose maker can restrict or withdraw it in a crisis has, on this view, a genuine supply-chain problem. And a vendor’s refusal to support lawful missions could be seen as a private veto over public policy.

The case for the vendor

The counter-arguments are also substantial. A developer knows its model’s failure modes better than any customer and may judge that certain uses, such as autonomous targeting or mass surveillance, are beyond what the system can do reliably. Usage restrictions can function as a safety margin where law has not yet caught up with the technology. If a government can penalise a supplier for maintaining such limits, every vendor learns that guardrails are negotiable under pressure, which may weaken safety practices across the industry, including for civilian customers.

The civilian contrast

Civilian procurement is moving in a different direction. On 10 September the General Services Administration announced a OneGov agreement giving federal, state, local and tribal governments consumption-based access to ChatGPT at a 50% discount for 27 months, effective from 1 October, with no platform fees or minimums. GSA says the earlier OneGov AI deals saved about $1.4bn across roughly 3.5m federal employees.

That arrangement is about price and access. It leaves model behaviour largely to standard commercial terms. The defence cases suggest that, as AI moves into higher-stakes uses, the content of those terms will become the contested part of the deal.

Who should decide

The most defensible answer may be neither the vendor alone nor the procurement office alone. Lines on military AI use that matter, on surveillance of citizens or on autonomy in lethal force, are questions of public policy. They are better set openly by legislatures and published doctrine than resolved contract by contract or case by case. Until they are, disputes like this one will keep landing in courtrooms, where judges are asked to answer policy questions through the narrow lens of procurement law.

What to watch

  • Whether Anthropic or the government seeks further review of the D.C. Circuit ruling.
  • Whether other vendors revise their usage policies for government customers.
  • Any congressional move to set statutory limits on military uses of AI.
  • Disclosure of the final terms agreed with providers on GenAI.mil.

Sources

  1. AP via KSAT — Federal court says Pentagon can label Anthropic a supply-chain risk
  2. CNN — Pentagon’s supply-chain risk action against Anthropic found unlawful
  3. TechCrunch — Anthropic gets its first court win over the Pentagon’s supply-chain risk label
  4. Fortune — Pentagon adds ChatGPT and Grok for military personnel
  5. The Intercept — Pentagon and OpenAI military contract
  6. GSA — GSA expands OneGov AI offerings with discounted OpenAI ChatGPT
  7. Nextgov — GSA unveils new token-based OneGov discount for OpenAI

AI & GPP reports on how artificial intelligence and automation are changing the way governments decide, regulate and campaign. Corrections and tips: contact the editors.

Get the stories that matter to decision-makers, weekly.

Discussion

No comments yet. Start the conversation.

Discussion is open to members.

Join free Sign in

Read next

Humain’s Pragmatic Turn: Chinese Weights, American Chips

Saudi Arabia's state AI champion has built its flagship Arabic model on Chinese open weights, run it on American hardware and gone looking for outside capital. Sovereign AI in the Gulf is becoming a hedging strategy, with new risks for partners.