Anthropic on Thursday updated its usage policy to bar people from “abusive or cruel behavior” behavior toward its Claude AI – even though it’s not, ya know, actually alive.
The strange update came as debate continues to rage about regulating AI, and as oddball corners of Silicon Valley insist the tech has a soul.
“We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models,” Anthropic says on a webpage about the updated policy.
“The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research,” the company adds.
Other changes to the user policy include new guidelines against using Claude for election interference, surveillance without the consent of the surveilled, and building weapons.
Those things were already banned. The new rules just clarify that, for instance, “our prohibitions include the software and components that make weapons work, as well as actions like arming drones and other autonomous vehicles,” Anthropic said.
The prohibition against being mean to your AI, first reported by the Verge, seemed sure to take some users aback.
It came after Anthropic co-founder Christopher Olah and colleagues reportedly spent a year meeting with religious leaders to discuss if their tech was conscious.
Olah almost cancelled a spring meeting at the Vatican after learning Pope Leo XIV planned to deliver a statement asserting AI is not, in fact, conscious, according to the New York Times.
“No computational system, however sophisticated, can create a heart that gives itself, or a conscience that discerns good from evil,” Leo states in his encyclical, the presentation of which Olah ended up attending.
Anthropic’s policy update document makes no mention of the thorny subject.
But the company – which has close ties to the controversial effective altruism movement – has a “constitution” for its Claude model that was reportedly based on a document internally known as the “soul doc.”
“As we discussed above, questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain,” the constitution states.
Anthropic’s new user policy drew mixed reactions online.
“Not that you should be rude to LLMs, but this is kinda getting ridiculous. Are we seriously giving a computer program that is nothing but matrices/vectors/tokens moral consideration here? Lol,” one person wrote on X, using the acronym for large language models.
“[B]etween this s–t, their insane valuation, and Elon’s connection to them, I have never hoped for a company to fail more than them,” another said.
The Post has sought comment from Anthropic.
