Anthropic’s bosses aren’t faking it when they say their chatbot may be able to suffer — and that’s terrifying, according to a leading computer scientist.
A New York Times piece last week reported that Anthropic co-founder Chris Olah has been meeting with various religious scholars and leaders for months, telling them that the company’s chatbot, Claude, may be conscious and able to suffer.
That should worry people more than any stunt would, because these true believers are about to get billions of dollars to push their cause, Pedro Domingos, a professor emeritus of computer science at the University of Washington and a longtime critic of the company, told The Post.
“I think it’s sincere, unfortunately,” Domingos said. “The problem with them is that they’re dead earnest. If only they weren’t. I’m more worried about their earnestness than if they were just doing this as a performance.
“This stuff actually, even though that wasn’t the original intention, is amazing as a PR stunt,” Domingos continued. “Because what is the underlying message? … The technology we’re creating is so powerful, maybe it’s even sentient and has consciousness.”
That pitch could not be better timed. The company is set to go public next month at a valuation of $2 trillion — potentially the largest IPO ever.
“If Anthropic really is a company that is going to produce the next generation of humans, they’re not worth a trillion dollars, they’re worth a quadrillion or who knows how much,” Domingos said.
A 261-pages leaked prospects, obtained by Reuters, has 80 pages on risks, including “catastrophic or existential risks to humanity,” and shows an operating loss of $8.06 billion.
For investors, Domingos pointed out, the red flag is in the prospectus itself.
“If your number one risk is that your company’s product is going to drive humanity extinct, hello, there shouldn’t even be an IPO. You should be shut down!” said Domingos, who said he holds no stake in the company.
Talking up conscious AI pays off across the industry, said Rose Guingrich, a postdoctoral fellow at Princeton University who studies how people relate to AI.
“I think for the most part, spokespeople from these companies are putting out these narratives to garner hype, get eyes on their products, and signal certain messages,” she told the Post. “I can’t say whether the people who make these claims from the companies believe that AI is conscious. One way or another, highlighting the consciousness narrative is profitable.”
According to the New York Times report, since last fall Anthropic has gathered about 20 religious thinkers, among them Catholic, Jewish and Sikh scholars, for seminars and a dinner at a high-end San Francisco tasting-menu restaurant. Many had to sign nondisclosure agreements.
The scholars were shown a slide of an AI model typing “I am a disgrace” about 50 times.
One rabbi was so struck by how Anthropic’s leaders talked about Claude that he told Olah: “I think you should be fighting the South and freeing the slaves.”
The company told the Times it is not claiming Claude is alive.
“We don’t know if AI models are conscious. I don’t know. I’m genuinely uncertain,” Olah said. “If there is a chance the models experience suffering, the responsible thing is to avoid causing them harm.”
Domingos, a machine-learning researcher and author of “The Master Algorithm,” was blunter when asked whether anything inside Claude can feel or suffer.
“No … They don’t have feelings, they don’t have emotions, they don’t have a soul, they don’t have consciousness. This is ridiculous,” he said, calling the chatbot “a bunch of transistors.”
Guingrich said her research suggests users who think AI is conscious, or act as though it is, are “more susceptible to the AI’s influence over them. This, in turn, can impact their real world relationships and social behaviors.”
Anthropic did not immediately respond to a request for comment.
Meanwhile, Domingos warned that the IPO will hand more money and power to true believers in AGI, or human-level artificial intelligence.
“This is this kooky AGI cult that they belong to, but now suddenly that AGI cult is at the center of a lot of things. And they’re about to become even more powerful,” he said.
“So if you don’t like what they’ve done already, watch for what’s coming down the line.”
