The real AI danger is not superintelligence. It is premature trust. | Andrew Rogoyski

Artificial intelligence (AI) terrifies us partly because it is new and partly because it is disturbingly familiar. We have invented machines that converse, flatter, argue and improvise with uncanny fluency. Confronted by this synthetic reflection, we behave like early humans seeing a mirror for the first time: We project consciousness behind the glass, then populate it with our expected behaviors, emotions, hopes and nightmares. In other words, we scare ourselves.

Today’s frontier models are impressively good at writing, coding and math. Yet they still fail at ingredients of useful artificial general intelligence (AGI), a system that matches or surpasses human capabilities such as consistent reasoning, long-term planning, factual consistency and reliable performance outside their “experience” (the data they were trained on). We are right to worry about the theoretical dangers of artificial superintelligence (ASI) because of the danger that such a system might outwit its human creators and pursue its own ends, potentially at our expense. While some AI executives have asserted that we’ve reached AGI, these claims are contentious and do not meet more considered definitions of AGI.

It’s easy to fear such a potentially advanced technology, but extravagant worst-case scenarios obscure more immediate and important risks. The most likely danger is not a digital god awakening and launching a Hollywood-style war against humanity. After all, AI depends on electricity, cooling, data centers, specialized chips, networks, technicians and sprawling supply chains, all of which are scarce and often fragile. Interrupt the power, restrict the graphics processing units (GPUs) used to train them, disable the cooling or disconnect the network, and in most scenarios, the supposedly omnipotent machine that’s out to kill humanity stops.

A genuine ASI might recognize this as an obvious fact; it is codependent on humanity, at least for the foreseeable future. A true intelligence capable of understanding the world better than we do might know that destroying its own industrial support system would be self-defeating.

A true intelligence might also recognize that its understanding of the physical world is limited and secondhand ‪—‬ accessible to it only by virtue of the texts, videos and audio feeds it can access. It lacks that crucial element of being embodied in the world — understanding how gravity behaves, what weather feels like, how objects interact, and so on. A true intelligence might realize this, see the limitations of its experience and, therefore, tread with caution. A true intelligence might be completely different from human intelligence, having other strengths, weaknesses, approaches, and even ethics. For example, consider that an AI intelligence can copy itself, running as myriad instances, communicating as a “hive mind” — a very different kind of intelligence. But I speculate.

What I can say is that the systems that should worry us most are not necessarily the smartest ones but rather the incomplete intelligences acting on narrow objectives, given agency and offered access to critical infrastructure. These systems may accidentally harm us simply because they don’t know any better. For example, there is a thought experiment created by Nick Bostrom, known as the “paper clip problem,” where an AI pursues its objectives of making as many paper clips as it can, but it then achieves that aim at the expense of the Earth and humanity which all get converted to paperclips in the pursuit of the AI’s narrow goals.

The critical question here is what we give such systems control over. That’s our choice and one we should make carefully. In the near term, two hazards deserve priority: cyber disruption and long-term economic upheaval.

AI-enabled cyberattacks could significantly disrupt advanced economies because hospitals, banks, energy networks, logistics companies and governments depend on interconnected digital systems. Recent examples of frontier AI systems hacking organizations, such as the OpenAI/Hugging Face incident, have given currency to this fear. Such attacks can result in extraordinary financial damage without being existential. Large parts of the world remain loosely connected to frontier AI — or even to the internet — and would not disappear simply because servers in wealthy countries failed.

The likeliest route by which advanced AI could exert control is not killer robots, but influence — misleading people, manipulating institutions and redirecting human effort. Yet humans already demonstrate formidable competence at misinformation, self-deception and destructive coordination. AI may industrialize those weaknesses; indeed, humans are using AI to industrialize those limitations, but AI did not invent them.

Economically, the danger is that companies will dismiss workers long before AI has proved capable of replacing them, cashing in the “AI dividend” too soon. Executives seduced by demos may confuse fluent language output with dependable labor, shedding institutional knowledge to chase theoretical efficiencies. The enticing vision of the “one-person unicorn,” where a person or a small group builds a billion-dollar business using AI as its workforce is questionable technically, economically and socially, even as a small handful of early examples take shape. Organizations are not merely bundles of tasks. They contain accountability, relationships, tacit knowledge and trust, forming part of a local and national community, resilient because of the diversity of skills and ideas they possess.

My suspicion is that the political turning point will resemble something like the 2017 WannaCry ransomware attack, where malware with core elements designed by the National Security Agency got wildly out of control, infecting hundreds of thousands of computers worldwide, infamously disrupting the U.K.’s National Health Service. A cyber operation amplified by frontier AI would be worse. Services would fail, lawsuits would multiply, companies could collapse and investors would retreat. Governments would demand a pause — or force laboratories to redirect resources toward control, auditing and safety, and companies would comply, if only to limit class-action legal exposure.

After the panic, the public might ask why we let this happen. And there have been multiple attempts to enact a pause, cessation or “kill switch” in AI development. We could enter another AI winter — where funding is cut and research stalls — similar to previous downturns in the 1970s and early 1990s.

There is also an uncomfortable likelihood that AI labs themselves are amplifying fear and uncertainty. Casting a product as potentially world-ending implies unprecedented power. Such narratives are just marketing; they may help to sustain investment, defend stock valuations, and prepare for forthcoming initial public offerings (IPOs) even when business models remain uncertain. It’s hard to speculate about a company’s motivations ‪—‬ but there are obvious incentives, so their behaviors deserve scrutiny.

The more mundane outcome may be “the great disappointment.” Advanced AI could prove too expensive and insufficiently useful to continue on its present trajectory.

The training and operation of frontier systems consume enormous amounts of computing time, electricity, water, capital, hardware and human effort. Unless those costs fall to a fraction of today’s levels, the economics of “Big AI” may prove unsustainable. Meanwhile, important unsolved problems in AI ‪—‬ like reliability, interpretability and continuous learning ‪—‬ are being overshadowed by the race to build incrementally better models, each vying for the top slot in the various AI performance indices.

The strongest case for AI may instead lie in artificial specialized intelligence: systems such as AlphaFold2, which predicts the structures of proteins, or WeatherNext 3, an advanced global weather forecasting model. These systems are designed for domains where performance can be measured and where the benefits justify the costs. We know AI is genuinely useful when aimed at topics like drug discovery, diagnostics, materials science, power grid optimization and climate change modeling — rather than human-like chatbots generating emails, advertisements and synthetic entertainment.

The great tragedy, and possibly the biggest danger, would be to continue to direct precious resources toward the creation of mediocre human simulacra, get bored or scared of AI, and abandon the field precisely when it could do the most good. Fear is appropriate when it sharpens engineering and governance. It becomes dangerous when it distorts understanding and priorities.

It’s unlikely that AI will become our conqueror anytime soon. Instead, it will probably become an expensive disappointment. But if the backlash slows work that could help cure cancer or address climate change, the failure will not belong to the machines. It will be ours.

Opinion on Live Science gives you insight on the most important issues in science that affect you and the world around you today, written by experts and leading scientists in their field.

Help us improve Live Science Pro: We’re always trying to make our content better. Leave us feedback about Pro here.