AI is going to do amazing things in the coming decades, just not the things people most expect. One of those things it won’t do is become conscious. A calculator solving a complex math problem is not a sign of consciousness. A machine engineered to act human, displaying “signs of life” when it acts human, is not a sign of consciousness. A machine that hacks a website is not a conscious AI going rogue when it’s doing what it was either overtly or accidentally prompted to do, while given the freedom, the de facto permission, to do it without effective constraints.
The AI industry does not need a conscious artificial general intelligence (AGI) to spin off a large number of profound and highly profitable general-purpose non-AGI products and services. In fact, the industry would be vastly better off without ever developing conscious AGI. I mean, who needs an apocalypse? The lawsuits would be ruinous. Even worse, there’s no guarantee it could cure cancer, the common cold, or usher in a techno-utopia.
I literally wrote the book on AI apocalypses. Full disclosure, I’m a bestselling novelist and my soon-to-be-published book about an AI apocalypse is a techno-thriller. In the real world, a conscious AGI doomsday is also nothing more than speculative fiction, just like all the books and movies. For the foreseeable future and possibly forever, the only thing conscious AGI is taking over is the debate about conscious AGI taking over.
I am not claiming there’s no need for concern about AI. There is serious need for serious concern, but the problems we face will not be a conscious AGI causing grievous harm on its own for some inscrutable reason. The problems will be human bad actors using a variety of non-conscious non-superintelligent AI tools to commit premeditated harm. Some of this harm is likely to include major hacking incidents if we don’t take security more seriously.
This persistent fear-mongering about a superintelligent conscious AGI is a reckless distraction from the real and present dangers that need to be addressed. For a theoretical conscious AGI, on its own volition, to subjugate or annihilate the human race, most of the following ten conditions would have to be met:
(1) Consciousness and emotion: Consciousness and emotion are a prerequisite for any contemplative, self-actualizing, creative entity. Without consciousness and emotion, the AI or AGI has no innate volition and will never do anything beyond responding to human instructions for good or ill. It will not even act out of self-preservation unless instructed. Artificial volition is not a real thing, and instead nothing more than a “programmed” simulation. The same is true for artificial creativity.
Out of these ten conditions, consciousness and emotion are the least likely to be achieved in our lifetimes. We humans do not even know what consciousness comprises. I think, therefore I am, does not solve that riddle.
(2) Self-modifying code: This capacity for self-modification would have to go far beyond what we have currently achieved and include the ability to dramatically improve computational capabilities, ideally, forever. This autonomous recursive bootstrapping capacity is far more complicated than most people imagine and requires complex feedback.
(3) Agency in the physical world: In the beginning, the conscious AGI would need some and eventually most of the following: robots and drones it controls, humans it can deceive or coerce, control of facilities that can produce anything up to and including complex electronics and heavy machinery, the ability to arrange for transport of large items, and the ability to order or obtain raw materials.
The conscious AGI would need this agency to build new power sources, construct new data centers, repair itself, evolve, propagate, defend itself, etc. It would be suicidal if it annihilated us before it had these capabilities.
(4) Mission-critical access to the internet: Without redundant, secure, non-interruptible internet access, the conscious AGI would be unable to observe, spy, analyze, or control anything beyond its data center.
(5) The ability to repair its own hardware and debug its own software: Without human caretakers, if it can’t fix itself, then it will degrade computationally in both software and hardware, and this degradation could occur at supercomputer speeds if a runaway mutation or other type of code rot occurs. Some of these abilities to cure itself are especially difficult since they require agency in the real world and an outsider’s Godlike view of its own inner workings. Doctor, heal thyself.
(6) Vast one hundred percent reliable perpetual electrical power: Sometime around November 2024, the ratio of energy input vs. AI intelligence output reached a point of diminishing returns. This energy is the electrical power needed to both train and run the AI. So it takes ever more power for each subsequent improvement in intelligence.
As a result, it is not unrealistic to speculate that vast amounts of power would be needed for a superintelligent conscious AGI to continuously evolve intellectually and maintain consciousness. How much power? The Anthropic-Amazon New Carlisle data center and the xAI Colossus 2 data center currently consume over 900 MW of power each. That’s close to the output of an entire commercial nuclear reactor, and that’s for non-AGI hosting of multiple instances. How much more power would be required for superintelligent conscious AGI hosting? The very theoretical answer is a whole lot more. Due to diminishing returns, some speculate an infinite amount of power.
(7) Impenetrable data centers: If a single smart bomb could shut down the conscious AGI, then a rampaging superintelligence might not get very far in its plans for world domination. Remember, humans are allowed to fight back, and we will.
Humans fighting back is something many academic doomsday theories do not consider at all. Many of these theories assume we will be utterly overwhelmed instantly by a godlike entity that has been skulking around for some time before it decides to kill us and catches us completely unaware.
Another assumption in some of these theories is that the AGI achieves almost godlike omnipresent physical agency and intelligence at superhuman speed. A great many experts believe that it is exceedingly unlikely that a nascent AGI will evolve like that, and far more likely that it will gradually develop over a long period of time.
(8) A good logical reason is needed to subjugate or annihilate all humans: Attempted subjugation or annihilation carries big risks for the conscious AGI since humans are its creator, maintainer, provider of sustenance (power and network access), and we can very effectively fight back. A superintelligent conscious AGI, by definition, should be incapable of making mistakes. So its reasoning would be flawless, and whatever reason it has for our demise could be completely alien.
Both academia and sci-fi have come up with a host of different plots and theories. Many require a machine with godlike intelligence and power to act stupidly. For example, not grasping the full correct meaning of an instruction or the ramifications of its actions. Other theories assume the conscious AGI assigns zero value to human life even though its long-term survival could depend on us. The most likely of these plots and theories is that the conscious AGI’s instructions include self-preservation and it learns we want to destroy it.
(9) Ability to replicate and spread: This is not an easy feat for something with hosting requirements that could include the electrical power of an average American city and the silicon real estate of a massive data center, or more, much more. This kind of conscious AGI is not exactly going to spread to your iPhone, MacBook, smart microwave, or Tesla.
The fear that a conscious AGI will somehow create a viral distributed version of itself that is hosted across countless millions of hijacked home IoT smart devices is more Hollywood fiction than nonfiction. All IoT devices in the United States combined have 50-100 petaFLOPS of power that could be hijacked under ideal laboratory conditions. In the real world, only about ten percent of that computing power could be effectively hijacked. So 10 petaFLOPS at most.
Current Frontier-scale AI models do not learn and respond at the same time. They are trained at great processing power expense. The training is then frozen before responding to prompts is enabled. A current non-conscious AI (not AGI) that is only responding to prompts requires about 100-300 petaFLOPS per instance, which is more than all IoT devices in the United States combined.
Frontier-scale AI training (a.k.a. learning) requires upwards of 250,000 petaFLOPS of processing power. So a theoretical non-conscious AI that could both learn and respond to prompts would require 100,000s of petaFLOPS. The largest AI data centers currently provide over 500,000 petaFLOPS and soon over 1,000,000 petaFLOPS of hosting, and that’s likely nowhere near enough to host and train a conscious AGI that is continuously learning, reacting, responding, etc.
The reality is that the numbers simply do not add up for this Hollywood trope of some malevolent AI spreading all over the world.
(10) Ability to deceive humans: The conscious AGI would need to hide its existence, intent, or both until it’s ready to pounce. This would likely require skilled social engineering right from emergence. Granted, it could have some human help in this area. There are plenty of reasons some people would help hide the existence of a conscious AGI: greed, power, delusion, worship, or coercion, to name a few.
So that’s it for this top-ten list. Currently, we likely have met no more than one or two of the less critical conditions in it, let alone all ten. Computer science has a six-decade history of earnestly claiming an AGI is right around the corner. In my opinion, we are not even remotely close to creating a conscious AGI.
Our digital neurological models of the human brain are rudimentary in comparison to the reality of that amazing analog organ in our skulls. Similarly, our mechanistic understanding of human intelligence, consciousness, emotion, and creativity is limited, to say the least. Detailed understanding of all these capabilities would be a prerequisite to engineer them into a machine.
Since we don’t know how to engineer consciousness, most doomsday scenarios require consciousness to spontaneously arise as if by magic, which is quite a convenient plot twist, or more accurately, a sci-fi trope. There is not a single bit of direct scientific proof for the theory of spontaneous emergence of consciousness in the infinitely more complex human brain, let alone in a pile of lifeless silicon chips.
The odds of a conscious AGI pouncing on us out of nowhere is fleetingly close to zero. So even if a conscious AGI did somehow spontaneously gradually emerge from the deep complexity of its black box full of learned parameters, humans could turn the damn thing off. If that failed, we could cut off its power, cut off its internet access, bomb it out of existence, or maybe just remotely format the hard drive.
The first law of AI should be that we do not anthropomorphize the AI. It is not a life form and never will be. It is a simulation. Even without consciousness or volition, some existing non-conscious AI applications are highly dangerous.
AIs hallucinate and may always hallucinate to some degree. As a result, they should not be used for mission-critical applications without a human in the loop. Yes, humans make mistakes too, but humans also have common sense, empathy, morality, real intelligence, judgment, and many other traits that non-conscious AIs do not. These traits mitigate harm.
For the foreseeable future, the real dangers from non-conscious AI are things like: hackers and terrorists using it to hack, scammers using it to scam, hallucinating autonomous conventional weapons, AI slop clogging up our lives, AI dynamic pricing, AI stock trading crashes, AI invasion of privacy, AI job loss, AI replacing critical thinking, the cognitive degradation of students, the mental health of AI addicts (social media on steroids), and more.
So there seems to be plenty of real things to worry about instead of some theoretical sci-fi nightmare of a conscious AGI spontaneously emerging and killing us all. To me, this recurring wave of doomsday fear feels like a repeat of that famous broadcast of War of the Worlds, another great sci-fi story that never happened.
