Amidst all the talk of worldwide apocalypse, Guerre and Shalom book author Ashok Panikkar identifies a more foundational malaise. Even if AI were to remain utterly docile, it would still pose a fundamental challenge, for in its utility it has and will continue to deprive humanity of thought. As Ashok taught us in his Guerre and Shalom classic The Ten Lies that Killed Democracy, thinking is the indispensable cornerstone of the democratic experiment, and in its absence we are utterly diminished as a citizenry — to the point of uselessness.
The article you are about read in one of the most original and thoughtful contributions to the AI debate, a subject I will return to in a future chapter of The Genocide of Truth. In the meantime, enjoy, like, spread and subscribe. And always ask yourself Ashok’s well-conceived question: Not “Can we?”, but “Should we?” - Daniel Clarke-Serret (editor)
The Latest Chapter of The Genocide of Truth:
“Sure we can. But should we?” by Ashok Panikkar
So, somewhere between Trump, Iran, the culture wars and whatever catastrophe is currently trending, we appear to have settled on the next thing that will destroy civilization: Artificial intelligence.
Excellent. At least we’re thinking big again.
As someone who has been banging on about the existential dangers of digital technology for decades—and who was teaching courses on the dangers of artificial intelligence when ChatGPT was still a mere glint in its creators’ eyes—I should probably be feeling rather vindicated right now.
Instead, I find myself slightly irritated. Not because everybody is suddenly worried about AI. But because we are debating how to keep the genie in the bottle after handing it the corkscrew. There’s something slightly comical about the whole discussion.
We are apparently terrified that one day machines will become incredibly intelligent and do something catastrophically stupid. Well, considering that we are creating them in our own image, we shouldn’t be surprised. If artificial intelligence eventually decides to destroy the human race, it will merely be carrying on a long human tradition.
I want to make a slightly different argument. For all our ingenuity, inventiveness and sheer technical brilliance, I am no longer entirely convinced that Homo sapiens is the brightest species on the planet.
Oh, we are certainly the cleverest. Beavers build dams. We build the Three Gorges. Birds navigate across continents. We put satellites into orbit so that Google can tell us how to get to the supermarket. Other animals occasionally fight over territory. We invented nuclear weapons so that we could threaten to vaporize the territory, ourselves and everybody else along with it.
No contest. We are spectacularly clever. But clever is not the same as wise. A species capable of splitting the atom but apparently incapable of asking what happens after it does so deserves, at the very least, a slightly more modest opinion of its own intelligence.
That is the distinction I want to make here. Our problem has rarely been a shortage of intelligence. Our problem is that our ingenuity repeatedly outruns our judgment. We invent first. Scale second. Celebrate third. And somewhere considerably further down the road, usually after something has gone badly wrong, somebody finally asks: Was this a good idea?
In fact, I suspect this may be one of the defining characteristics of our species. We are astonishingly good at figuring out how to do things and considerably less impressive at deciding whether we should do them.
That distinction matters.
Because some of the people creating the technologies that are transforming human civilization are obviously brilliant. I couldn’t begin to understand much of what they know. They can build neural networks, write millions of lines of code, create machines that recognize faces, translate languages, diagnose diseases, write poetry and explain quantum mechanics to a twelve-year-old.
Extraordinary.
But here’s the uncomfortable question. Does any of that brilliance safeguard against irresponsibility or stupidity? Does any of that brilliance make them wise- because intelligence and wisdom are not the same thing. Intelligence asks: Can we do this? Wisdom asks: What happens if we do? And really good judgment asks the question we almost never ask: And then what?
We keep worrying that AI will become too intelligent. Perhaps we should worry that intelligence was never the problem.
We Can’t Say We Weren’t Warned
This is what bothers me about the present panic over AI. We’ve been here before. Not with machines becoming conscious and deciding that humans are an annoying infestation, admittedly. But with technologies transforming human beings and human society in ways their creators barely understood. And contrary to the story we sometimes tell ourselves, this time around nobody can say we weren’t warned.
Marshall McLuhan had warned us that communications technologies don’t merely carry information- they change the people using them. Neil Postman warned that television was turning public life into entertainment and citizens into spectators—perhaps entertaining us right out of our capacity to be citizens. Joseph Weizenbaum—one of the early pioneers of artificial intelligence, no less—warned us about handing human judgments over to computers merely because computers could make them.
Much later, Sherry Turkle—whose discussion group I had the privilege of attending sometime in the early 2000s—began asking what our growing relationship with computers might be doing to us.
I don’t remember precisely where it was, Harvard or MIT- or even what the gathering was called. But I distinctly remember the question. Perhaps because I had been asking some version of it since the early Nineties, when I watched designers scrambling to learn AutoCAD, Photoshop and CorelDRAW. Everyone was understandably excited by what these extraordinary new tools would allow us to do.
I found myself wondering about something else: What would we forget how to do?
There were others who asked similar questions. And those who might actually have had a say in these things listened, nodded and looked appropriately concerned. “Very interesting. Very vital questions.” And then: “Now, where were we?”
Ah, yes. Building the thing before somebody else did.
The Amazing Computing Machine
Think about the computer, that amazing machine that was going to liberate us from drudgery. And it did. It calculated for us, stored things for us, remembered things for us, corrected our spelling, found information, and kept our calendars. It told us where we were and even told us how to get where we wanted to go.
All wonderful. Except we didn’t spend much time asking a rather obvious question: What happens to the skills and abilities we no longer need to use? I used to know telephone numbers, lots of them. Now I cannot remember a single number other than my own- and that too only after two cups of coffee.
This isn’t a catastrophe. Civilization will probably survive my inability to remember my dentist’s telephone number. But the principle is worth thinking about. Whenever technology removes the necessity for a skill or a human capacity, what happens to them?
We didn’t really ask ourselves that simple question. We were too busy being delighted by the convenience.
The Extraordinary World-Wide Connector
Then came the internet and suddenly almost the entire accumulated knowledge of humanity was available to ordinary people. This was genuinely miraculous, so naturally we assumed that having vastly more information would make us all vastly more knowledgeable.
How’s that working out for us? It turns out that information and knowledge are not the same thing- nor is understanding. Who knew?
Then we got social media, another magnificent idea. Let’s connect everybody! Let’s give everybody a voice! Let’s democratize publishing! Let’s allow anyone, anywhere, to communicate instantly with anyone else! What could possibly go wrong?
Well. We discovered that human beings like attention. Then we discovered that companies could make money from that attention. Then the companies discovered that anger and hatred gets even more attention. Then politicians discovered what the companies had discovered. Then everybody discovered outrage. And here we are.
I exaggerate, but only slightly. The extraordinary thing is that nobody planned this. There was no room in Silicon Valley with twelve evil men sitting around a table saying: “Gentlemen, how shall we destroy civil society?” Everybody concerned made perfectly rational decisions.
The engineer improved engagement. The company increased advertising revenue. The influencers attracted followers. The politicians got attention. The users got another little dopamine reward. Each decision made supreme sense.
And collectively? We create something none of us intended. I keep coming back to this because I think it is one of the most important principles of judgment: A society can destroy itself through millions of perfectly sensible individual decisions.
The Miraculous Intelligence Generator
So here we are again. Except this time the technology doesn’t merely transport us, entertain us, connect us or give us information. It thinks for us. Or at least performs activities that we have traditionally associated with thinking. And once again we are asking only the questions we know how to ask.
How powerful can it become? How fast can it get? What jobs can it do? How much money can companies save? Can it cure cancer? Can it write software? Can it replace lawyers? Can it replace teachers? Can it replace me? All reasonable questions.
But I’m increasingly interested in another one. What happens to human beings when thinking becomes unnecessary?
Not all thinking, obviously, but enough of it.
Suppose AI never becomes conscious, suppose there is no Terminator and no malevolent superintelligence. Suppose there’s no robot coup and suppose the machines remain wonderfully obedient forever. Will we be okay? Will we still remain ‘human’? How will humanity itself change?
From our experiences of the printed book, the automobile, the radio, TV and the internet, I can safely assume that AI will still change us profoundly. Because why struggle to write something when the machine can write it? Why learn to construct an argument? Why memorize anything? Why struggle through a difficult book? Why sit with a question you don’t understand? Why learn another language? Why develop the patience required to understand somebody whose worldview annoys the hell out of you? Why spend three hours trying to figure something out when you can get the answer in three seconds?
We are terrified that AI will become too intelligent. But we may be looking in the wrong direction. Perhaps we should be even more frightened of losing our own capacity to think.
A Small Blessing For Friction
We have spent the last half-century trying to remove friction from human life. One click and presto- seamless satisfaction. Instant entertainment. Instant answers. Instant gratification. Frictionless everything.
Wonderful. Except friction serves a vital function. Muscles develop because of resistance. Character sometimes develops because things don’t go our way. Relationships deepen partly because other human beings are inconvenient and need us to invest energies to understand them. And thinking develops because understanding complex ideas is difficult. The confusion and the false start- the irritating paragraph you have to read three times and the argument that doesn’t quite work. The moment when you realize the problem isn’t the other fellow’s stupidity but your own assumption- when you realize that struggle (in and of itself) isn’t an unfortunate obstacle on the way to thinking. That struggle is what we call thinking.
So, what happens when we remove the struggle? As someone who has taught thinking skills for four decades, I have a few suspicions. None of them is particularly comforting. But this much I do know: We aren’t asking the question nearly often enough.
What Aren’t We Worrying About?
This, for me, is the interesting question. Every age worries about the obvious consequences of its new technology. The really important consequences often arrive through the side door.
So, forget, for a moment, whether AI is going to kill us. We have plenty of very clever people worrying about that already.
I’m interested in what we’re not worrying about.
What happens to children who grow up never having to be intellectually frustrated? What happens when your AI companion is more patient than your husband, wife, friend or child- after all real, flesh and blood humans, can be terribly frustrating. They disagree. They misunderstand you. They want you to do things you don’t want to do. They sometimes refuse to tell you how wonderful you are.
An AI companion presumably won’t disagree with you or fail to validate your every feeling. So, what happens when machines become easier to love than people? What happens when every one of us has our own personalized and curated universe? Do we still have a shared reality? What happens if we don’t have one?
What happens when political persuasion can be individually tailored—not to demographic groups, but to you? Your fears and your resentments. Your hopes and your weaknesses. Your vocabulary and your sense of humor. Your psychological vulnerabilities.
What happens when companies stop hiring young people to do the boring grunt work because AI can do it? Sure, we will have eliminated the boring grunt work. But that still leaves us with a small problem. The boring grunt work was how juniors became seniors. It was where they made mistakes, learned the ropes, developed instincts, discovered what the textbooks had left out and, eventually, acquired judgment.
Will our young just become very good at writing prompts? And then graduate to supervising agents, designing agent loops, meta-prompting—or whatever new skill AI requires of us next? And if AI does the work through which novices once became experts, how will novices become experts? Will we no longer require them?
And here’s another one.
What happens when AI-generated voices, faces and behaviors become indistinguishable from those of real people? How will you know whether the person—or photograph, voice, video, essay, lover, adviser, political activist—on the other side of the screen actually exists?
I don’t know. Neither does anyone else.
But an even more unsettling question keeps me awake at night: Will anyone even care anymore?
Perhaps We Need a Department of “Then What?”
Every large technology company probably has thousands of engineers. Product designers, data scientists, strategists, lawyers, marketing people and finance people.
Perhaps every one of them should also have a small department whose only job is to be annoying. The Department of Then What? Someone proposes a brilliant idea. They ask: Then what?
We can make children never struggle with homework again. Then what? We can create an AI companion that understands you perfectly. Then what? We can automate 40 percent of junior professional work. Then what? We can personalize every person’s information environment. Then what? Everyone would hate this department. Which is probably how we’d know it was doing its job.
The Genius of Not Doing Something
One of the smartest people I knew in college—who went on to a stellar career as a world-wide expert on innovation—once mocked me for suggesting that perhaps technological development needed limits.
“Technology is simply the next phase of human evolution,” he said. “Of course, some people will suffer. But humanity will go on. You can’t put limits on human creativity. If we’d listened to the Luddites, we’d still be living in the Stone Age.”
It was a powerful argument. It was also precisely the problem. We had come to regard our ability to invent something as sufficient justification for inventing it.
I sometimes wonder what he makes of all these silly people worrying about AI destroying humanity. Luddites, presumably.
We live in a civilization that is obsessed with intelligence. We measure it, test it, credential it, hire it, reward it and fund it. And, not surprisingly, we have spent the last few decades valorizing the twenty-something Ivy League dropout with a laptop, an app and absolutely no idea what his invention might do to society.
We call him a visionary. If he attracts a few million users, we call him a genius. If he attracts a few billion dollars, apparently we are no longer permitted to ask whether the thing he built was a good idea. Entire generations of extraordinarily bright technical people have been encouraged to ask: Can I build it? Can I scale it? Can I monetize it? Can I become fabulously rich before I’m thirty?
Rather less prestige attaches to the poor fellow in the corner asking: “Yes, but what will this do to children, families, communities, work, attention, relationships or democracy?” I guess there isn’t much venture capital in perhaps we shouldn’t.
And this is not really an indictment of young entrepreneurs. They are responding perfectly rationally to the culture and incentives we created for them. We turned technological innovation and it’s ugly sibling, disruption, into a moral good. Move fast and break things? Sure. Just don’t ask what those “things” might be.
Over the past hundred years, we have broken a great many things in the name of progress. Ancient crafts and traditions. Old industries and institutions. Communities. Entire ways of life. And occasionally, human capacities we had decided had outlived their usefulness.
We gathered extraordinarily intelligent people together, gave them enormous amounts of money and asked them to discover what was possible. Perhaps we should have also hired a few people to ask what was desirable.
Because intelligence solves problems. Wisdom asks whether solving this problem creates another one somewhere else.
Intelligence asks how. Wisdom asks why. Judgment asks: What happens next? And really good judgment asks: What happens after that?
That is what worries me about AI. Not merely that machines may become smarter than human beings. Something much stranger may be happening. Our machines are becoming extraordinarily intelligent at precisely the moment when we are becoming increasingly willing to outsource the difficult work required to develop our own cognitive capabilities. Our judgment.
We should probably think about that. Preferably using our own minds, before asking ChatGPT.
YES, WE CAN. BUT SHOULD WE?
The main danger with AI and technological genius, is our rather unfortunate habit of thinking about consequences after we’ve created them. Perhaps that is the habit we most urgently need to recover. When confronted with some dazzling new possibility, our first thought is usually: Can we do it? And being the astonishingly clever creatures we are, the answer increasingly is: Yes, we can.
Perhaps it is time for the Third Thought: But should we?
A lifetime of being impressed by showy intelligence, shiny creativity and dazzling genius has taught me that the most powerful minds are distinguished by something much quieter. They can ask themselves: “What if I’m wrong?”
Everything before that may be intelligence. Thinking begins there.
The Ten Lies that Killed Democracy
by Ashok Panikkar
Preface: The Ten Lies that killed Democracy
The First Lie: Liberalism //The Second Lie: Cosmopolitanism // The Third Lie: Progress // The Fourth Lie: Freedom and Liberty // The Fifth Lie: Justice // The Sixth Lie: Open-Mindedness // The Seventh Lie: Equality // The Eighth Lie: Power // The Ninth Lie: Diversity // The Tenth Lie: a. Proving Truth, b. Destroying Truth, c. The Return from Woke to Reality
Conclusion: The Mother of all Lies
Thank you for republishing my article, Daniel.
👏👏👏 An excellent article by the great Ashtok Panniker on the potential dangers of AI! I was beginning yo think you’d retired from Substack! The problem we humans have is our ambition outpaces our better judgment. I will quote Dr. Ian Malcolm from Jurassic Mark for this one, “Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." We need to stop looking at AI as some folks do, either as an unmitigated good or an unmitigated bad. Neither is the case! AI can be a good thing or a bad thing, it depends on how we use it and program it. We need to use it wisely and carefully! We also need to put strict regulations on its use as well. The lack of nuance surrounding the public discourse in the West on AI doesn’t surprise me at all. I think it is time for one of my nuance exercises again.
I think dinosaurs are cool and fascinating. But I wouldn’t want them to be brought back to life. I think Robert DeNiro is a good guy and a great actor, but I would prefer he stop talking about politics and going on his rants about Trump. I applaud the AfD for standing with Israel but I don’t like their reasons for doing so. I think Edward VIII and Wallis Simpson meeting with Adolf Hitler was a HUGE mistake. But I don’t think either of them are as bad as they’re made out to be and sympathize with them. I think that Ulysses S. Grant was a great general but a mediocre President. I recognize that Theodore Roosevelt was racist and had an imperialist streak. But I also admire him and consider him one of United States’ greatest Presidents. I admire Winston Churchill while criticizing him for how bad the Gallipoli Campaign went. I think Hollywood movies generally are garbage. But I thought Oppenheimer and Toy Story 5 were both great movies. I’m all for protecting endangered species of animals but I don’t think we should bring species who’ve gone extinct back. That just opens up a whole new can of worms.
Let’s say I’m living in Germany during World War I. I would be a patriotic German, would proudly fly my country’s flag and sing the German national anthem, love traditional German culture and food, enthusiastically celebrate Oktoberfest, support our soldiers as they went off to war, I would admire the Hohenzollern Dynasty, would be furious with the British Navy for their blockade of our country, and help treat German soldiers who’d been wounded in combat. All that being said, I would oppose German entry into World War I, despise Kaiser Wilhelm II and protest for democracy in Germany, be appalled by the atrocities committed by German soldiers in Belgium, be shocked and revolted by the German Army’s use of poison gas against Allied soldiers, tout the service of African Colonial Soldiers, defend German Jewish soldiers, support women receiving the right to vote and run for office in 1918, urge Germany to surrender in 1918, and rejoice at the coming of the Weimar Republic.
Alright, let’s say I lived in Egypt in the 1940s, 1950s and 1960s. I would be an Egyptian patriot, revere my Ancient Egyptian ancestors, be proud of my country’s thousands of years of history, look up to the Pharaohs of old, denounce British exploitation of our country, call for gradually moving towards full sovereignty for Egyptians over their country, denounce British atrocities and injustices against my people, advocate for gradual independence and sovereignty for other Arab countries, have sympathy for the Palestinian refugees, cheer the Construction of the Aswan Dam, and embrace my Syrian, Jordanian, Iraqi, and Lebanese brothers. However, at the same time I would admire and support the British Empire, generally think positively over their influence on our country, tout the benefits of British rule and involvement in Egypt, love and respect King Farouk, refuse to work with the Nazis and support the Allies in World War II, oppose the 1952 Revolution, be horrified by Gamal Abdel Nasser and denounce him as a madman, welcome the birth of the new state of Israel, refused to serve in the Egyptian Army during the 1948 Arab-Israeli War and the Six-Day War, I would be a proud Muslim Zionist, I would urge Egypt to make peace with Israel and engage in peaceful trade and commerce with the Jewish state, I would call for the Suez Canal to remain in British hands and for their buffer zone and troops to remain their, cheer on the British, French and Israelis during the 1956 Suez Crisis, embrace and help protect the Egyptian Jewish community, oppose the expulsion of British, French, Greek, Maltese, and Armenian foreign nationals and lobby the government to let them stay, applauded women getting the vote in 1956 and would call to eliminate all restrictions on voting for them, and would urge Egyptians to see themselves as Arabized Africans and to push back against Pax Arabica.
Last example, let’s say I lived in Japan in the 1920s, 1930s and 1940s. I’d be a patriotic Japanese citizen, be proud of my Samurai ancestors, revere Emperors Taishō and Hirohito, I would be frustrated that Japan didn’t get what it deserved at the Versailles Conference for helping the Allies win the war, I would love Japanese culture and traditions, cheer the arrival of universal male suffrage in 1925, enthusiastically support modernization and parliamentary democracy in Japan, help Japanese tenet farmers during the Great Depression, and love seeing tradition and modernity being blended together in Japan. But at same time, I would be most wary of the rise of nationalism in my country, curse the ultranationalists in the Japanese military for taking over the country, oppose strict media censorship, be appalled by the Japanese invasions of Manchuria and China, express my shock and revulsion at the war crimes the Japanese Army committed in China, oppose Japan pulling out of the League of Nations, detest General Hideki Tojo and his government, oppose the Tripartite Pact with Nazi Germany and Fascist Italy, urge good relations and friendship between the United States and Japan, condemn the attack on Pearl Harbor, be appalled by Japanese war crimes against Asian civilians and Allied POWs, oppose Tojo’s successors as Prime Minister, would be a proud Japanese dissident who’d send valuable intelligence to the United States and the Western Allies, would urge Japan to surrender after the dropping of the two atomic bombs and the Soviets entered the War in the Pacific, and would welcome the American occupation.