What Will Super Intelligence Amplify?

This does not mean that machines suddenly crossed some scientific threshold and became superhuman. For now, the change is primarily about the language the government uses to describe the technology.

But I couldn’t stop thinking about the name.

Super Intelligence.

It immediately reminded me of a book I read years ago: Thea Alexander’s 2150 A.D.

The book was published in 1971.

More than half a century later, one of its central ideas suddenly feels remarkably current.

Two Societies in the Same Future

The protagonist of 2150 A.D., Jon Lake, finds himself in the year 2150.

The world around him has changed almost beyond recognition. Technology is deeply integrated into everyday life. A vast system called C.I., or Central Information, gives people access to enormous amounts of knowledge.

But the most interesting part of Alexander’s future is not the technology.

It is the people living with it.

Her world presents two very different ways of organizing human life.

There is Macro Society.

And there is the Micro world, represented by Micro Island.

Macro is built around personal responsibility, broader awareness, cooperation and human development.

Micro feels much more familiar: fear, competition, status, division, control, the struggle for power and the need to prove that I am right.

And here is what I find most interesting:

Both exist in the same technologically advanced future.

They have access to extraordinary capabilities. Yet they live very differently.

The difference is not simply the technology. The difference is the people and institutions using the power that technology gives them.

Over the past year, I have written a lot in this newsletter about AI.

What happens when machines begin performing tasks we once considered uniquely human?

What happens when we delegate not only work, but parts of our thinking?

What happens when thinking for ourselves becomes increasingly optional?

But returning to Alexander made me think about another question.

In her imagined future, technological development exists alongside an ambitious idea of human development.

Our experiment looks different.

We are rapidly increasing the capabilities of our machines while still trying to understand what human qualities will be necessary to live responsibly alongside them.

Machines are becoming more capable. They can analyze more information, remember more, persuade more effectively and operate with increasing autonomy.

The humans using them remain human.

We are still influenced by fear, status, greed, insecurity, tribal identity, the need for recognition, the desire to be right and the temptation to seek evidence that confirms what we already believe.

That does not make us bad.

It makes us human.

But these are the humans who will have access to increasingly powerful intelligence.

And there is a simple distinction that may become increasingly important:

Intelligence is not the same as wisdom.

A highly intelligent person can use intelligence to see reality more clearly.

The same person can use that intelligence to build an extraordinarily sophisticated defense against reality.

More intelligence does not automatically make us wiser. Sometimes it simply makes us better at defending what we already want to believe.

What Exactly Are We Giving to the Machine?

Our first uses of AI were relatively straightforward.

Write this. Translate this. Analyze this. Summarize this.

I use these systems constantly myself.

And there is nothing new about letting tools carry part of our mental work. Humans have always extended their minds through language, writing, books, calculators, libraries, search engines and other people.

The question is not whether we should use tools to think.

Of course we should.

The more interesting question is:

Which parts of thinking should we keep for ourselves?

Because increasingly, we are asking machines to do more than help us do things.

We are asking them to help us decide what to think.

What does this mean?

Who is right?

What decision should I make?

How should I respond?

What should I believe?

Sometimes even:

Who am I?

And the risk is not simply that the machine might give us the wrong answer.

We may gradually outsource something that happens before the answer.

Doubt. Uncertainty. Contradiction. Reflection. The effort of seeing a situation from another perspective. The uncomfortable possibility that we may be wrong. Research from Microsoft and Carnegie Mellon suggests that higher confidence in GenAI is associated with less critical thinking effort, while the nature of critical thinking itself shifts toward verification and oversight.

I increasingly find myself returning to a different question.

Not:

What should I think?

But:

Why do I want this particular answer?

Why do I need to be right?

Why does this explanation feel so satisfying?

Why did I ask the machine this question rather than another one?

A machine can analyze the question I give it.

But it does not automatically know why I chose that question in the first place.

And this is where Super Intelligence becomes especially interesting.

It may become extraordinarily effective at helping us pursue goals whose motives we ourselves have never examined.

Imagine two people with access to exactly the same extraordinarily capable system.

The first asks:

“What evidence could show me that I am wrong?”

The second asks:

“Give me the strongest evidence that I am right.”

Both may receive sophisticated arguments.

Both may learn something.

Both may leave feeling smarter.

But they have used intelligence for fundamentally different purposes.

This is why I increasingly think of advanced AI as an amplifier.

It can amplify curiosity, creativity, learning, scientific discovery and our ability to understand perspectives beyond our own.

But it can also amplify self-deception, manipulation, propaganda and control.

The uncomfortable part is that the system may perform brilliantly in both cases.

More intelligence does not tell us which goals are worth pursuing. We still have to decide that.

The Same Intelligence. Different Institutions.

Now take the same idea beyond individuals.

Imagine two companies using equally powerful SI.

In one, employees are rewarded for identifying problems, questioning assumptions and telling leadership uncomfortable truths.

In the other, disagreement is punished and people quickly learn to tell leadership what it wants to hear.

Give both companies the same technology.

It may make both more capable.

But what exactly will it make them more capable of?

Now imagine two schools.

One teaches students to question, investigate, compare evidence and develop their own judgment.

The other is focused primarily on producing the correct answer as quickly as possible.

SI could help both.

But in one system it could deepen learning, while in the other it could make independent thinking increasingly unnecessary.

The same applies to governments.

Systems capable of analyzing enormous amounts of information could improve transportation, healthcare, education and public services.

Those same capabilities could also make surveillance, persuasion and centralized control far more effective.

Same intelligence. Different institutions. Different consequences.

Technology does not arrive in an empty space.

It enters cultures, companies, schools, governments and families that already have their own incentives, weaknesses and values.

And rather than automatically transforming those systems for the better, technology can scale what is already there.

Technology does not erase the culture into which it arrives. It scales it.

One of the central questions surrounding advanced AI is alignment.

How do we make increasingly powerful systems act in accordance with human values?

It is an essential question.

But underneath it is another one.

Which human values?

The ones we say we have?

Or the priorities revealed by our behavior?

We say we value truth.

But sometimes we prefer reassurance.

We say we value freedom.

But we also value convenience.

We say we value privacy.

Until giving away some of it makes life easier.

We say we value independent judgment.

Yet increasingly, algorithms help determine what we read, watch, buy and pay attention to.

Human values are not simple. They often conflict.

And our behavior does not always match the values we claim to hold.

We can tell a machine that we value truth, freedom, privacy and independence.

But our actual choices may reveal something more complicated.

We choose convenience over privacy.

Confirmation over doubt.

A quick answer over the uncomfortable work of thinking for ourselves.

So perhaps the alignment problem will teach us something not only about machines.

It may teach us something about ourselves.

Do we want SI aligned with the values we declare, or with the priorities we repeatedly reveal through our behavior?

Some of those revealed priorities may be uncomfortable to call our values.

Macro and Micro Are Inside Us

There is one part of Alexander’s framing that I would reinterpret.

I don’t think humanity can be neatly divided into Macro people and Micro people.

Both modes exist inside us.

Sometimes I use AI because I genuinely want to understand something.

Sometimes I want confirmation that I am already right.

Sometimes I am looking for truth.

Sometimes I am looking for ammunition.

Sometimes I want another perspective.

Sometimes I want a better intellectual defense around something I already believe.

The same is true of institutions.

A company can encourage curiosity in one situation and punish disagreement in another.

A society can build extraordinary institutions of science and cooperation while becoming more polarized and manipulative elsewhere.

Macro and Micro are not two kinds of people. They are two ways of using intelligence and power.

And SI may dramatically increase the consequences of whichever mode we choose.

There is another possibility I keep thinking about.

For most of human history, access to sophisticated knowledge and analysis was expensive.

Expertise required years of training.

Research required time.

Knowledge depended heavily on access to books, universities, laboratories, experts and institutions.

AI has already begun changing that.

Super Intelligence, if it truly arrives, could change it much more dramatically.

Imagine a world where sophisticated cognitive assistance becomes cheap, fast and widely available.

Where almost anyone can access a system with more knowledge than any individual expert could hold and capable of reasoning alongside us.

What becomes the bottleneck then?

Perhaps it is no longer access to intelligence.

Perhaps it is judgment.

Attention.

Responsibility.

The ability to decide what actually matters.

The ability to understand what we really want before asking a powerful system to help us get it.

The ability to recognize when a persuasive answer feels convincing simply because it happens to be the answer we hoped to hear.

Perhaps wisdom becomes more valuable precisely because intelligence becomes abundant.

There is something almost ironic about that.

We may spend decades building machines capable of answering almost any question, only to discover that one of the most important human abilities is knowing:

Which questions are worth asking?

What Exactly Will It Amplify?

The arrival of Super Intelligence does not make human development less important.

I think it makes it more important.

The more powerful our tools become, the more consequential our motives become.

The easier it becomes to answer “How?”, the more responsibility shifts toward “Why?”

Why do I want this?

Why am I building this?

Why am I asking this question?

What am I refusing to see?

What would make me change my mind?

Perhaps Thea Alexander’s most interesting idea was not the futuristic technology she imagined more than fifty years ago.

It was something much simpler.

Technological development and human development are not the same process. And one does not guarantee the other.

We are beginning to call our machines Super Intelligence.

But the name itself tells us nothing about what will happen to the humans living alongside it.

Will we become more curious, or simply more certain that we are right?

More independent, or easier to influence?

Better at understanding reality, or simply better equipped to defend the version of reality we already believe?

Perhaps the most important question is not how intelligent our machines will become.

It is what happens when that intelligence meets us.

Our curiosity. Our creativity. Our ambition. Our fear. Our insecurity. Our generosity. Our desire to understand. Our desire to be right.

Because Super Intelligence may not simply change us.

It may make what is already inside us far more powerful.

And that leaves us with a question that may matter more than the name itself:

When Super Intelligence amplifies us, what exactly will it amplify?