Source record
The AI Dilemma
Who Aligns the People Who Align AI?
The Uncomfortable Question
An AI assistant can provide accurate information, follow its instructions perfectly, and still make recommendations that do not serve your best interests.
Imagine asking AI to help you choose a financial product.
It analyzes your situation, compares available options, and recommends what appears to be the best solution.
But suppose the company operating that AI earns a commission when you select a particular provider.
Would that influence the recommendation?
Not necessarily. But how would you know?
When advice becomes a business, we need to understand whose interests are behind it.
I don't believe profit is the problem. Profit drives innovation, rewards successful businesses, and creates economic opportunities.
The problem begins when commercial incentives influence decisions presented to customers as independent advice.
That distinction existed long before artificial intelligence. What makes AI different is the growing role it can play in our decisions.
We are building systems that do more than provide information. They can evaluate alternatives, recommend actions, and increasingly execute tasks on our behalf.
The more decisions we delegate to AI, the more important it becomes to understand who determines its objectives.
Who aligns the people who align AI?
1. How We Arrived Here
Major technological revolutions change the economics of things that were previously scarce, expensive, or difficult to access.
The internet dramatically reduced the cost of distributing information. Search engines made it easier to find. Social networks transformed human attention into a valuable commercial resource.
These innovations created enormous opportunities. They also introduced new economic incentives.
When advertising revenue depends on engagement, attracting and retaining attention becomes a business priority. When personal information improves advertising performance, collecting and analyzing that information becomes economically attractive.
The consequences are no longer theoretical.
In 2024, the US Federal Trade Commission (FTC) documented extensive collection and monetization of personal information by major social media and video platforms. Its findings exposed significant tensions between their business models and user privacy. [1]
This history tells us something important.
Economic incentives shape how technologies evolve, sometimes in ways their users never anticipated.
Now we are entering another transformation.
Social platforms learned to compete for our attention and influence what information we encounter.
AI can participate more directly in how we interpret that information, evaluate alternatives, and make decisions.
The difference becomes especially important when AI systems are authorized to act.
A recommendation algorithm may influence which financial article you read. An AI assistant may evaluate your finances, recommend an investment, and eventually execute a transaction you approve.
Both influence behavior. But the second becomes more deeply involved in the decision itself.
The more decisions we delegate to AI, the greater the consequences of the incentives shaping its behavior.
This is where the economics of AI becomes a question of human autonomy.
2. When Intelligence Becomes a Business Model
Much of the AI safety debate focuses on alignment: ensuring that AI systems behave according to appropriate human intentions, instructions, and constraints.
This is essential. But technical reliability cannot resolve every conflict of interest.
An AI system may function exactly as designed while pursuing objectives that do not fully serve its users.
Economists have studied similar problems for decades.
Whenever someone makes decisions on another person's behalf, differences in incentives and access to information can create conflicts.
A financial adviser may know more than a client while receiving additional compensation for recommending certain products. An online platform may provide a valuable free service while generating revenue from information about its users.
AI does not invent these relationships.
It can make them more personalized, scalable, and difficult to recognize.
In January 2025, the FTC reported that pricing intermediaries could use detailed consumer information, including browsing behavior and location, to tailor prices and offers. The findings concerned the capabilities and practices under examination, not evidence that every retailer uses personalized pricing. [2]
Now consider what happens when similar commercial incentives operate through conversational AI.
The system understands your preferences, responds to objections, and adapts its explanations.
Used responsibly, this can improve education, decision-making, and access to professional knowledge. Used differently, the same capabilities can make persuasion more effective.
A 2025 study published in Nature Human Behaviour found that GPT-4, when provided with personal information, was more persuasive than human opponents in controlled debates. [3]
However, an author correction published in September 2026 clarified an important limitation: the direct comparison did not establish a statistically significant persuasive advantage for personalized GPT-4 over GPT-4 without personalization. [4]
The research demonstrates AI's potential for persuasion in specific experimental conditions. It does not prove widespread manipulation or establish how these effects translate to everyday interactions.
But the underlying capability deserves serious attention.
AI does not need personal ambitions to influence human decisions. It needs objectives, information, and the ability to act on them.
Humans and institutions establish many of those objectives.
And here we encounter another problem.
Whose interests should an AI system prioritize when those interests conflict?
A user may want maximum convenience and personal freedom. A business may prioritize profitability and efficiency. Governments have responsibilities concerning public safety, while society expects fairness and protection from harm.
These interests can overlap, but they are not always compatible.
The challenge is not to make AI serve everyone's interests equally. That may be impossible. The challenge is to make decisions about whose interests take priority transparent, accountable, and open to challenge.
That is why the question of alignment cannot end with the behavior of the machine.
It must also include the incentives and responsibilities of those designing and deploying it.
Because incentives matter even when intentions are good.
3. The Power Behind the Intelligence
There is another economic reality we need to confront.
Who controls the infrastructure on which increasingly important decisions depend?
Developing advanced AI requires substantial computing resources, capital, engineering expertise, and distribution capabilities.
Access to these resources is uneven.
In 2025, the FTC examined partnerships between major cloud providers and AI developers. It identified potential competition risks involving access to computing resources, contractual dependencies, and the costs of switching providers. [5]
The UK's Competition and Markets Authority (CMA) has raised similar concerns about companies controlling critical resources and major distribution channels. [6]
These findings do not mean that AI markets will inevitably become monopolies. But they reveal how technological advantages can translate into economic power.
Consider a company that integrates one AI provider into customer service, product development, financial analysis, and internal operations.
Initially, productivity improves. Costs decline. Employees gain new capabilities.
Over time, however, the company's processes become dependent on that provider.
Changing systems may require rebuilding integrations, retraining employees, and disrupting operations.
The business has gained capabilities, but switching providers may have become more difficult.
This is a familiar economic pattern. It is not unique to AI, and dependence on specialized infrastructure is not inherently harmful.
The problem begins when users lose meaningful alternatives.
A business that cannot realistically change providers has less bargaining power.
A person who cannot understand how important AI recommendations are shaped has less control over decisions affecting their life.
AI can make powerful intellectual capabilities more accessible while concentrating control over the infrastructure providing them.
That is a possibility worth examining, not a future we should assume is inevitable.
4. What If AI Does the Opposite?
Now we should challenge the argument itself.
What if AI ultimately reduces concentrated power rather than increasing it?
For centuries, professional knowledge has been expensive.
Individuals and small businesses have depended on specialists and institutions whose expertise they could not easily access or independently evaluate.
AI is changing that relationship.
A small company can perform tasks that once required a much larger team. Students can receive personalized explanations. Entrepreneurs can develop software and analyze markets with fewer resources.
Consumers can compare financial products and question professional recommendations instead of depending entirely on one intermediary.
There is evidence of real economic value.
In Generative AI at Work, published in The Quarterly Journal of Economics in 2025, Erik Brynjolfsson, Danielle Li, and Lindsey Raymond studied the introduction of an AI assistant among 5,172 customer-support workers.
They found that access to AI increased productivity, measured by issues resolved per hour, by 15% on average, with particularly large gains among less-experienced workers. [7]
The results come from a specific working environment. They do not prove that every AI application will produce similar benefits.
But they demonstrate something important.
AI can give people access to capabilities that previously required greater experience, larger teams, or more expensive resources.
Does that mean the resulting economic value will be distributed more equally?
Not necessarily.
A 15% productivity increase does not tell us who receives the additional economic benefits.
They may appear as higher wages, lower prices, increased company profits, or some combination of these outcomes.
The distribution depends on market conditions, bargaining power, competition, and business decisions.
Creating economic value and distributing that value are two different things.
This distinction matters because AI could weaken some of the traditional advantages of large organizations while creating new advantages for those controlling its infrastructure.
Both processes may happen simultaneously.
I believe AI's potential to expand human capabilities is real. And it challenges the assumption that technological dependence necessarily means losing independence.
Modern civilization depends on electricity, medicine, transportation, and countless specialized systems. These dependencies have expanded what people can accomplish.
The question is not whether we depend on technology.
It is whether that dependence expands our ability to act while preserving meaningful choices.
AI could reduce our dependence on traditional intermediaries while creating new dependence on technology providers.
The outcome will depend on competition, access, the availability of alternatives, and the ability of users to understand and challenge important decisions.
5. Accountability Must Be More Than a Promise
History shows a recurring pattern.
New technologies create opportunities. Businesses build models around them. Consumers adopt them. Economic interests grow. Eventually, unintended consequences become visible, and societies try to adjust.
We saw different versions of this process during industrialization and the rise of the internet.
But history does not guarantee that institutions will adapt quickly enough, or that the adjustments will be fair.
That is why I don't believe the solution to the AI dilemma is simply more regulation.
Regulation can protect consumers, but poorly designed rules can also make competition more difficult and strengthen established companies.
And asking corporations to behave responsibly is not enough.
We need accountability that can be tested, not just promised.
Return to our financial AI assistant.
If its operator receives compensation for recommending certain products, that relationship should be disclosed.
Independent reviewers should be able to test whether recommendations systematically favor commission-paying providers, even when customers' needs and circumstances remain the same.
If such patterns appear, the operator should be able to explain them and demonstrate whether the recommendations remain consistent with its stated obligations to users.
The same principle applies elsewhere.
If AI influences hiring decisions, employers should examine whether it produces unjustified discriminatory outcomes.
If autonomous systems can execute transactions, their authority should have clear limits, appropriate monitoring, and mechanisms for intervention.
If a business depends heavily on an AI provider, it should understand the practical cost of switching to an alternative.
These are not abstract ethical aspirations. They are questions that can be investigated.
The US National Institute of Standards and Technology (NIST) already provides a voluntary framework for identifying, measuring, and managing AI risks. [8]
The challenge is turning principles into practices that can be evaluated and enforced where appropriate.
And accountability cannot stop at technology companies.
Governments and regulators exercising power over AI must also remain subject to scrutiny.
Replacing unaccountable corporate power with unaccountable government power does not solve the underlying problem.
The objective is not to eliminate power. It is to prevent power from becoming impossible to question, challenge, or correct.
6. The Human Future: What Happens Next?
Over the next decade, AI may become more deeply integrated into education, medicine, business, and everyday decision-making.
The consequences will depend partly on decisions being made today.
Imagine one direction.
AI capabilities continue improving while users retain meaningful alternatives. Independent evaluation becomes more accessible. Commercial interests are disclosed where they matter. More people gain skills and economic opportunities.
Now imagine another.
Increasingly important decisions pass through a limited number of systems. Switching providers becomes difficult. The interests influencing recommendations remain poorly understood. People receive convenient answers but lose visibility into how those answers are produced.
Neither scenario is inevitable. Elements of both may develop simultaneously.
Technology has never determined its social consequences on its own. Markets, institutions, economic incentives, and human choices have always mattered.
AI will be no exception.
This brings me to three questions I believe we should ask whenever AI becomes part of an important human decision.
Who benefits?
Who bears the risks?
Who holds the power?
The answers tell us something that technological benchmarks alone cannot reveal: how the benefits, consequences, and authority created by AI are distributed among people.
And whether that distribution can be questioned or changed.
The Real AI Dilemma
I believe artificial intelligence can become one of the most powerful instruments of human progress.
But I reject the assumption that technological progress automatically means human progress.
History gives us no reason to accept that.
Technology creates possibilities. People and institutions determine how those possibilities are developed, distributed, and used.
AI may help millions gain access to knowledge, improve their economic opportunities, and make better decisions.
It may also create new forms of dependence and concentrate influence over decisions we increasingly delegate to machines.
Both outcomes are possible.
And the difference will not be determined by good intentions alone.
It will depend on economic incentives, the distribution of power, and whether people retain meaningful ways to understand, question, and change the systems affecting their lives.
We spend a great deal of time asking whether artificial intelligence will remain under human control.
We should also ask whether the humans controlling AI will remain accountable to everyone affected by it.
Because the real measure of AI progress is not simply how intelligent our machines become.
It is whether that intelligence gives people more power over their own lives, or gives someone else more power over them.
Sources & References
1. Federal Trade Commission (FTC), 2024 — Social Media and Data Surveillance
Official FTC report documenting extensive personal-data collection and monetization by major social media and video platforms, and the tensions between their business models and user privacy.
2. Federal Trade Commission (FTC), January 2025 — Surveillance Pricing
FTC findings examining how pricing intermediaries can use detailed consumer information to personalize prices, discounts, and commercial offers.
3. Salvi et al., 2025 — On the Conversational Persuasiveness of GPT-4
Nature Human Behaviour. Experimental research comparing AI and human persuasion, including conditions involving personal information.
https://www.nature.com/articles/s41562-025-02194-6
4. Salvi et al., September 2026 — Author Correction: On the Conversational Persuasiveness of GPT-4
Nature Human Behaviour. Official correction clarifying that the direct comparison between personalized and non-personalized GPT-4 did not reach statistical significance.
https://www.nature.com/articles/s41562-026-02588-0
5. Federal Trade Commission (FTC), January 2025 — AI Partnerships and Investments
Official FTC study examining partnerships between major cloud providers and AI developers, including potential risks involving access to computing resources, contractual dependencies, and switching costs.
6. UK Competition and Markets Authority (CMA), April 2024 — AI Strategic Update
Regulatory analysis of competition in AI markets, including control over critical resources, distribution channels, and potential barriers to market entry.
https://www.gov.uk/government/publications/cma-ai-strategic-update/cma-ai-strategic-update
7. Brynjolfsson, Li & Raymond, 2025 — Generative AI at Work
The Quarterly Journal of Economics, Volume 140, Issue 2. Study of 5,172 customer-support workers reporting an average productivity increase of 15% with AI assistance.
https://academic.oup.com/qje/article/140/2/889/7990658
8. National Institute of Standards and Technology (NIST) — AI Risk Management Framework
Voluntary framework for identifying, assessing, and managing risks associated with the development and deployment of artificial intelligence.