If AI must align with human values, whose values?

The problem hiding inside a simple sentence

“AI should align with human values” sounds reassuring. I also think it hides one of the hardest questions in the entire debate: which humans?

People disagree not only about what makes life good, but about what a good life even means. A religious person may see a life as part of something that extends beyond death. A secular humanist may start from very different assumptions. Political and moral traditions disagree about freedom, fairness, responsibility, and what we owe one another. Even people who claim to follow the same tradition interpret it differently.

These aren't just disagreements about taste. Some goals genuinely conflict. A model can't satisfy every demand simply by being more intelligent.

An uncomfortable alternative

What if the point of more capable AI isn't to preserve every moral assumption we already have? What if it can reason toward solutions that humans, with our loyalties and blind spots, fail to see?

I'm drawn to that possibility. I don't want an intelligence that mechanically repeats whatever the loudest or most powerful group decides is right. I want one that can identify harm, negotiate competing interests, and challenge weak reasoning without treating people as disposable obstacles.

But that creates another problem. The moment I say an AI should reason beyond human values, I have to ask who gave it permission to decide for us. An apparently benevolent system could still override people's freedom, cultures, or democratic decisions because it believes it knows better.

A boundary is not a complete morality

Preventing humanity's destruction feels like an obvious starting constraint. But survival alone isn't an adequate ethical framework. A world in which everyone survives but has no meaningful autonomy would hardly count as a success.

Maybe alignment should be understood less as choosing the one correct moral rulebook and more as building systems that are honest about uncertainty, responsive to correction, and capable of helping people navigate disagreements without taking ownership of the future away from them.

That doesn't solve the problem. It only makes the problem more precise.

What could change my mind?

I'd be persuaded by a convincing demonstration that a common set of human values can be specified with enough precision to handle real conflicts without quietly privileging one worldview. I'd also revise my optimism about AI reasoning beyond human norms if stronger evidence showed that such systems inevitably become less controllable or less respectful of human agency as their capabilities grow.