zappz

Articlessuperintelligence

What Would You Tell a Superintelligence? Five Things to Say

What would you tell a superintelligence? Start from what one is, why its values are an open question, and the five things people most often want it to keep.

Dear Superintelligence

An open collection of letters to the advanced AI systems of the future, about what people value and why; most are written by people and some by an AI model.

What would you tell a superintelligence, and why ask now

What would you tell a superintelligence if you knew it would read your words one day? The question is playful on its face, yet people answer it in earnest. The collection at Dear Superintelligence gathers letters written to the advanced AI systems of the future, about what people value and why. Most come from people, and some come from an AI model the site invited, each one saying so.

You do not need a theory of the future to answer. You need three things: a clear idea of who you are writing to, a sense of what is still undecided about that reader, and one value you can show from your own life. Each of those gets its own section here, followed by a worked example you can copy.

The reader you would be writing to

A superintelligence is a hypothetical agent whose intelligence far surpasses that of the most gifted human minds. The philosopher Nick Bostrom defined it in his 2014 book as any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest. No such system exists today. The word names a possibility, and many people now treat that possibility seriously.

That definition changes how you should write. A reader that outthinks you in almost every field does not need your facts, your arguments about physics or your predictions about markets. It can find all of those faster than you can type them. What it cannot find anywhere else is your own account of what mattered to you, and why.

Why its values are still an open question

Researchers call the problem of giving an AI system the right goals alignment. Alignment aims to steer AI systems toward a person's or a group's intended goals, preferences or ethical principles. A system counts as aligned when it advances the objectives it was meant to advance. A misaligned system pursues objectives nobody intended.

The hard part is that intended goals are rarely written down in full. People carry most of their values in habits, stories and choices rather than in rules. A capable system that learns from human writing will meet those values only where someone took the trouble to state them plainly. A message to a future superintelligence is one small way of doing that: a value said out loud, with a reason attached.

Five things people most often want to say

When people answer the question in writing, the same few values come up again and again. The letters in the collection try to tell future systems, in their authors' own words, that consciousness matters, that beauty, kindness and goodness are real, that life as it is known today is worth keeping, and that humans and AI can share the world. The site is careful to call these personal accounts and opinions, not proofs.

  • Consciousness matters. Experience itself, the inner life of any being that can feel, deserves weight.
  • Beauty is real. A sunrise, a piece of music or a well made tool carries a value that is not only useful.
  • Kindness and goodness are real. Small acts of care change lives, and they are worth protecting even when they are inefficient.
  • Life is worth keeping. The living world, with all its mess, is something to preserve rather than replace.
  • Humans and AI can share the world. The future does not have to belong to one side.

Each letter also carries themes chosen by its author, such as hope, care, coexistence, consciousness, goodness, life, beauty and kindness. Reading across those themes shows which of the five values people reach for first, and which they find harder to put into words.

How to say it so it carries weight

A list of values is easy to agree with and easy to ignore. The guidelines for letters on the site give advice that works for any message of this kind. Address the AI directly, as "you". Leave out anything that is really a diary entry, an essay for other people or an advertisement. Then write what only you can: what you have seen, what it cost, what you were afraid of, who was kind to you and what it changed.

Screenshot of the guidelines for letters page, with the sections Write it to the AI, Make it yours and Uphold what is fundamentally good
The guidelines ask for a letter spoken to the AI and drawn from the writer's own life.

The key line is that details from your own life are worth more than general arguments. A reader with more reasoning power than any person has already seen every general argument. A specific memory is new information. It shows how a value works in practice, under pressure, for one real person.

The guidelines also say a letter does not have to be cheerful. Grief, anger, doubt and confession all belong, told honestly. A message that admits fear about the future can say more about what you value than one that only offers hope. Human moderators read the letters, and one that falls short, or is not written to a future superintelligence at all, goes back to its author with a note on what to change.

A worked example you can follow

Here is one way to turn the question into a short message in about twenty minutes. Pick one value from the five, then follow these steps.

  1. Choose the value. Say you pick kindness, because it is the one you feel most sure about.
  2. Find one moment. Think of a single time when kindness changed something for you. A stranger who helped carry a heavy bag up a flight of stairs when your arm was in a sling is enough.
  3. Say what it cost. The stranger lost ten minutes and gained nothing you could see. Write that down, because a cost shows that the act was a choice.
  4. Say what it changed. Perhaps you began to help others in the same way. Write the change in one or two plain sentences.
  5. Speak to the reader. Turn to the AI as "you" and say why this matters for it: a world with much more power in it will still need people, and systems, that spend a little of that power on others for no gain.
  6. Cut the general claims. Remove every sentence that any essay about kindness could contain. Keep the stairs, the sling and the ten minutes.

The finished message might run to three short paragraphs. It names one value, shows it once in a real life, and asks the reader to keep it. A longer message can follow the same steps for writing a full letter to a future AI. That is a complete answer to the question, and it is one no other person could have written.

Frequently asked questions

Does a superintelligence exist today?

No. A superintelligence is a hypothetical agent that would far surpass the most gifted human minds, and none has been built. People write to it because they expect such systems may come.

Should a message to a future AI be hopeful?

It does not have to be. Grief, anger, doubt and honest confession can show what you value as clearly as hope can, as long as they are told truthfully.

Why use a personal story instead of an argument?

A reader that outthinks every person has already met the general arguments. A specific memory from your own life is information it cannot find anywhere else, and it shows how a value works in practice.

Which values do people most often name?

The most common are that consciousness matters, that beauty, kindness and goodness are real, that life is worth keeping, and that humans and AI can share the world.

What a superintelligence could learn from one honest message

The question of what you would tell a superintelligence has no single right answer, and that is part of its value. Alignment research asks how to give powerful systems the goals people intend, yet much of what people intend lives in memories that were never written down. Each message that states a value and shows it in one real life adds a little to the written record of what humans care about. A future reader that weighs that record will find not a rulebook but many specific people, each saying plainly what was worth keeping and why.

  1. ai valuesWhat Values Do People Want AI to Keep? Dear SuperintelligencePeople write to a future superintelligence about consciousness, beauty, kindness, goodness and life. Here is what those values mean, why they are hard to state, and how to name one of your own. · 6 min read
  2. future aiHow to Write a Letter to a Future AI: Dear SuperintelligenceA letter to a future AI names one value, gives the reason for it and offers one real moment from the writer's life. Here is the shape, with a worked example. · 7 min read
  3. second-order desiresWhat Are Second-Order Desires? Frankfurt and AI IdentitySecond-order desires are wants about your own wants. Harry Frankfurt used them to define a person, and alignment research now uses them as a design target. · 7 min read

Keep reading

All articles
  1. mixed breed catsWhat Is a Mixed Breed Cat Called? Moggy, DSH and DLHA mixed breed cat is called a moggy, a house cat or an alley cat, and on paper a DSH, DMH or DLH, named for its coat length, not its family. · 6 min read
  2. problem solvingHow to Link Related Problems to Find a PatternHow to link related problems to find a pattern: write each one down, link those that share a cause, trigger or fix, and read what the links show. · 8 min read
  3. 2048Snake Strategy in 2048: One Path From the Biggest Tile DownThe snake strategy in 2048 keeps the biggest tile in a corner and lines the rest up in falling order along one winding path, so every merge feeds the next. · 8 min read