By living in our world, we collect intuitions on how the physical stuff around us behaves, and on the most practical ways to do particular things. Like fry an egg, bounce a basketball, carry a couch, stay level on bouncing platform, or get a door unstuck. These intuitions come with a sense of their generality, and many of them you feel pretty confident that your intuitions are pretty general. They feel like the best way to do a thing in 99%+ of relevant situations in which humans today find themselves.

But of course our universe is vast, and the physical context in most of that universe is quite different from our usual human contexts in terms of temperature, pressure, acidity, vibrations, radiation, etc. Thus many of these 99%+ reliable general ways to do physical things among humans today likely also fail in 99%+ of the physical environments out there in the universe. Which is to say that our physical intuitions are context dependent; they were developed in a context, and may be reliable there, but not necessarily far outside of it.

Living in our world also gives us moral intuitions, about the best ways to treat other people. And through most of human history, these intuitions were also plausibly mostly about practicality: what ways to treat each other work well, and are adaptive, for individuals and their communities.

We are also often confident that our moral intuitions are pretty general, and indicate the best ways to treat people in 99%+ of related situations that we have seen. And while we do have incentives to exaggerate that confidence, to signal loyalty to our associates, we are also plausibly often right about such judgments.

However, few of us have seen more than a tiny fraction of all the social situations that humanity has ever seen. And relative to all the social situations that our descendants may yet see, the fraction is likely far smaller. Thus few of our moral intuitions can be anything like fully general, working consistently well across the vast space of possible social worlds. Yet many mistakenly talk and act as if their intuitions were that general.

For example, many say it could never be moral to enslave human-like minds. (Prison and the draft don’t count as “slavery” to many.) But war was endemic to the ancient world, and war winners there faced a hard choice: if freed, those who they conquered might well seek revenge. The safer options were to kill or enslave the vanquished, and of those slavery seems the most humane option. Thus slavery seems a sensible way to treat those you defeated in war in the ancient world. (In polls, folks say 5-1 that ancients didn’t know “in their hearts” that slavery is immoral, and 5-2 that we don’t know arguments that would have persuaded smart ancients of this.)

As another example, many say that murder is generally immoral (even if they allow exceptions for war and justice). But my book Age of Em describes in great detail a possible future world wherein the dominant minds usually suffer little when a single copy of them is “killed”. They don’t feel physical pain unless they choose to, and archive copies are made of their minds every subjective five minutes. Most mind copies are only a few hours old, and will be erased within a few hours. Unexpectedly erasing such a copy is only a minor inconvenience, analogous to stepping on someone’s toes, or farting nearby for us, minor harms that don’t rise to the level of moral violations.