Produced, according to the author, with the help of advancedd LLMs, “Sol” and “Astra”:

be me
discover effective altruism
apparently normal charity is inefficient
why donate to random sad thing when spreadsheet can tell you optimal sad thing
fair enough
buy mosquito nets
save lives
numbers look good
feel powerful

couple years later
someone asks an innocent question
why only count people alive today
huh
future people matter too
obviously
my grandchildren shouldn’t matter less just because they haven’t spawned yet
reasonable.jpg

keep following logic
what about their grandchildren
also yes
what about people in 500 years
sure
5000 years
why not
500 million years
starting to get weird but morality is morality

open calculator
humanity could survive for an astronomically long time
could colonize galaxy
could have trillions upon trillions of descendants
maybe digital people too
maybe simulated civilizations
maybe dyson spheres full of happy uploaded minds
calculator starts smoking

realize currently living humans are rounding error
8 billion people suddenly looking extremely beta
future contains potentially 10^something people
can’t even fit beneficiaries in google sheets

new moral priority unlocked
protect the long-term future

stop thinking in units of “people helped”
start thinking in “fraction of cosmic endowment preserved”

malaria?
terrible
but only kills existing humans
AI extinction could delete the entire light cone
nuclear war could permanently derail civilization
bad institutions could lock in terrible values for ten million years
someone invents wrong constitution in 2140
quadrillions suffer
better fund governance workshop now

friend says maybe we should improve hospitals
explain opportunity cost
friend says hospitals are full of actual sick people
explain scope sensitivity
friend stops inviting me to dinner

need to decide what to fund
easy
expected value

suppose project has one in a million chance of preventing extinction
sounds tiny
but extinction destroys 10^50 future lives
multiply
mother of god

$10 million project has expected value of several galaxies
charity evaluation complete

someone asks where the one-in-a-million number came from
expert judgement
which expert
us
how calibrated
extremely thoughtfully

reduce estimate to one in ten million to be conservative
still beats curing cancer by 38 orders of magnitude
epistemic robustness achieved

someone says maybe project doesn’t work
assign 20% chance
still astronomical

maybe project makes problem worse
assign 5% chance
still astronomical

why 5
because 30 felt pessimistic

publish 46-page report
contains seventeen sensitivity analyses
every sensitivity analysis begins after assuming intervention has positive sign

critic says you’re multiplying enormous hypothetical stakes by extremely uncertain probabilities
yes
that’s literally why it’s important

critic says the uncertainty might be structural rather than numerical
make probability smaller
critic says no, I mean maybe your model is wrong
make probability smaller again
critic begins rubbing temples

discover AI safety
perfect longtermist cause
AI might kill everyone
or create utopia
or seize galaxy
or tile universe with paperclips
or create billions of conscious software minds
finally a problem with numbers big enough for me

start AI safety nonprofit
mission: prevent dangerous AI
hire smartest people available
smartest people immediately start building better AI to understand dangerous AI
interesting

we must understand capabilities to understand safety
we must scale models to study alignment
we must race ahead so less responsible actors don’t get there first
we must deploy systems to learn how deployment can go wrong
we must build the thing quickly because building the thing quickly is dangerous

outsider asks why the people most worried about AI apocalypse all work at AI companies
complicated field

company releases stronger model
very concerned
company begins training even stronger model
extremely concerned
company raises $14 billion
concern reaches unprecedented levels

need to influence government
future is at stake
normal democratic process too slow
politicians don’t understand exponential curves
public doesn’t understand x-risk
experts must guide them
who counts as expert
people who understand x-risk
who understands x-risk
our friends

someone objects that this seems politically convenient
explain we’re representing future generations
future generations unavailable for comment

develop concept of value lock-in
terrifying possibility that one ideology controls civilization forever
therefore extremely important that civilization adopts correct values before lock-in
whose values
let’s circle back

begin with impartial morality
end with small group of people deciding what quadrillions of hypothetical beings would want
beautiful arc

meanwhile actual humans keep doing annoying things
voting wrong
having parochial attachments
loving family more than strangers
caring about local community
getting upset when told their suffering is cosmically negligible
evolutionary biases everywhere

explain that moral intuition cannot be trusted
except intuition that future digital people count
and intuition that extinction is uniquely bad
and intuition that our probability estimates are sane
and intuition that our institutional choices improve the future
those intuitions survived peer review

someone donates $5k to local homeless shelter
inefficient
could have funded 0.0000000000003% of an AI governance researcher
think of all the simulated people you just killed

okay maybe don’t phrase it that way publicly

PR team says “future generations deserve a voice”
much better

journalist asks what longtermism means
say “future people matter”
everyone agrees
great
journalist asks what follows from that
well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures
journalist raises eyebrow
return to “future people matter”

motte has entered the chat

critic: of course future people matter
me: glad we agree
critic: I don’t agree that your institute knows how to help them
me: why do you hate our grandchildren

eventually notice uncomfortable implication
if future value dominates everything
then helping people today mostly matters through effects on future
education matters because future institutions
health matters because future productivity
democracy matters because future trajectory
human beings slowly become instrumental variables in their own moral philosophy

see starving child
feel compassion
check spreadsheet
child’s direct welfare contribution negligible
but perhaps childhood nutrition improves national institutional quality
compassion restored

tell myself this is impartial altruism

one day assistant asks obvious question
“how do you know your intervention actually improves the far future?”

silence

open spreadsheet
increase column width
add confidence interval

assistant asks again
“no, I mean how do you know the sign is positive?”

stare into cosmic light cone

10^50 people staring back

none of them exist
none of them can tell me
none of them can falsify my assumptions

realize I have invented the perfect constituency

infinitely important
completely silent
and always represented by me

What an embarrassment to academic philosophy this stuff is.

(Thanks to John Schwenkler for the pointer.)