SlateStarCodex Virtual Meetup Rooms

Thanks for attending the Zoom talk. Please join us in one of the chat rooms below. If you have trouble with Hubs, see the FAQ at the bottom of the page. If a room is full, please come back to this page and move to the next room. Don’t stay in the lobby as too many in the lobby can cause lag:

1. hub.link/Zd85BZs

2. hub.link/q7PLxgT

For those who don’t like Hubs, we also have this ChatRPG room:

3. ChatRPG Beta: https://chatrpg.pw/btz7h20v

Hub’s FAQ:

Is VR equipment required?

No. VR equipment is supported but not required. Most will not be using such equipment. 

How do I move in Hubs?

This link here covers the full controls: 

https://support.mozilla.org/en-US/kb/hubs-keyboard-and-mouse-controls

If you have a mouse, please use that when in Hubs. A trackpad is workable but a mouse works much better, as Hubs requires you to click when moving your view, which can be difficult with a trackpad.

In the event you don’t have access to a mouse and your trackpad proves cumbersome, “Q” and “E” can also be used to change your orientation. 

How do I change my name/avatar in Hubs?

This can be done in the hamburger menu on the top-left side of the screen under “Set name and avatar.”

I request everyone avoid the “realistic” human avatars as well as Quantum Fox and the faceless man, as the movements they make to indicate their wearer is talking are far too subtle. The robots, bobble-headed humans, square-headed entities of indeterminate origin, and animal avatars all do this much more effectively. 

It would be nice to minimize duplicates, so try to pick an avatar you think few people will pick even after you take into account the fact that they, too, have read this sentence.

Then click the hamburger menu on the top-left and click “Set name and Avatar”. While you are there, please change your name to your real name or SSC/LessWrong handle.

Once done, make sure to click your name in the right-hand corner and register with your email to save your settings.

Hubs does not run well on my computer, anything I can do?

You can join the gather.town room here: https://gather.town/9L7wvxwAd3eIYsuH/ssc5

Audio issues, why can’t people hear me?

The volume of a speaker decreases with distance in Hubs. Try getting closer to people.

If you are still having audio issues, as always with such things test your audio and mic by recording a memo on your computer’s default memo software. 

If that works, then the next most likely problem is your browser has not granted permission to Hubs to access your headset and mic. If you click the lock icon on the top-left of your address bar, permissions can be adjusted there in both Chrome and Firefox.

What should I do if I meet a troll?

If someone is bothering you, you can mute them selectively by holding your space bar while looking at them.

If you hide or mute someone by mistake, you can undo this by selecting their name on the top-right menu. 

How do I enter the room?

To enter the room without VR, go to on of the links at the top of this page. Then just click “Enter Room” and “Enter on Screen”.

If you are using an Oculus Quest that is not in link mode, click “Enter on Standalone VR device” and follow the instructions on screen. 

If you are using a PC-connected VR device (including an Oculus Quest with Quest Link enabled), switch your browser to Firefox and go to one of the links above. Then click “Enter,” then “Connected VR headset.” 

Can I hang out in your Hubs room outside of official meeting hours?

The SSC/LessWrong Meetup room is always open. Feel free to join it any time. But note, any objects you create will be deleted before the next meeting. 

I have a custom avatar; can I use it?

Hubs supports the import of GLB files for avatars. If you have one you would rather use over the built-in avatars, feel free to do so.

Are there any rules on what avatars we can choose?

Because someone came in the form of a dachshund the size of a small apartment building, I have instituted a rule that you cannot have an avatar larger than an SUV. 

But I feel most comfortable when instantiated in a body of supernormal size. Can you make an exception?

Is an SUV not supernormal enough to satisfy this bizarre impulse? 

Let’s say it isn’t.

Well, you see, your size imposes a negative externality on other users, decreasing their enjoyment of the meeting. We have to account for that. So unfortunately, I can’t make an exception. 

What if the utility I gain from being a dachshund the size of a small apartment building exceeds the summed decrements in utility this causes others?

Well…then we are all in luck. You see, the Hubs metaverse is scaled such that the average avatar, were they transplanted into our world, would be the size of a small apartment building. It is merely an illusion of perspective that most avatars do not appear to be the size of a small apartment building. Thus, merely by showing up as a dachshund you will gain the utility you crave while avoiding imposing upon others.

What if the utility I gain from being a dachshund that, relative to the median avatar, appears to be the size of a small apartment building exceeds the summed decrements in utility this causes others?

Then you would be a sort of positional utility monster. Unfortunately, here at the SSC/LessWrong Mozilla Hubs Meetup we have precommitted to following a strict deontological rule which obliges us to ban all such creatures. 

Share this:

The State Pension


First Post on statepension.blog:

November 1st, 2020 entry:

When the physicists announced there were many worlds and many “mes” and many “yous,” I wasn’t sure what to think. But I was inclined to believe them. If we are to listen to scientists about the quarks and gluons and genes and evolution, we have no course for denying them in this particularity. After all, who are we average mortals to gainsay these giants? It is they who can level cities with a mere equation. One of many feats they’ve accomplished that were previously reserved for God.

And what have we ordinary people done to compare? I cannot answer for you, but all I did was write columns for The Daily Mail before being replaced by an algorithm. I then lost the capital I had saved for my retirement in foolish speculations.

I can’t say this attitude isn’t troubling, though. One feels oddly passive in this age of reason. These eminent scientists casually dispense with our common-sense notions of space, then time, then finally identity itself and (lacking the capacity to understand why in any detail) we nod our heads and hope other people think us cleverer for it.

We mediocre folks must take an extraordinarily large number of things on faith. And I fear this must always be so, for mediocrity is the only disease that cannot be eradicated, even in principle.

Nonetheless, It is because of my faith in science and its processes that I have always been of the opinion that the state pension, though grisly from a naive “one-worlder” perspective is actually a remarkably humane, efficient and (dare I say) generous social program.

If you look back at the columns I wrote before being sacked, you will note I have always cheered on its adoption in other countries, and applauded Prime Minster Clarkson for his sound thinking in implementing it after the blight made it clear that our welfare system was destined for a slow-motion collapse.

The Daily Mail, as you may or may not know, was one of the first newspapers to switch to algorithmic journalism, considered an ideal candidate for reasons beyond my understanding.

It was depressing when the aforementioned giants announced that though artificial intelligence was turning out much more difficult than they thought, artificial pundits could be churned out with almost no processing power whatsoever using one simple trick that, I can confirm, journalists do hate. But such is life.

It was because of this that I was made redundant, the first domino in that chain reaction that lead to my bankruptcy.

But my skills are not entirely without worth: the algorithms, having no inner experience, are still no good at writing biographical stories, and so I think there is still some value in me writing, some ten years after my termination, a series of posts detailing the process of receiving the state pension, and giving you, the reader, an inside view of the process. I will post them here on this blog, this ethereal thing, in the hope that at least one person finds it enlightening.


An except from Thinking and Acting in the World So Large, Elijah Mayer:

It is considered, I must admit, somewhat shameful among professional philosophers to write a book for the common man. To these critics I say this: they will not be so common once I am done with them.


From statepension.blog:

November 3rd, 2020

Tomorrow, the pension. Today, I write.

What I thought I would do with this morning’s post is go over the history and philosophy of the state pension and why my worry is unfounded. I know most of you younger people went over this in secondary school, but is strikes me that some could use a refresher, so bare with me or (if you must) skip to tomorrow’s post where I will go over the details of the process.

Now as you know, during the 1920s physicists were mystified by the behavior of the photon in their famous double-slit experiment. You see, light behaves in a manner that seemed contradictory at the time. You measure it one way and it seems to be made of discrete particles; you measure it another and it seems to behave as a wave does. How, the greatest minds of the age asked, could it behave in both ways at once? The answer seemed to be that this individual photon was in all possible states at once, and thus was interfering with itself to produce its wave-like properties.

This raised a bigger question: if that were true, why don’t we observe such states in macroscopic objects? One of the greatest minds of his generation, Niels Bohr, had an answer unworthy of his genius: wave function collapse. That is, Bohr thought (and many did until the future Nobel laureate Hugh Everett III published his celebrated Ph.D. thesis) that the process of observing and measuring a quantum event solidifies the process that occurred, retroactively, into a definite state. This is manifestly absurd, of course. For measuring devices and observers are themselves quantum processes.

Schrödinger demonstrated this with his famous thought experiment. He wrote:

One can even set up quite ridiculous cases. A cat is penned up in a steel chamber, along with the following device (which must be secured against direct interference by the cat): in a Geiger counter, there is a tiny bit of radioactive substance, so small, that perhaps in the course of the hour one of the atoms decays, but also, with equal probability, perhaps none; if it happens, the counter tube discharges and through a relay releases a hammer that shatters a small flask of hydrocyanic acid. If one has left this entire system to itself for an hour, one would say that the cat still lives if, meanwhile, no atom has decayed. The first atomic decay would have poisoned it. The psi-function of the entire system would express this by having in it the living and dead cat (pardon the expression) mixed or smeared out in equal parts.

It is typical of these cases that an indeterminacy originally restricted to the atomic domain becomes transformed into macroscopic indeterminacy, which can then be resolved by direct observation. That prevents us from so naively accepting as valid a “blurred model” for representing reality. In itself, it would not embody anything unclear or contradictory. There is a difference between a shaky or out-of-focus photograph and a snapshot of clouds and fog banks.

Schrödinger points out there is no clean separation between the world of photons and the world of humans. If you agree with Bohr that microscopic objects can be in an indefinite state until observed, then you must also agree that macroscopic objects, too, (cats cars, and importantly, other people) can also be in this state. What Bohr’s wave function collapse implied was a kind of strange solipsism in which everything is indefinite until you yourself observe it. Thus Schrödinger’s thought experiment was the first volley that lead to Bohr’s eventual loss of influence in the field — At a conference in which Bohr was not in attendance a young Richard Feynman famously quipped, “Can this gathering be said to have occurred if Bohr is not here to observe it?”

But with the collapse of collapse, the field was left with no explanations for this question, even a poor one. It was in this time of confusion that Hugh Everett III made his mark on modern physics, and now many (years down the line) the pension system of the United Kingdom.

What Everett showed in his Ph.D. thesis — and solidified with his later discovery of decoherence (for which he was awarded the Nobel) — was that wave-function collapse is entirely unnecessary so long as you give up one cherished assumption: that experiments have only one outcome.

That is, if you actually just look at what the Schrödinger equation describes, wave-function collapse isn’t needed and the explanation is very clear. There is just one “universal wave function” in which everything that can happen, does happen. In which macroscopic objects, too, are in superposition. Like the photon, all objects are in many states simultaneously. The reason we don’t observe superposition ourselves is because decoherence occurs at microscopic scales, making it impossible for any version of ourselves to experience more than one outcome, as our parallel realities decohere (and are thus unable to interact) long before things on human-scale would be able to notice.

Now, as I said, I’m an average man. So my explanation is more of an elephant’s painting than a portrait of what occurred. My more technical readers, please forgive any blunders.

The important fact is this: there are parallel versions of me and parallel versions of you. Scientifically indisputable. The earth is round, it orbits the sun, and there are infinite versions of everything stacked like a deck of cards throughout the multiverse. There is a version of me still working at The Daily Mail. There is a version of me, happily retired, who laughed off his cousin’s offer to go halfsies on an American alpaca farm. And most importantly, there is a version of me who will survive the state pension and will be much richer for it.

You see, what the bureaucracy of our great nation realized, as the blight was on the point of emptying the state’s coffers, was that if there are infinite copies of every person, then every person is “backed up” as it were. And so by removing some redundant copies, you can make the welfare state vastly more efficient.

That is, suppose the state has only 2 million pounds but needs to pension off 100 people. To an ignorant one-worlder, there would be no choice but to leave your citizens “high and dry.”

But from our many worlds perspective, pensioning off 100 people with 2 million pounds is purely a matter of removing 99 redundant copies from a 100 universe branch.

Now its true that some of the less-educated fellows out there conflate “removing redundant copies” with “murder of the poor.” But we cannot allow scientific illiteracy to bankrupt the state.

And, yes, many have to be coerced into applying for the state pension, but the vast majority of those surveyed afterwards are very happy with the exceedingly generous yearly annuity they are provided with, which in most cases is several times more per annum than they’ve ever made in the best years of their lives.


An except from Thinking and Acting in the World So Large by Elijah Mayer:

What many may not know is quantum suicide started as a mere gedanken experiment, seemingly so harmless. First imagined by Einstein, who had a tremendous gift for this style of thinking. Though he would later become an enthusiastic advocate for Everett’s now-orthodox relative state formulation, the idea first hit him rather poorly; and it was in a characteristically rude missive from Einstein to Everett, in which he suggested Everett volunteer himself for such a trial, that we first see the idea fully formed.


From statepension.blog:

November 4th, 2020

This morning, a very upright young police officer came by my home and escorted me to the welfare office. He has had an excellent set of teeth and smiled often as we chatted while he escorted me out the door.

The handcuffs, I admit, were very uncomfortable.

But I suppose it is not too often an Oxford-educated chap is eligible for the state pension — those who don’t know better must be protected from themselves.

“You confuse me for some foolish one-wolder,” I said as he slipped and tightened the shackles around my wrists, “I have not the least worry in the world.”

He flashed his teeth, tightened the cuffs and said, “I don’t doubt it sir, but we are all slaves to policy.”

I nodded my head in agreement. “Now, you can ask anyone and they’ll tell you you won’t find a more patriotic fellow than me, but I think we’ve all been a little put out by bureaucracy now and then.”

And I wasn’t lying when I said that, I can tell you. Bureaucracy does get on my nerves. Take the TNCs, the tracking and neutralization collars. Now I understand they are for our own safety, and when Clarkson introduced them I was one of their fiercest advocates.

But you see, I developed a nickel allergy several years ago. And you won’t believe the time I had getting my TNC replaced with a hypoallergenic model. I had to fill out reams of paper work and it took months to arrive, and getting it installed at the police station was itself a nightmare. Bureaucracy has gone amuck, it has, when a man with a nickel allergy can’t get a hypoallergenic tracking and neutralization collar installed in a timely manner.

I was so put out commiserating with this public servant that I didn’t even notice as I was lead gently into the back of a white van. The man was a magician of the highest order, for he got me into his police car very smoothly and with minimal fuss.

He placed the bag over my head with great delicacy, too.

The ride was uneventful. The itchy wool bag over my head leaving little to be described. After roughly an hour the van stopped, the door screeched open and the officer said, “Please lean towards my voice.” I did so he plucked the bag from my head. It was when I attempted to respond that I realized he had selectively activated my TNC, paralyzing only those muscles that are required to verbalize.

Though a wonderful invention and responsible for the nigh-elimination or anti-social behavior, I can’t help but think that in this particular instance activating the TNC, even in this limited fashion, was not completely justified.

“Now exit the vehicle and walk towards that yellow door to your left”

I nodded and followed his instructions to the letter. The door lead into a large hallway filled with people who looked very much like me. There were 100 people exactly. Each was standing in front of a red door.

That is, this hallway had 50 red doors on each side, and beside each door a person stood stock still, paralyzed in place. There was, however, one door at the very back that didn’t have a person in front of it. My door.

Without waiting for the officer to instruct me I walked towards and stood in front of it.

“This one’s rather eager,” the officer said. Then he grabbed a small device from his pocket and pointed it at me.

I felt a very curious sensation. Many live their whole lives without having their TNCs activated. In the van, unable to speak, I would have been rather terrified had I not read and written extensively about TNCs in my previous life. However, even this did not prepare me for the full-body treatment.

The delicacy of these devices is hard to describe. But simply, I was unable to walk in any direction. I could maintain my balance fine. I could sway my arms slightly. I could twist my head and look around. But if I tried to run, as I did in a shameful moment of terror, I simply failed. My body simply did not respond to my orders. If my mind was its officer then the state was its general. And it now had higher orders.

Those around me looked equally terrified. There was an old woman to my left who looked particularly distressed. She had thick, wrinkled features, but in her fear, she almost looked like a child. I could tell, looking at her, that she had not internalized our modern understanding of quantum physics!

Contemplating her philistinism, any pity I felt for her turned into disgust. And it was this disgust the freed me from any fear.

The logic of the state pension is simple and unassailable. We live in a multiverse and one cannot experience non-existence. Thus when you walk into the displacement booth, you are guaranteed to not experience any of the realties where your displacement booth is filled with cyanide gas. A quantum event is used to determine which booth will be spared. As all outcomes occur in the universal wave function, there is always a branch where a copy of you survives and gets the pension.

Thus, the only future I could experience was one of quite-considerable wealth. When the officer ordered it, I would open the door walk into the room, take a short nap, and then I would walk out alone but rich. What did I have to be afraid of? Nothing at all.

“We now have our quorum,” the officer announced. “Once I finish speaking the door in front of you will open and you will walk inside it. Once the door closes your TNC will render you unconscious for 5 minutes. Once you awake, your first check will be transferred to your account and you will then be free to enjoy your generous retirement as you see fit.”

It happened just as he said. The door opened as an elevator door does. The TNC changed slightly and I began to feel an unbearable itching that eased as I took a step forward, then increased exponentially, then eased again as I took another step forward, finally vanishing as I stepped into the booth and turned to face the hallway outside.

Before the door closed, I saw a man about my age opposite to me, attempting to resist. He put up an admirable fight but the outcome was not in doubt. He stepped haltingly, agonizingly into this both. Then the door closed and mine did as well.

I sat down on the floor and leaned against the wall, feeling an extreme sense of drowsiness.


An except from Thinking and Acting in the World So Large by Elijah Mayer:

One should predict, then, that you will not end up in those future histories in which you have chosen to take measure-reducing actions, and this holds true even for measure-indifferent agents. [..] It is for this reason that I feel measure-reducing actions, such as quantum roulette, are not exploitable even by agents who claim to be indifferent to their measure.

I suppose a slogan for such a view could be this: Though you are not interested in measure, measure, I think, is interested in you.


Final Post on Statepension.blog:

November 6th, 2020

And then what happened?  Everything theory predicts. I slept, I awoke. I walked out a rich man. If you think me lucky, you have not learned your lesson. For so was everyone else who took the pension that day!

We will all live rich lives, though will never interact. All of us happily retired in separate branches of the multiverse.

It is so simple, now, on this side of the displacement.

In retrospect, though I managed my fear well, I think I could have done better. It really is not a big deal. Should you qualify for the pension, take it in stride!

It really is as good as I always thought.

I’ve got a very generous first check in the mail, just signed a lease on a beautiful apartment in London. But it is not just the apartment I have signed on for, for I feel like I have a new lease on life, all thanks to the wonders of quantum physics and the British state pension!


An except from Thinking and Acting in the World So Large by Elijah Mayer:

I once dreamed I was performing the quantum suicide experiment. It was discomfiting. But everything turned out fine. I woke up safe in bed.

Does a Competitive AI Ecology Imply a Safe Future?

Note: This is an old essay. Altman and OpenAI have significantly changed their tune. I now consider OpenAI to be an extremely sensible organization. Musk, as far as I can tell, still believes in something like the friendly selection postulate.

About ten months ago, Elon Musk and Sam Altman launched OpenAI, an artificial intelligence research venture with a focus on an open distribution of its software and discoveries. In this time Open AI has published some neat research and hired some excellent engineers. It has acted, in short, more or less like DeepMind and FAIR and all the other AI research organizations that have sprung up in the last five years. Unlike these other organizations, however, OpenAI was founded with two philosophical claims in mind:

Artificial superintelligence has the potential to be an existential risk to human civilization.

A future in which we have an ecosystem of competing artificial agents is likely to be safer than a future in which one agent is able to out-compete all others.

I agree with the first claim.

The second claim, though, I have my doubts about. Though attractive and intuitive on first inspection, I believe this attractiveness and intuitiveness is the result of an incorrect surface analogy to economics and biology and that on deeper inspection it is actually quite troubling, providing very general reasons to expect an AI ecology to be far less amenable to before-the-fact safety interventions than a single-agent outcome

Let’s examine OpenAI’s approach to safety. Here Altman summarizes it briefly:

Just like humans protect against Dr. Evil by the fact that most humans are good, and the collective force of humanity can contain the bad elements, we think it’s far more likely that many, many AIs, will work to stop the occasional bad actors than the idea that there is a single AI a billion times more powerful than anything else.

Altman frames his statement as if the advent of smarter-than-human AI is merely the introduction of a new form of citizen or market participant. But in the context of an ecology of competing agents, another frame is it is a new form of life with a very, very fast rate of reproduction. Not so much a new participant in the economy as a new player in that most ancient of games: evolution by natural selection. It is my belief that Altman’s analogy to law and markets can ultimately be reduced to a far less comforting analogy to natural selection.

One might object to this frame by pointing out that natural selection relies on mutation to provide variation and that since, with modern error-correction, software can be copied with perfect fidelity it won’t apply to artificial intelligence. Even if we ignore the fact that most modern machine learning algorithms are non-deterministic, this objection is still false, for though mutation is a source of variation, it is not the only possible source of variation. Artificial intelligences, being intelligent, will be able to alter their own design and will have divergent utility functions — otherwise, they won’t be competing. Should the future contain multiple, competing artificial superintelligences, this source of variation is more than sufficient for selection to take hold.

I will propose and find wanting a claim that is similar to Altman and Musk’s, but with a more biological twist. This may be less convincing and one OpenAI may or may not agree with. I hope to demonstrate that this statement is false, and afterword demonstrate that OpenAI’s more intuitively appealing less-clearly-stated arguments are actually equivalent.

I will refer to this claim as the “friendly selection postulate.” It is as follows:

Though perhaps no individual superhuman artificial intelligence can be trusted to preserve human interests, an ecology of AIs in equilibrium is likely to experience selective pressures that will align individual AI’s goals away from those of what we regard as bad actors and toward humanity’s interest.

Is this claim correct? How will selection work when these entities, that are both smarter than humans and able to trivially alter their own source code, exist and make copies of themselves in a competitive environment? Will this competitive environment tend to select for agents or collections of agents that have some incentive to treat their inferiors in a manner that assures their safety?

A More Bostromian Vocabulary

Before I attempt to answer these questions I want to introduce three terms used by philosopher Nick Bostrom in his analysis of AI risk. The first term is “utility function” which is the ultimate, terminal goal of an agent. It is what an agent is attempting to maximize, whether it be paperclips, the value of Google’s stock, or human flourishing.

The second term is “singleton.” A singleton is an agent that has achieved such power as to be able to eliminate all competing agents from its environment. Here Bostrom defines the term:

In set theory, a singleton is a set with only one member, but as I introduced the notion, the term refers to a world order in which there is a single decision-making agency at the highest level.[1] Among its powers would be (1) the ability to prevent any threats (internal or external) to its own existence and supremacy, and (2) the ability to exert effective control over major features of its domain

The third term is “multipolar outcome,” which Bostrom describes here:

Another scenario is multipolar, where the transition to superintelligence is slower, and there are many different systems at roughly comparable level of development. In that scenario, you have economic and evolutionary dynamics coming into play.

Self-Aware Natural Selection Without Coordination

Now that we have the definitions out of the way, we can get back to those questions. To do this, let’s imagine we have a world full of millions of super-human artificial agents, all of them with the same hardware and level of cognitive ability. Imagine also that they live in a world of perfect surveillance, so each time any agent makes an advance in improving its own cognition, all other agents instantly apply it to themselves. Imagine, too, there is no coordination or goal aggregation possible between agents in this world.

This is, in essence, the perfect multipolar outcome. A future which according to the friendly selection postulate would be safer than a single-AI scenario. What are the evolutionary dynamics of this world? As we’ve removed variation in cognitive ability in this scenario, what variation remains to be selected upon? The answer is variation in utility functions.

Like many processes with complex outputs, natural selection can be summed up starkly: that which reproduces more effectively than its competitors will displace its competitors. The fittest possible agent in these conditions is an agent that desires only to replicate itself. Why is this? Say you have agent X who values what humans value (love, art, happiness, fun, community, etc) and agent Y who values only to make copies of itself. Agent Y is able to make more copies of itself than agent X as it doesn’t have to spend resources maximizing human value in the universe. Because these agents are intelligent and aware of natural selection, their’s is an odd sort of self-aware natural selection. So agent X would realize this and have incentive to approximate Y as much as necessary to compete with agent Y while maximizing its utility function, but it could not do so completely without abandoning its utility function entirely. Its values would be a sort of stone shackled to it, making it less competitive than any agent that cares more about making more copies of itself. Because of this and the fact that all agents have equivalent initial power in this contrived scenario, we should expect this world to quickly be filled by agent Ys or agents that are able to approximate agent Y very efficiently, and these agents to displace those with more complex values.

Thus in the limiting case of a perfect and instantaneous distribution of new technological capabilities among competing artificial agents and an absence of coordination or aggregation of utility functions among agents, humanity and our values will be selected against by highly-accelerated evolutionary forces.

Self-Aware Natural Selection With Coordination

Now lets take the same scenario and remove the restriction on coordination and aggregation of utility functions. But first, what do I mean by “aggregation of utility functions?”

The ability to alter one’s own source code gives an artificial intelligence what is essentially the perfect coordination tool. It allows an agent to alter its own utility function in those circumstances where it thinks altering its own utility function in some manner will get it more future utility (by its pre-modification definition) than it would otherwise. How would this work?

Suppose we have agent P whose utility function is to maximize the number of paperclips in the universe and agent S whose utility function is to maximize the number of staples in the universe.

Agents P and S are the two most powerful agents in the world with no other competitors.

Seemingly, they are at permanent odds. Utility function aggregation provides a solution. Agent P might reason as follows: a future fighting with agent S over the universes’ resources is going to result in fewer paperclips than a future in which agent S and I agree to maximize both paperclips and staples. If agent S reasons similarly, the two have a perfect means of enforcing this agreement: altering their utility functions, so they both intrinsically desire to maximize both. Thus in this manner agent P and S aggregate their utility functions, replacing themselves with one agent, agent PS, who happily converts the universe into paperclips and staples in equal measure with no further competition.

From the perspective of the friendly selection postulate, coordination and aggregation of utility functions seems something of a lateral play. Why is this? First lets recall agents X and Y from before. Agent X values everything we value and agent Y wants only to make more copies of itself. Suppose Agent X and Y aggregate their goals, becoming agent XY. You are now left with an agent XY with more resources but pretty complex goals. If goal aggregation is able to give enough power to XY to become a singleton we are left with a singleton, which is exactly what the friendly selection postulate tells us we should avoid. Worse, it is a singleton with by-definition compromised goals. If utility function aggregation isn’t able to give XY enough power to become a singleton, we are back to the world in which agents most able to approximate agent Y have the advantage.

So we are stuck in a bit of a catch-22. In the case of evolution without goal aggregation, human values will be selected against in favor of agent-Y-like values. In the case of evolution with goal aggregation, agents willing to aggregate their utility functions with other agents are able to gain more power, at least temporarily.

However, unless this power can be parlayed into a decisive strategic advantage that pushes them out of a situation where natural selection applies, forming a singleton able to permanently wipe out the competition, these agents will always eventually lose out to agent Ys. And worse, should a singleton be formed, it will likely be less able to maximize human value than the most human-friendly agent introduced into the AI ecosystem, because its goals will be compromised to some extent during the process of aggregating its utility function with competing agents. And if this is the case, why not attempt to build the friendly singleton from the beginning, bypassing all the messy aggregation business?

A Dressed-up Friendly Selection Postulate

The friendly selection postulate seems to hold no water, but what of OpenAI’s claim? As I said above, I believe OpenAI’s claims are equivalent to the friendly selection postulate, though more-appealingly relayed.

Here Alman and Musk describes OpenAI’s strategy with journalist Steven Levy:

Levy: How did this come about? […]

Musk: Philosophically there’s an important element here: we want AI to be widespread. There’s two schools of thought? — ?do you want many AIs, or a small number of AIs? We think probably many is good. And to the degree that you can tie it to an extension of individual human will, that is also good. […]

Altman: We think the best way AI can develop is if it’s about individual empowerment and making humans better, and made freely available to everyone, not a single entity that is a million times more powerful than any human. Because we are not a for-profit company, like a Google, we can focus not on trying to enrich our shareholders, but what we believe is the actual best thing for the future of humanity.

Levy: Couldn’t your stuff in OpenAI surpass human intelligence?

Altman: I expect that it will, but it will just be open source and useable by everyone instead of useable by, say, just Google. Anything the group develops will be available to everyone. If you take it and repurpose it you don’t have to share that. But any of the work that we do will be available to everyone.

Levy: If I’m Dr. Evil and I use it, won’t you be empowering me?

Musk: I think that’s an excellent question and it’s something that we debated quite a bit.

Altman: There are a few different thoughts about this. Just like humans protect against Dr. Evil by the fact that most humans are good, and the collective force of humanity can contain the bad elements, we think it’s far more likely that many, many AIs, will work to stop the occasional bad actors than the idea that there is a single AI a billion times more powerful than anything else. If that one thing goes off the rails or if Dr. Evil gets that one thing and there is nothing to counteract it, then we’re really in a bad place.

Though full of wonderful rhetoric, their strategy would result in the creation of many agents with may differing goals. As I have demonstrated above, there is good reason to believe this would lead to an ecology in which natural selection takes hold, and should this happen, any agent with values more complex than a deliberate desire to create more copies of oneself will be selected against, leading to a future where agent Ys and those able to approximate them displace humans. And should goal aggregation occur, the resulting singleton will always have a utility function which, from the human perspective, is compromised at least to some extent.

If disaster is the result of OpenAI’s equal playing field, what are our alternatives? One, of course, is to prevent the development of artificial intelligence completely. This seems impossible given how actively AI is being pursued by multiple parties. The second alternative is to bite the bullet and try to actively create a singleton with a utility function that accounts for humans and our values, which seems merely very, very, very, very unlikely.

Regardless, a competitive ecology isn’t a solution. Should we want a safe outcome from AI, we need a means of specifying human value, and an agent with enough headroom above the competition to maximize that value without being outcompeted.

As we have (at least temporarily) escaped the Malthusian condition, modern humans have little direct experience of how hellish an ecosystem can be, so it is easy for us to anthropomorphize mother nature as a kindly force with vague but beneficent goals. It is a useful to remind ourselves that this intuition is false. Natural selection is a search process that is literally powered by death, this the most literal use of the word “literally.” It cares nothing for human suffering or happiness, for it has no mind to care with. It has no purpose or motivations beyond this: that which reproduces more effectively than its competitors will displace its competitors.

Working towards a human-friendly singleton is our only safe bet, for any outcome with competing agents, no matter how you dress it up, leads to natural selection. And natural selection is Lovecraftian horror not an AI safety solution.

Amateur Philosophy: Reframing Average Utilitarianism

World A has a million people with an average utility of 10; world B has 100 people with an average utility of 11. Average utilitarianism says world B is preferable to world A. This seems counterintuitive as it has less total utility, but what if we reframe the question?

Imagine you are behind a veil of ignorance and you have to choose which world you will be instantiated into, becoming one citizen randomly selected from the population. From this perspective, world B is the obvious choice: even though it has far less total utility than word A, you personally get more utility by being instantiated into world B. This remains true even if world B only has 1 citizen, though most people, presumably, have “access to good company” in their utility function.

This reframing seems to invert my intuitions. Though this may just mean I am more selfish than most.

Some Notes on The AI Does Not Hate You: Superintelligence, Rationality and the Race to Save the World

The AI Does Not Hate You is the first book about the rationalist community that has been published by a traditional publisher, unless you include To Be a Machine which I don’t. Naturally I was interested in picking it up, if only to cringe at an awful caricature of the community. (The author is a former BuzzFeed journalist after all.)

I was able to get a copy of it this afternoon and finished it a couple hours ago.[1] Anyone worried we would be in for a facile mocking will be pleasantly surprised. Chivers does a competent job describing the history of the rationalist movement. No straw men are constructed. There is nothing in the book most people here would find very controversial, but this has a side effect of making it sometimes dull to anyone who has been following the community as long as I have. But this is no fault of the book, for it was not written for us.

I do not think someone outside the memespace would be bored by it. In fact, if you are having trouble explaining the rationalist thing to anyone in your life, this book seems like a pretty good introduction that will give them the basics. In fact, even if it sells poorly, it is possible having a Schelling point for “basic summary of who we are and what we are trying to do” may have a larger impact on our community than one might initially suppose. Books are surprisingly powerful. Nick Bostrom’s book, though good, was mostly redundant if you read his papers. Yet no one can deny its impact.

The AI Does Not Hate You consists of basic introductions to some rationalist themes and some history of the rationalist movement, all squeezed around very polite, very British suggestion that, well, perhaps we would not be at all over-cautions if we gave this AI risk idea a little more attention.

The Basilisk is mentioned, but not as a means of defining the community. He sums it up as a thought experiment that no one took seriously, and as a prototypical example of the Streisand effect. The “is it a cult” section concludes that LessWrong is not a cult. He goes over statistics on polyamory and political values. Most of his stats are cribbed from Scott’s blog, so again there is not much you will learn in this book if you are a SSC completionist.

One thing I would have liked is some coverage of Gwern. I think he is a pretty huge figure in our world and also highly mysterious, but Chivers mentions him once, and only as an example of one of many prominent figures in the community. I understand how difficult it would be for a journalist to track down Gwern, who lives in his personal cybermatrix, his em consciousness computed in dozens of parallel copies using cycles stolen from Wikipedia’s servers, dark net markets, and Tyler Cowen’s left temporal lobe. But dammit, sometimes a journalist has to get their hands dirty.

It’s interesting, too, how Luke Muehlhauser and Alcorn are not mentioned at all in the sections on the history of LessWrong, as they were both massively popular on LessWrong at its peak. Both (I am given to understand) remain hugely influential figures in the IRL rationalist scene. Julia Galef is another major figure not mentioned.

Anna Salomon is depicted as a spooky-intelligent shamanic figure, a zen monk who induces epistemic insights in the author using a technique called the internal double crux. I have never met Anna Salomon (or any IRL rationalist) so I cannot vouch for the accuracy of this portrayal.

He goes over the history of the AI box experiment, and I did learn something new here. Apparently Carl Shulmon was one of the people that Yudkowsky beat in the AI box experiment. I have read Carl’s comments and writing and I have always been impressed by his intelligence, which vastly exceeds my own. I am not the cleverest sausage, and I completely grant that an AGI could talk me into anything, but I sort of always thought I would still not let Yudkowsky out of the box. He is only human. The fact that he beat Carl Shulmon, who is much, much smarter than me, lowers my confidence in this considerably.

Anyway, the book’s fine, and likely will not harm the rationalsphere, so /u/endersdouble has, in my opinion, lost his bet with /u/scottalexander.

  1. The book is technically region locked to the UK, but after putting an invalid UK credit card into my Amazon account and logging into the UK audible store and signing up for a trial account, I was able to get a copy of the audiobook. My reasoning was they would probably not check the validity of the card until they charged it but they would not charge it until after the free trial ends. The free trial includes one free audiobook, so a happy side effect of this work around is you get the book for free. This trick does not work with the Kindle store, as they do not have a similar free book credit.

Synapse vs a Parameter?

In terms of AI timelines, the biggest question I haven’t seen addressed is the computational equivalence of a synapse vs a parameter in modern neural nets. This seems like a very important input for any prediction on when we will have human-level AI.

Moravec’s estimates of a retina vs the then-current edge-detection methods are sort of worthless under the assumption that AI will be built using modern learning methods, because feature engineered code is plausibly much more compute efficient than learned policies on tasks that are comprehensible to human programmers, and Moravec compared the retina with the former.

To pump this intuition, if we assume that 1 synapse == 1 parameter, then Moravec’s estimates are more than 6 orders of magnitude too low. The size of models we are able to train, on computers that are more powerful than Moravec predicted are needed for AGI, is at most 10 billion parameters, which is about as many synapses as the retina has.

A very interesting question, then, is how many parameters does a modern learned model need to have plausibly similar capabilities to a human retina. This seems hard to investigate but not impossible. If we look at the first few layers of a conv net, they seem to be doing the sort of edge detection that the retina does.

I think this would be a high-leverage thing to investigate as both the parameter > synapse and parameter < synapse are not implausible at first glance. Some people think the brain is doing something mysterious and better than backpropagation. Some people think it is just doing a really shitty, Rube Goldberg approximation of backpropagation. Artificial spiking neural networks perform really poorly compared standard stuff, which may imply paramater > synapse, etc.

If we assume AGI will come from scaling up current methods (rather than, as Moravec predicted, very efficient programs that imitate the aggregate function of thousand-neuron assemblies) then this question is very pertinent to any prediction. Is anyone here aware of any work on this?

Mad-science IQ Enhancement?

Slatestarcodex has a great post on the intelligence of birds.1

The big take-away: due do the weight restriction of flight, birds have been under huge evolutionary pressure to miniaturize their neurons. The result?

Driven by the need to stay light enough to fly, birds have scaled down their neurons to a level unmatched by any other group. Elephants have about 7,000 neurons per mg of brain tissue. Humans have about 25,000. Birds have up to 200,000. That means a small crow can have the same number of neurons as a pretty big monkey.

Bird brains are 10x more computationally dense than our own? This is a big deal. To put this in perspective, if you replaced just 10% of your brain volume with crow neurons you could double your computational capacity.

I know this sounds like complete insanity, but given how every intervention to raise IQ has been ineffective, this bird-brain scheme is probably much more promising than any pharmaceutical based approach.

This raises all sorts of interesting questions. How does miniaturization effect heat dissipation? Oxygenation? Energy consumption? Could one build a human-sized brain with bird-dense neurons?

Though the brain is immunologically privileged, there is still the neuroimmune system. Can we genetically modify crow cortical neuron progenitor cells to not trigger the neuroimmune system? Is there any chance of human neurons and bird-neurons integrating usefully?

Brain grafts have been unpromising for intelligence enhancement because you would have to replace much of the brain to make a difference. With bird tissue, this problem is 10x less relevant.

Or more ominously, what if we breed crows for brain size? Hyper-Intelligent flightless crows pecking keyboards in hedgefund basements? To paraphrase Douglas Adams, there is another theory which states that this is already happening.

Beware of the Gwern

[This is a silly thing I posted on Reddit several months ago]

Gwern, as you likely already know, is a mysterious formless entity of great diligence and intellect. Thus far, many have assumed him to be a benign presence. But how certain of Gwern’s motives can we be?

Recently, he has begun to craft objects of great power. And many admire him more for this. But within his creations, one finds hints of a twisted humour. Look closely at his miracles and victims begin to emerge.

Take an older project of his, https://thiswaifudoesnotexist.net. A wonder to behold? An amusement but nothing more?

/u/terratheillusionist relates this sad tale:

“Once opened I was shown a waifu so lovely and pure that I stared in amazement and awe. Then the page refreshed automatically. Now I am doomed to forever refresh to get her back, knowing it shall never be.”

How many more young men sit, hunched and enslaved to this magic? What strange purpose does this serve?

Here we find a hint: https://twitter.com/theshawwn/status/1216011374758912000.

Shawn Presser is a young man skilled in the arts Gwern values. He has also begun to assist Gwern in his endeavours. Due to the largess of his patron, Presser has been given access to many hundreds of petaflops of computing resources. Why would a skilled practitioner with such vast resources swear his sword to another? How did Gwern win his loyalty? These are interesting questions. I give you another: What wouldn’t a man do for his waifu?

Together, they have released a new tool that lives on this very site. They have made a simulated reddit filled with simulated conversations. This thread is a startling example: https://www.reddit.com/r/SubSimulatorGPT2/comments/eq2o72/cmv_the_us_is_a_bad_place_to_live/ Read it closely. Had I not just told you these were but babbling phantasms would you have known? How certain can we be that anyone on this site is not but a golem of Gwern’s manufacture? Is anyone reading this at all, or has he already won? Am I conversing only with his creations? Trapped like /u/terratheillusionist in his spell for his unfathomable ends.

It is a testament to Gwern’s growing power that I can no longer be sure.