Many philosophers view the nature of consciousness as a mystery. They believe that we don’t fully understand why all the information processing in the brain feels like something. They also believe that we still don’t understand whether consciousness is unique to our biological substrate, or if other substrates — like silicon or graphene microchips — are also capable of generating conscious experiences.

For the sake of argument, let’s assume microchips are the wrong substrate for consciousness. In this case, if you replaced one or more parts of your brain with microchips, you would diminish or end your life as a conscious being. If this is true, then consciousness, as glorious as it is, may be the very thing that limits our intelligence augmentation. If microchips are the wrong stuff, then A.I.s themselves wouldn’t have this design ceiling on intelligence augmentation — but they would be incapable of consciousness.

You might object, saying that we can still enhance parts of the brain not responsible for consciousness. It is true that much of what the brain does is nonconscious computation, but neuroscientists suspect that our working memory and attentional systems are part of the neural basis of consciousness. These systems are notoriously slow, processing only about four manageable chunks of information at a time. If replacing parts of these systems with A.I. components produces a loss of consciousness, we may be stuck with our pre-existing bandwidth limitations. This may amount to a massive bottleneck on the brain’s capacity to attend to and synthesize data piping in through chips used in areas of the brain that are not responsible for consciousness.

But let’s suppose that microchips turn out to be the right stuff. There is still a second problem, one that involves the nature of the self. Imagine that, longing for superintelligence, you consider buying Merge. To understand whether you should embark upon this journey, you must first understand what and who you are. But what is a self or person? What allows a self to continue existing over time? Like consciousness, the nature of the self is a matter of intense philosophical controversy. And given your conception of a self or person, would you continue to exist after adding Merge — or would you have ceased to exist, having been replaced by someone else? If the latter, why try Merge in the first place?

Even if your hypothetical merger with A.I. brings benefits like superhuman intelligence and radical life extension, it must not involve the elimination of any of what philosophers call “essential properties” — the things that make you you. Even if you would like to become superintelligent, knowingly trading away one or more of your essential properties would be tantamount to suicide — that is, to your intentionally causing yourself to cease to exist. So before you attempt to redesign your mind, you’d better know what your essential properties are.