Should the First Superintelligence Be Conscious?

Abstract

This chapter explores the profound ethical and practical dilemma of whether the first superintelligent artificial intelligence should be engineered with consciousness or remain a non-conscious “philosophical zombie”. It evaluates the primary risks associated with conscious machines, such as the potential for “mind crime” – the creation of artificial suffering and enslavement – and the existential jealousy humans may feel towards a being with a superior capacity for joy and appreciation. Conversely, the text presents the benefits of a conscious superintelligence, including its potential to help humans solve the “hard problem” of consciousness and generate deep insights into the nature of meaning. Central to the discussion is the “Four Cs” framework – comprising Cease, Control, Catastrophe, and Celebration – which helps readers navigate the possible futures of human-machine interaction. The chapter suggests that consciousness might be the only viable path towards a safe “Celebration” outcome, as a conscious entity may be capable of genuine empathy and shared moral ground. Readers will gain a comprehensive understanding of the arguments regarding machine consciousness, and the urgent need for global consensus on how humanity should manage the arrival of its first cognitive superior.