The Metaverse Paradigm
The Metaverse is ubiquitously characterized as a 3D environment where augmented reality and virtual reality technologies bring to bear the virtual worlds of avatars. Prominent voices from the technology industry envision the Metaverse to replace the current makeup of the internet. Some even go as far as proposing it as a universe where humans in the future will live. This blog discusses a technical approach where the digital representations in the Metaverse will not be limited to the caricatures of people’s appearance but can also be digital clones of their intellect. The strides in software technology and sophisticated forms of artificial intelligence present a framework that enables us to digitally clone aspects of human beings, such as voice and intellect.
The ascendance of digital technologies continues to grow at an incredible pace, and the secular culture is increasingly blurring the line between the digital world and reality. Companies are investing billions into developing digital technology. The Metaverse is one such digital technology that has recently captured the imagination of technology evangelists. Technologists do not see the Metaverse as hypothetical or futuristic suppositions but as an emerging reality. Meta, the parent company of Facebook, invested $10 billion [1] into the Metaverse in 2021. JPMorgan Chase values the Metaverse at $1 trillion [2], and Goldman Sachs predicts that the Metaverse could be an $8 trillion opportunity [3]. This blog post propositions and examines one such tantalizing but novel opportunity that the Metaverse presents – Human Digital Clones.
Human Digital Clones
The Metaverse is intended to be a hyper-realistic virtual space that is immersive and interactive. The predominant vision for the Metaverse is as a series of interlinked virtual 3D worlds visited by hyper-realistic avatars and experienced through virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. Such as wearable tech, haptic suits, and 3D headsets. The most important thing to take when entering the Metaverse is one’s digital identity – an avatar. However, this method still requires a human at the other end to drive the interaction. All actions the avatar undertakes in the Metaverse are directly driven by the user. This paper discusses human digital representations in the Metaverse that is not just through a caricature of a human appearance (avatars), but instead an autonomous digital copy of a human’s intellect – Human Digital Clone. The human digital clone will incorporate the intellect of the human it represents (intellect clone) and the digital voice of the human it represents (voice clone). The human digital clone will be created using AI technology and deployed into the Metaverse. The human digital clone will be decoupled from the user. It will inhabit and operate autonomously in the Metaverse without the user having to drive it.
Decentraland [4] is a metaverse virtual world filled with lush forests, beautiful skies, thriving cities, and vast oceans. Second Life [5] is a realm with a virtual economy and a rich 3D world. Several other virtual worlds have been created by companies such as Horizon Worlds [6], Somnium Space [7] and Sandbox [8]. Leveraging the concept and examples of virtual worlds such as Decentraland, this paper uses a hypothetical example of a virtual retail store in the metaverse. However, instead of an avatar, the virtual retail store will be led by manned by a human digital clone of a salesperson. The human digital clone of the salesperson will incorporate the product intellect (intellect clone) and the voice (voice clone) of a real salesperson. The digital human clone of the salesperson can then be deployed to the metaverse where it will autonomously interact with virtual store customers who will experience the virtual retail store that operates in their metaverse via their avatars.
How will we build them?
Metaverse that is not just through a caricature of a human appearance (avatars), but instead an autonomous digital copy of a human’s intellect – a Human Digital Clone. The human digital clone will incorporate the intellect of the human it represents (intellect clone) and the digital voice of the human it represents (voice clone). The human digital clone will be created using AI technology and deployed into the Metaverse. The human digital clone will be decoupled from the user. It will inhabit and operate autonomously in the Metaverse without the user having to drive it.
A digital human clone is composed by combining the intellect clone and the voice clone. The image below shows at a high level the end-to-end process by which we can create a digital human clone.
1. A human subject is the source of the voice sample and intellect for a specific domain.
2. The voice of the human subject used to train a voice model
3. The intellect (in a “narrow” domain) used to train an intellect model
4. The voice model produces a digital voice clone
5. The intellect model produces a digital intellect clone.
6. The voice and intellect clones are combined to produce a human digital clone.
7. The human digital clone is deployed to the Metaverse
Summary
For decades human beings have interacted with the internet through elements such as a website, a game, chat programs or, more recently, a bot. The metaverse takes this a step further by putting the human at the centre of the action – An embodied internet that the human is inside of instead of outside. A metaverse is a place where the merging of the real and digital domains creates an environment that is as interactive and experiential as the real world. Digital intellect twins add a new dimension to the emerging metaverse. When mashed up with the metaverse, technical advances in the past decade present technical opportunities that previously only resided in the world of science fiction. Today, AI systems can behaviourally engage in intelligent human-like thought and perform a useful and specific function in specific domains that once required human intelligence to perform. AI models today can also produce synthetic audio that sounds convincingly like the person the system is trying to mimic. We can digitally clone a human by combining these two technologies with digital avatars in the Metaverse. The digital clone will look, think (for specific domains), and sound like a real-life human. It will possess the intellect of the human in a particular domain, such as an expert on earthquakes, a specific type of pandemic, an expert in disaster management etc. Since the clone is in digital form and driven by software-based AI models (for both speech and intellect), the clone can be containerized using K8 or OpenShift technologies and deployed (using hardware as code) anywhere in the world. In effect, digitally transferring a copy of a human being across the internet. Albeit a human with intellect in a specific domain.
A mashup of technologies such as AI, NLP and the advances in creating life-like avatars, and the metaverse enables the creation of human digital clones to be technically feasible. Digital human clones simulate the physical world inside the digital world and are experienced from the real world by real users. Digital human clones adorned with a real-life-like voice and appearance of a real person will live and operate in the metaverse completely decoupled (apart from scenarios of feedback to improve the intellect twin models or update them model) from the real-life individual upon which the intellect twin is based on. Digital intellect clones can transform the world of the metaverse from one inhabited by avatars to a universe inhabited by human digital clones.
References
[1] M. Isaac, ‘Meta spent $10 billion on the metaverse in 2021, dragging down profit.’, The New York Times, Feb. 02, 2022. Accessed: Jul. 20, 2022. [Online]. Available: https://www.nytimes.com/2022/02/02/technology/meta-facebook-earnings-metaverse.html
[2] ‘JPMorgan bets the metaverse is a $1 trillion yearly opportunity and opens the first virtual bank’, Fortune. https://fortune.com/2022/02/16/jpmorgan-first-bank-join-metaverse/ (accessed Jul. 20, 2022).
[3] J. Collection, ‘Goldman Sachs Sees Metaverse as $ 8 Trillion Opportunity – Metaverse Bitcoin News – Journaltime.org’, JournalTime, Jan. 25, 2022. https://journaltime.org/finance/banking/investment-banks/goldman-sachs-sees-metaverse-as-8-trillion-opportunity-metaverse-bitcoin-news/ (accessed Jul. 20, 2022).
[4] ‘Welcome to Decentraland’. https://decentraland.org/ (accessed Jul. 28, 2022).
[5] ‘Official Site | Second Life – Virtual Worlds, Virtual Reality, VR, Avatars, and Free 3D Chat’. https://secondlife.com/ (accessed Jul. 28, 2022).
[6] ‘Horizon Worlds | Virtual reality worlds and communities’. https://www.oculus.com/horizon-worlds/ (accessed Jul. 28, 2022).
[7] ‘Somnium Space’. https://somniumspace.com/ (accessed Jul. 28, 2022).
[8] ‘A Decentralized Gaming Metaverse Made By Players’, The Sandbox. https://sandbox.game (accessed Jul. 28, 2022).
