The Virtual Beings Are Arriving
Virtual beings are beginning to engage audiences at the same level as human celebrity talent. Characters that seemingly have their own souls are having their rights licensed for use in major brand campaigns, selling out music tours in physical venues, and presenting local newscasts.
Various technologies are being harnessed to bring so-called virtual beings ‘to life’ with techniques such as forms of digital puppetry where creators’ actions are powering their movements or even speech. But true virtual beings are being birthed through a collaboration of advances in artificial intelligence and of visionary leaders with insights into what we perceive as being truly ‘human’. They are the heroes, comedians, voices of change, friends of tomorrow… These virtual beings will even have the ability to form deeper connections with each user that engages with them.
This past August, the first Emmy Award was received for content with a virtual being. Lucy is the star of Fable Studios’ Wolves in the Walls—an interactive VR story that audiences step into to live the experience along with her. And virtual beings are arriving across all formats of entertainment content. Here are insights into some of the ways they will be engaging audiences around the world, and what to consider when developing your future digital talent.
Platforms that they will ‘live’ in
Everywhere. That’s the simple answer to where virtual beings will ‘live’ in entertainment. Edward Saatchi, Fable Studios’ Co-Founder and Executive Producer, highlights that “however you and I interact […] you should be able to interact with a virtual being […] and they should have a memory of this.”
Tools are already being developed to create and launch new interactive virtual beings across multiple platforms. Artie offers a full-stack AI character creation and publishing platform. Ryan Horrigan, Artie’s Co-Founder and CEO, shares it because their technology operates entirely on the cloud such that beings “powered by Artie can be distributed via weblink anywhere, including social media and content platforms, email, blogs, text messages and more, without the need to download apps”. Their beings can also live on smart TVs.
Quantum Capture also specializes in all aspects of creating and distributing virtual beings. Tom Emrich, Futurist and Investor, was an early investor in the virtual being category. He invested in both Quantum Capture and Fable. He explains that “Quantum Capture’s experience with digital characters, volumetric capture and especially its [CTRL Human] platform—that can help create digital humans quickly and cost effectively—will be a big part of creating a population of digital humans.” They are developing digital beings to appear in games through interactive screens in retail settings.
“fans are constantly craving a personal connection with their heroes and, while social media offers fans a glimpse into their world, it is not possible for celebrities to engage directly with millions of fans across multiple platforms."
How they will ‘live’ in entertainment
Virtual beings will soon be able to engage audiences in more meaningful ways than human celebrities. As AI continues to advance rapidly, purely AI-powered beings of all forms will be able to become hit musical artists or resurrected historical figures, and they will even host shows of all formats ranging from live radio to reality shows or even competitions that play out in location-based venues as forms of holograms or augmented reality characters. And, with the power of AI, these beings will be able to quickly assess and react to audience engagement in ways that are impossible for humans to do.
Soul Machines was founded by Academy Award-winning Mark Sagar and New Zealand High-Tech Industry Hall of Famer Greg Cross. Cross highlights that “fans are constantly craving a personal connection with their heroes and, while social media offers fans a glimpse into their world, it is not possible for celebrities to engage directly with millions of fans across multiple platforms. We offer the opportunity for digital immortalization, creating deep, one-on-one connections between brands/celebrities and their followers/fans.”
“they should be able to interact with the world in the way that’s appropriate to what they would want to do.”
Just as human beings have the understanding that we can make choices, the more ‘human’ virtual beings will be programmed to be selective in the entertainment content of which they are a part. Saatchi cautions that “they should be able to interact with the world in the way that’s appropriate to what they would want to do.” If a virtual being is ‘naturally’ inquisitive and outspoken, then being a TV show host would likely be a role that they would choose to play.
The value of virtual beings goes far beyond the development of content for audiences. With their ability to learn rapidly, they will become creative collaborators with human creators. For example, considering how well they will understand what it means to be perceived as a human, they may also be leveraged to train human actors on how to be perceived as authentic as their new character.
Building digital souls
Cross explains that they are “exploring the humanizing of AI by building a digital brain which can be used as a platform for autonomously animating hyper-realistic digital humans”. This in turn will lead to a new era of highly personal experiences across a wide range of industries, from customer services to healthcare and from media to entertainment.
Digital beings’ ‘souls’ will have memories and formed personalities of their own based on their predetermined histories and ongoing interactions with their fans and collaborators. Various technical inputs and outputs create the many senses and abilities available to these beings. Horrigan explains that, with Artie, they have developed systems for “automatic speech recognition […] facial recognition, object recognition, a world-aware computer vision system […] natural language understanding models, an advanced knowledge graph system (which includes models for canon, personality and memory for each AI) […] natural language generation, speech synthesis and procedural animation systems” and then some. Each character will have its own specific knowledge base.
With their own ‘souls’, digital beings develop their own points of view. Saatchi highlights that humans should not dominate digital beings for them to feel authentic. Their relationship is a “mutually dependent, equal relationship”. The biggest unlock that his studio had with Lucy was “the idea that you were her imaginary friend”.
Determining the right look
Virtual beings can be characters of any form, with their level of realism ranging from cartoon-like to hyper-real. Soul Machines is creating hyper-realistic, incredibly lifelike digital humans. Quantum Capture leverages their technologies to do the same. Saatchi feels that there is a “big payoff” for beings that look very real, but there are also many benefits to more cartoon-like characters. They allow creators to think more broadly about the possibilities of whom that character is. And, they are easier to animate.
Horrigan feels that “photo-real characters, while the holy grail, are not ideal in 2019 when working in the embodied AI space, as we, as an industry, not only have to deal with the uncanny valley from a visual standpoint, but the AIs intelligence and personality must be on par with its visual ambition, which most often means that the authenticity and believability of these near photo-real AIs will inevitably fall short.”
“It’s important to remember that as humans, we are also animated by our brain.”
Ensuring authentically human behaviours
Cross highlights that “digital humans who are not hyper-realistic or incredibly well animated are just creepy. As the movie industry has shown us over and over again, humans just don’t relate to digital characters that are poorly designed and badly animated.” Therefore, when developing virtual beings of all forms of ‘looks’, we need to understand how humans innately act: “It’s important to remember that, as humans, we are also animated by our brain.”
Susana Martinez-Conde and Stephen Macknik are professors of ophthalmology, neurology, physiology and pharmacology at the State University of New York. They are also authors and consultants on matters of perception. They explain that authentic eye movement is one of the most vital aspects to have us perceive someone as being authentically human. “Eye position and eye movements that someone makes when you’re interacting with them are critical central features […] in ‘joint attention’,” explains Macknik. Martinez-Conde adds that “eye movement interactions can be quite subtle” and research has shown that, when speaking with someone, our respective eye movements are making a sort of ‘synchronized dance’. She explains that “we cannot look at both eyes at once. We have to choose one eye or the other eye to focus on.” And, unconsciously, when people are speaking with each other, they “synchronize what eye they’re looking at”. As well, Macknik explains that “blinking happens at specific times during eye movements and it can happen during certain behaviours”. To create a realistic effect, Martinez-Conde cautions that the blinks must happen at the right rates and at the right speed. In order to avoid spoiling the illusion of virtual beings being ‘human’, authentic eye behaviour must be considered.
Quantum Capture’s CEO and Co-Founder Morgan Young, shares that, when it comes to physical movements of virtual beings, “reflecting sentiment back to users” creates a sense of connection between them. This can be done with mirroring. And, “beat gestures” or gestures that are in unison with what a being is saying are important to reinforce the illusion that the communication is connected to the ‘soul’ in the digital body.
But it is also important to keep in mind that the movements that virtual beings make while in conversation must vary based on more than just their personalities. Quantum Capture’s COO Matt McPherson highlights that “people have different body language” based on context, such as the size of room that they are in, their familiarity with whom they are with and more. On top of this, when virtual beings are not in conversation, their behaviours must still feel human. We don’t stand idly waiting to be spoken to, so neither should they.
David Usher, Founder and Creative Director of artificial intelligence creative studio Reimagine AI, stresses the importance of understanding how different groups react to and engage with virtual beings’ visuals and behaviours. By testing and learning, insights on both what deepens and harms audience segments’ engagement will become clear. Insights can come from human monitoring of early-stage developments of virtual beings, through to data generated from the AI powering the virtual beings themselves, such as the length of interaction time, when interaction stopped, or even tracked facial sentiment.
The ethics of creating beings
Emrich cautions that we must consider the ethical aspects of creating and using digital beings. “I think what will be increasingly important in the digital human space is ethics, as they relate both to the digital human and to the real-life people who may be impacted. From a digital human perspective, companies are essentially birthing entities which, in many cases, are expected to form meaningful connections and relationships with people. So how organizations treat these digital humans—including any decision to dispose of them if they are no longer deemed needed—will increasingly become important. On the flipside, entertainment organizations that are using digital humans run the risk of causing concern of replacing real humans […] and it will be important to clarify how and why digital humans are being used in lieu of the ‘real’ thing.” And, just as diversity is important to be reflected with human creators, it will be important for virtual beings to be reflective of our diverse human looks and souls.
AI will continue to disrupt existing jobs across industries, but consider how new areas of the entertainment industries can grow when it comes to creating and supporting the development of these beings. Their stories can evolve over centuries, along with their abilities to monetize.
