In this morph target animation system four "expressions" have been defined as deformations of the geometry of the model. Any combination of these four expressions can be used to animate the mouth shape. Similar controls can be applied to animate an entire human-like model.
Human image synthesis is technology that can be applied to make believable and even photorealistic renditions of human-likenesses, moving or still. It has effectively existed since the early 2000s. Many films using computer generated imagery have featured synthetic images of human-like characters digitally composited onto the real or other simulated film material. Towards the end of the 2010s deep learningartificial intelligence has been applied to synthesize images and video that look like humans, without need for human assistance, once the training phase has been completed, whereas the old school 7D-route required massive amounts of human work.
The 1994 film The Crow was the first film production to make use of digital compositing of a computer simulated representation of a face onto scenes filmed using a body double. Necessity was the muse as the actor Brandon Lee portraying the protagonist was tragically killed accidentally on-stage.
In 2009 a digital look-alike of a younger Arnold Schwarzenegger was made for the movie Terminator Salvation though the end result was critiqued as unconvincing. Facial geometry was acquired from a 1984 mold of Schwarzenegger.
In SIGGGRAPH 2013Activision and USC presented a real time "Digital Ira" a digital face look-alike of Ari Shapiro, an ICT USC research scientist, utilizing the USC light stage X by Ghosh et al. for both reflectance field and motion capture. The end result both precomputed and real-time rendering with the modernest game GPU shown here and looks fairly realistic.
In 2014 The Presidential Portrait by USC ICT in conjunction with the Smithsonian Institution was made using the latest USC mobile light stage wherein President Barack Obama had his geometry, textures and reflectance captured.
In 2016 a digital look-alike of Peter Cushing was made for the Rogue One film where its appearance would appear to be of same age as the actor was during the filming of the original 1977 Star Wars film.
In SIGGRAPH2017 an audio driven digital look-alike of upper torso of Barack Obama was presented by researchers from University of Washington. (view) It was driven only by a voice track as source data for the animation after the training phase to acquire lip sync and wider facial information from training material consisting 2D videos with audio had been completed.
Late 2017 and early 2018 saw the surfacing of the deepfakes controversy where porn videos were doctored using deep machine learning so that the face of the actress was replaced by the software's opinion of what another persons face would look like in the same pose and lighting.
In 2018 at the World Internet Conference in Wuzhen the Xinhua News Agency presented two digital look-alikes made to the resemblance of its real news anchors Qiu Hao (Chinese language) and Zhang Zhao (English language). The digital look-alikes were made in conjunction with Sogou. Neither the speech synthesis used nor the gesturing of the digital look-alike anchors were good enough to deceive the watcher to mistake them for real humans imaged with a TV camera.
In September 2018 Google added "involuntary synthetic pornographic imagery" to its ban list, allowing anyone to request the search engine block results that falsely depict them as "nude or in a sexually explicit situation."
At the June 2019CVPR the MITCSAIL presented a system titled "Speech2Face: Learning the Face Behind a Voice" that synthesizes likely faces based on just a recording of a voice. It was trained with massive amounts of video of people speaking.
Since 1 September 2019Texas senate bill SB 751 amendments to the election code came into effect, giving candidates in elections a 30-day protection period to the elections during which making and distributing digital look-alikes or synthetic fakes of the candidates is an offense. The law text defines the subject of the law as "a video, created with the intent to deceive, that appears to depict a real person performing an action that did not occur in reality"
In September 2019Yle, the Finnish public broadcasting company, aired a result of experimental journalism, a deepfake of the President in office Sauli Niinistö in its main news broadcast for the purpose of highlighting the advancing disinformation technology and problems that arise from it.
1 January 2020, Chinese law requiring that synthetically faked footage should bear a clear notice about its fakeness came into effect. Failure to comply could be considered a crime the Cyberspace Administration of China stated on its website. China announced this new law in November 2019. The Chinese government seems to be reserving the right to prosecute both users and online video platforms failing to abide by the rules.
Key breakthrough to photorealism: reflectance capture
The scientific breakthrough required finding the subsurface light component (the simulation models are glowing from within slightly) which can be found using knowledge that light that is reflected from the oil-to-air layer retains its polarization and the subsurface light loses its polarization. So equipped only with a movable light source, movable video camera, 2 polarizers and a computer program doing extremely simple math and the last piece required to reach photorealism was acquired.
For a believable result both light reflected from skin (BRDF) and within the skin (a special case of BTDF) which together make up the BSDF must be captured and simulated.
For believable results also the reflectance field must be captured or an approximation must be picked from the libraries to form a 7D reflectance model of the target.
The whole process of making digital look-alikes i.e. characters so lifelike and realistic that they can be passed off as pictures of humans is a very complex task as it requires photorealistically modeling, animating, cross-mapping, and rendering the soft body dynamics of the human appearance.
Synthesis with an actor and suitable algorithms is applied using powerful computers. The actor's part in the synthesis is to take care of mimicking human expressions in still picture synthesizing and also human movement in motion picture synthesizing. Algorithms are needed to simulate laws of physics and physiology and to map the models and their appearance, movements and interaction accordingly.
Often both physics/physiology based (i.e. skeletal animation) and image-based modeling and rendering are employed in the synthesis part. Hybrid models employing both approaches have shown best results in realism and ease-of-use. Morph target animation reduces the workload by giving higher level control, where different facial expressions are defined as deformations of the model, which facial allows expressions to be tuned intuitively. Morph target animation can then morph the model between different defined facial expressions or body poses without much need for human intervention.
In the late 2010s, machine learning, and more precisely generative adversarial networks (GAN), were used by NVIDIA to produce random yet photorealistic human-like portraits. The system, named StyleGAN, was trained on a database of 70,000 images from the images depository website Flickr. The source code was made public on GitHub in 2019. Outputs of the generator network from random input were made publicly available on a number of websites.
Similarly, since 2018, deepfake technology has allowed GANs to swap faces between actors; combined with the ability to fake voices, GANs can thus generate fake videos that seem convincing.
The speech synthesis has been verging on being completely indistinguishable from a recording of a real human's voice since the 2016 introduction of the voice editing and generation software Adobe Voco, a prototype slated to be a part of the Adobe Creative Suite and DeepMindWaveNet, a prototype from Google.
Ability to steal and manipulate other peoples voices raises obvious ethical concerns.
^Physics-based muscle model for mouth shape control on IEEE Explore (requires membership)
^Realistic 3D facial animation in virtual space teleconferencing on IEEE Explore (requires membership)
^"Images de synthèse : palme de la longévité pour l'ombrage de Gouraud".
^ abcDebevec, Paul (2000). "Acquiring the reflectance field of a human face". Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00. ACM. pp. 145–156. doi:10.1145/344779.344855. ISBN 978-1581132083. S2CID 2860203. Retrieved 24 May 2017.
^Pighin, Frédéric. "Siggraph 2005 Digital Face Cloning Course Notes" (PDF). Retrieved 24 May 2017.
"St. Andrews Face Transformer". Futility Closet. 30 January 2005. Retrieved 7 December 2020.
West, Marc (4 December 2007). "Changing the face of science". Plus Magazine. Retrieved 7 December 2020.
Goddard, John (27 January 2010). "The many faces of race research". thestar.com. Retrieved 7 December 2020.
^In this TED talk video at 00:04:59 you can see two clips, one with the real Emily shot with a real camera and one with a digital look-alike of Emily, shot with a simulation of a camera – Which is which is difficult to tell. Bruce Lawmen was scanned using USC light stage 6 in still position and also recorded running there on a treadmill. Many, many digital look-alikes of Bruce are seen running fluently and natural looking at the ending sequence of the TED talk video.
^ReForm – Hollywood's Creating Digital Clones (youtube). The Creators Project. 24 May 2017.
^Debevec, Paul. "Digital Ira SIGGRAPH 2013 Real-Time Live". Retrieved 24 May 2017.
"Scanning and printing a 3D portrait of President Barack Obama". University of Southern California. 2013. Retrieved 24 May 2017.
Giardina, Carolyn (25 March 2015). "'Furious 7' and How Peter Jackson's Weta Created Digital Paul Walker". The Hollywood Reporter. Retrieved 24 May 2017.
^ abThies, Justus (2016). "Face2Face: Real-time Face Capture and Reenactment of RGB Videos". Proc. Computer Vision and Pattern Recognition (CVPR), IEEE. Retrieved 24 May 2017.
^Suwajanakorn, Supasorn; Seitz, Steven; Kemelmacher-Shlizerman, Ira (2017), Synthesizing Obama: Learning Lip Sync from Audio, University of Washington, retrieved 2 March 2018
^Roettgers, Janko (21 February 2018). "Porn Producers Offer to Help Hollywood Take Down Deepfake Videos". Variety. Retrieved 28 February 2018.
Takahashi, Dean (21 March 2018). "Epic Games shows off amazing real-time digital human with Siren demo". VentureBeat. Retrieved 10 September 2018.
Kuo, Lily (9 November 2018). "World's first AI news anchor unveiled in China". Retrieved 9 November 2018.
Hamilton, Isobel Asher (9 November 2018). "China created what it claims is the first AI news anchor — watch it in action here". Retrieved 9 November 2018.
Harwell, Drew (30 December 2018). "Fake-porn videos are being weaponized to harass and humiliate women: 'Everybody is a potential target'". The Washington Post. Retrieved 14 March 2019. In September [of 2018], Google added “involuntary synthetic pornographic imagery” to its ban list
"NVIDIA Open-Sources Hyper-Realistic Face Generator StyleGAN". Medium.com. 9 February 2019. Retrieved 3 October 2019.
Paez, Danny (13 February 2019). "This Person Does Not Exist Is the Best One-Off Website of 2019". Inverse. Retrieved 5 March 2018.
"§ 18.2–386.2. Unlawful dissemination or sale of images of another; penalty". Virginia. Retrieved 1 January 2020.
"Relating to the creation of a criminal offense for fabricating a deceptive video with intent to influence the outcome of an election". Texas. 14 June 2019. Retrieved 2 January 2020. In this section, "deep fake video" means a video, created with the intent to deceive, that appears to depict a real person performing an action that did not occur in reality
Johnson, R.J. (30 December 2019). "Here Are the New California Laws Going Into Effect in 2020". KFI. iHeartMedia. Retrieved 1 January 2020.
Mihalcik, Carrie (4 October 2019). "California laws seek to crack down on deepfakes in politics and porn". cnet.com. CNET. Retrieved 14 October 2019.
"China seeks to root out fake news and deepfakes with new online content rules". Reuters.com. Reuters. 29 November 2019. Retrieved 8 December 2019.
Statt, Nick (29 November 2019). "China makes it a criminal offense to publish deepfakes or fake news without disclosure". The Verge. Retrieved 8 December 2019.
^Synced (9 February 2019). "NVIDIA Open-Sources Hyper-Realistic Face Generator StyleGAN". Synced. Retrieved 4 August 2020.