Researchers from Walt Disney Studios are using groundbreaking new artificial intelligence technology to change the age of actors on the screen. Performers can be made to look older or younger with a simple click on a screen rather than extensive manual visual effects.
Called FRAN, for Face Re-Aging Network, Disney Research Studios in Switzerland has created a production-ready network that can automatically re-age face images in video without identity loss and with extensive artistic control. Disney said the software provides temporally stable results on videos depicting faces under free viewpoint, depth, motion and illumination conditions.
Though photorealistic digital re-aging of faces in video is becoming increasingly common in entertainment and advertising, it has not been simple to accomplish. The predominant 2D painting workflow often requires frame-by-frame manual work that can take days to accomplish — even by skilled artists.
Recent research on facial image re-aging has attempted to automate and solve this problem, but has resulted in facial identity loss, poor resolution and unstable results across subsequent video frames. In a breakthrough, Disney has now created the first practical, fully-automatic and production-ready method for re-aging faces in video images.
Researchers began by addressing the problem of collecting longitudinal training data for learning to re-age faces over extended periods of time — a task that is nearly impossible to accomplish for a large number of real people. They determined that a longitudinal dataset can be constructed by leveraging the current state-of-the-art in facial re-aging that provides photoreal re-aging results on synthetic faces.
Then they leverage the synthetic data and formulated facial re-aging as a practical image-to-image translation task that can be performed by training U-Net architecture, a convolutional neural network. This was done without the need for complex new network designs.
They use U-Net to advance the re-aging of real faces on video, with unprecedented temporal stability and preservation of facial identity across variable expressions, viewpoints and lighting conditions.
Finally, FRAN incorporates simple and intuitive mechanisms that provides artists with localized control and creative freedom to direct and fine-tune the re-aging effect, a feature that is important in real production pipelines and often overlooked in research.
FRAN can predict which parts of an image are altered by age. For example, wrinkles can be added or removed in a layer over the original image. To create accuracy, the performer’s appearance and identify is preserved, even in movement or in changing lighting configurations. The final result is tweaked by a human artist.
Disney claims FRAN reformulates re-aging as a simple image-to-image translation task that is naturally and effectively solved using the familiar U-Net architecture. As with most deep learning approaches, a large amount of high-quality training data is key in enabling FRAN to work effectively.
FRAN is trained in a simple supervised fashion by leveraging a large longitudinal dataset of photo-realistically re-aged, synthetic face pairs. It then provides realistic and continuous re-aging within a range of 18 and 85 years of age.
Disney said FRAN is the first method to provide high-resolution, temporally stable re-aging results on videos showing faces in different expressions.
The Disney researchers said FRAN is designed with entertainment applications in mind, but they acknowledge its potential for misuse. While techniques for detecting altered images do exist, they said it also important to broadly educate people about the dangers of image and video modification algorithms.
Also, in its current configuration, FRAN has some technical limitations. As with typical U-Net architectures, large image changes are more difficult to generate, making it challenging to re-age to and from very young ages. For such cases, other methods are preferable.
Another limitation is the fact that the graying of scalp hair is not captured in the current training data, and is not reflected in the current images created by FRAN.
Also, re-aging can also introduce variation in body mass index (BMI) whose effects on the face cannot be currently controlled. FRAN does not yet provide artists with a means for specifying what structures are actually added to or removed from the face during re-aging (e.g., moles, specific wrinkling patterns or other particular skin signs).
Currently, FRAN only improves upon the predominant 2D re-aging workflow. More research must be done in cases of 3D re-aging, the researchers said.
This new generation of AI software opens the door to a new opportunities for actors. In the new Indiana Jones 5 film due for release next year, 80-year-old Harrison Ford is de-aged to the same way he looked in the 1980s versions of the film series.
“This is the first time I’ve seen it where I believe it,” Ford said in an interview. “It’s a little spooky. I don’t think I even want to know how it works, but it works.”