Meet MEGANE: A Novel AI Approach For Synthesizing Realistic And Morphable Eyeglass
As social beings, humans daily communicate and express themselves through their behavior and accessories. With the expansion of social life into the online realm through social media and gaming, virtual representations of users, usually called avatars, have become increasingly important for social presence. The consequence is a growing demand for digital clothing and accessories.
Among all the others, eyeglasses are a common accessory worn by billions of people worldwide. For this purpose, in order to achieve realism, work is still needed to model eyeglasses in isolation. Not only the shape but also their interactions with the face must be taken into account. Glasses and faces are not rigid, and they deform each other at contact points, which means that the shapes of eyeglasses and faces cannot be determined independently. Additionally, their appearance is affected by global light transport, and shadows and inter-reflections may appear and affect the radiance. Therefore, a computational approach is necessary to model these interactions in order to achieve photorealism.
Traditional methods use powerful 2D generative models to generate different glasses models in the image realm. While these methods can create photorealistic pictures, the absence of 3D information causes view and temporal inconsistencies in the produced outcomes.
Recently, neural rendering approaches have been investigated to achieve photorealistic rendering of human heads and general objects in a 3D consistent manner.
Although these approaches can be extended to consider faces and glasses models, interactions between objects are not
considered, leading to implausible object compositions.
Unsupervised learning can also be employed to generate composite 3D models from an image collection. However, the lack of structural prior about faces or glasses leads to suboptimal fidelity.
In addition, all these approaches are not relightable, which means that the produced glasses will suffer from inconsistencies in novel illumination conditions.
To overcome the aforementioned issues, a novel AI Morphable Eyeglass and Avatar Network (MEGANE) has been developed.
An overview of the strategy is depicted in the figure below.
Unlike existing approaches, MEGANE is both morphable and relightable, representing the shape and appearance of eyeglass frames and their interaction with faces. A hybrid representation combines surface geometry with a volumetric representation to achieve shape customization and rendering efficiency.
This hybrid representation makes the structure easily deformable based on head shapes, offering direct correspondences across glasses. Furthermore, the model is associated with a high-fidelity generative human head model. This way, the glasses models can specialize in deformation and appearance changes.
The authors propose glasses-conditioned deformation and appearance networks for the morphable face model to incorporate the interaction effects caused by wearing glasses.
Additionally, MEGAN includes an analytical lens model, which provides the lens with photorealistic reflections and refractions.
To jointly render glasses and faces in novel illuminations, the authors incorporate physics-inspired neural relighting into their proposed generative modeling, which infers output radiance given optic and lighting conditions. Based on this relighting technique, the properties of different materials, including translucent plastic and metal, can be emulated within a single model.
The reported results in comparison with the state-of-the-art GeLaTO are reported below.
With an in-deep look at the figures above and according to the authors, GeLaTO lacks geometric details and generates incorrect occlusion boundaries in the face-glasses interaction. MEGANE, on the other hand, achieves detailed and realistic results.
This was the summary of a novel AI framework for 3D-aware Blending with Generative Neural Radiance Fields (NeRFs).
If you are interested or want to learn more about this framework, you can find a link to the paper and the project page.
Check out the Paper and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 15k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.
Daniele Lorenzi received his M.Sc. in ICT for Internet and Multimedia Engineering in 2021 from the University of Padua, Italy. He is a Ph.D. candidate at the Institute of Information Technology (ITEC) at the Alpen-Adria-Universität (AAU) Klagenfurt. He is currently working in the Christian Doppler Laboratory ATHENA and his research interests include adaptive video streaming, immersive media, machine learning, and QoS/QoE evaluation.