Computer aided modeling of human faces still requires a great deal of expertise and manual control to avoid unrealistic, non-face-like results. Most limitations of automated techniques for face synthesis can be described either as the problem of finding corresponding feature locations in different faces or as the problem of separating realistic faces from faces that could never appear in the real world. Using a 3D digitizer/scanner is the most direct method for acquiring the shape of a face. However, 3D digitization requires special high-cost hardware and a powerful workstation. And the digitizing process requires considerable time. The simple and effective method to reconstruct a facial model is the image-based technique. Photogrammetry of a set of images can be used for estimating 3D face shape information. Typically, the same set of surface points are located and measured in two or more different face images. However, the reconstruction of a 3D face model from only one image still remains as a challenge.
I developed a novel image-based 3D face modeling method to reconstruct a photorealistic 3D face model from a single front-view face image in real time. The method adapts 2D geometry of a 3D generic face mesh in the image plane to fit it to the face features extracted from the input image. Exploiting the statistics of a large dataset of 3D face scans, a face depth estimator has been developed to recover domain knowledge about face depths by applying machine learning methods. Given the adapted 2D shape of the generic face mesh, the depth values of all its mesh vertices can be generated from the depth estimator instantly and 3D face model of the specific person in the input image is reconstructed.
Based on this method, I designed and developed a photo-realistic 3D face reconstruction system, 3D Face Synthesizer, that works in real-time with a single frontal-view face image. It is a windows application programmed by C++, OpenGL, and MFC. The software has the following advantages:
The 3D reconstruction is robust, capable of generating a 3D photorealistic face model from a single front-view face image taken by an un-calibrated camera;
Novel approaches to 3D model fitting and texture mapping;
The runtime 3D face reconstruction is computationally efficient and performs in real time;
The approach is based on a generic 3D face model and hence does not require expensive and slow 3D digitizer.
This cutting-edge technology has many significant applications, including virtual agents and avatars, character animation in entertainment (e.g. games and movies) and advertising, computer synthetic facial surgery, model-based teleconferencing, and advanced man-machine interface. This innovative technology is a showcase technology of A*STAR (Agency for Science, Technology and Research) in Singapore. The image-based 3D face reconstruction system has been residing in the FusionWorld, an interactive and multi-sensory showcase gallery of award-winning inventions and cutting-edge technologies from A*STAR's research institutes, since October 2008. It has been demonstrated to many distinguished visitors, including the current Prime Minister of Singapore, Mr. Lee Hsien Loong. This system was also exhibited in SIGGRAPH Asia 2008 and reported by a number of high-impact websites such as CNET, HPCwire, Science Codex, Machines Like Us, etc.
Input 2D image
Feature extraction and fitting
Reconstructed 3D face
Copyright 2005-2013, Yu
Zhang. This material may not be published, modified or otherwise
redistributed in whole or part without prior approval.
