Region-Based Face Morphing Under Anthropometric Control

Introduction

    One of the most challenging tasks in graphics modeling is to build an interactive system that allows users to model varied, realistic geometric models of human faces quickly and easily. Applications of such a system range from entertainment to communications: virtual human faces need to be generated for movies, computer games, advertisements, or other virtual environments, and facial avatars are needed for video teleconference and other instant communication programs. Some authoring tools for character modeling and animation are available (e.g., Maya, Poser, DazStudio, PeoplePutty). In these systems, deformation settings are specified manually over the range of possible deformation for hundreds of vertices in order to achieve desired results. An infinite number of deformations exist for a given face mesh that can result in different shapes ranging from the realistic facial geometries to implausible appearances. Consequently, interactive modeling is often a tedious and complex process requiring substantial technical as well as artistic skill. This problem is compounded by the fact that the slightest deviation from real facial appearance can be immediately perceived as wrong by the most casual viewer. While the exiting systems have exquisite control rigs to provide detailed control, these controls are based on general modeling techniques such as point morphing or free form deformations, and therefore lack intuition and accessibility for novices. Users often face a considerable learning curve to understand and use such control rigs.
    To address the lack of intuition in current modeling systems, we aim to leverage the anthropometrical measurements as control rigs for 3D face modeling. Traditionally, anthropometry - the study of human body measurement - characterizes the human face using linear distance measures between anatomical landmarks or circumferences at predefined locations. The anthropometrical parameters provide a familiar interface while still providing a high level of control to users. While this is a compact description, they do not uniquely specify the shape of the human face. Furthermore, particularly for computer face modeling, the sparse anthropometric measurements taken at a small number of landmarks on the face do not capture the detailed shape variations needed for realism. The desire is to map such sparse data into a fully reconstructed 3D surface model. Our goal is a system that uses model priors learned from pre-recorded facial shape data to create natural facial shapes that match anthropometric constraints specified by the user. The system can be used to generate a complete surface mesh given only a succinct specification of the desired shape, and it can be used by expert and novice alike to create synthetic 3D faces for myriad uses.
    We present a new method for interactively generating facial models from user-specified anthropometric parameters while matching the statistical properties of a database of scanned models. We use a three-step model fitting approach for the 3D registration problem. By bringing scanned models into full correspondence with each other, the shape variation is represented by using principal component analysis (PCA), which induces a low-dimensional subspace of facial feature shapes. We explore the space of probable facial feature shapes using high-level control parameters. We parameterize the example models using the face anthropometric measurements, and predefine the interpolation functions for the parameterized example models. At runtime, the interpolation functions are evaluated to efficiently generate the appropriate feature shapes by taking the anthropometric parameters as input. Apart from an initial tuning of feature point positions, our method works fully automatically. We evaluate the performance of our method with cross validation tests. In addition, the anthropometric-based face synthesis method, combined with our database of statistics for a large number of subjects, opens ground for a variety of applications. Chief among these is analysis of facial features of different races. Second, the user can transfer facial feature(s) from one individual to another. This allows a plausible new face to be quickly generated by composing different features from multiple faces in the database. Third, the user can adapt the face model to a particular population group by synthesizing characteristic facial features from extracted statistics. Finally, our method allows for compression of data, enabling us to share statistics with the research community for further study of faces.
    The main contributions of our work are:

Method Overview

    A block diagram of the system architecture is illustrated in the floowing figure. It consists of an offline processing unit and a runtime unit.

    Overview of the data-driven 3D face synthesis system.
    The offline processing makes use of a generic model and 3D scanned models. Using the same generic model for fitting to each scanned model makes trivial the feature correspondence among the scanned models. Feature correspondence is then used to form a low dimensional local shape space of each facial feature. We use the face anthropometric measurements to parameterize the facial feature shape of the examples in a measurement space. The shape synthesizer generates new facial feature geometries using anthropometric measurements as input parameters. It is derived by determining the mapping from the control parameters to the facial feature geometry through an RBF-based interpolation network. For synthesizing local facial textures, the mesh parameterization procedure prepares the example textures in mutual correspondence. Similar to the shape synthesizer, texture constraints derived from the statistics of scanned models are used to guide local texture synthesis. At runtime, the system generates new geometries and textures of individual features from the input parameters by evaluating the constructed synthesizers and PCA models.

Face Data and Preprocessing

    We use the USF face database that contains Cyberware face scans of 186 subjects with various age, gender and race. The laser scans provide face structure data which contains approximately 80,000 surface points and RGB-color values that are stored in a 360*524 image with 8 bit per channel for texture-mapping. We use a generic head model which is a wire-frame of numbered vertices in 3D coordinate space.

    Generic model and scanned face data.
    Since the number of vertices in scanned face models varies, we need to resample all faces in the database so that they have the same number of vertices all in mutual correspondence. Feature points are identified to guide the resampling. In our method, the 3D feature points are identified semi-automatically with a 2D feature mask consisting of polylines groups a set of feature points that correspond to the facial features. Our generic model is already tagged with the corresponding set of feature points by default.

    Left: semi-automatic feature point identification Center: feature points identified on the scanned model. Right: feature points specified on the generic model.

Model Fitting

    A model fitting approch is applied to all scanned faces in the database to establish the vertex-to-vertex correspondence. The approach consists of three steps: global warping, local deformation, and multiresolution model generation.

    Left to right: generic model after global warping; after local deformation; level-2 normal mesh; target scanned model; textured.

    Construction of the normal mesh level by level.

Region-Based Face Shape Synthesis

    Forming Local Shape Spaces

    We perceive the face as a set of features. In this work, the global face shape is also regarded as a feature. We manually partition the generic model into four regions: eyes, nose, mouth and chin. This segmentation is transferred to all normal meshes to generate individualized feature shapes with correspondences. A shape space is then formed for each facial feature using PCA.

    Anthropometric Parameters

    Arguably, face anthropometry provides a set of meaningful measurements or shape parameters that allow the most complete control over the shape of the face. The measurements are taken between the landmark points defined in terms of visually-identifiable or palpable features on the subject's face using carefully specified procedures and measuring instruments.we have chosen a subset of 38 landmarks from the standard landmark set for anthropometric measurements. The measurements are categorized into five types. As shown in the following figure, ch-ch refers to the shortest distance between the landmarks at the corners of the mouth, n-sn refers to the axial distance between the midpoint of the nasofrontal suture and junction between the lower border of the nasal septum, al-prn refers to the tangential distance measured on the face surface from the most lateral point on the nasal ala to the nose tip, the angle of inclination is exemplified by the inclination of the upper lip sn-ls with respect to the vertical axis, and the angle between locations is exemplified by the labiomental angle (the angle at the lower lip).

    Left: Head geometry with anthropometric landmarks (green dots). Right: anthropometric measurements.

    Feature Shape Synthesis

    From the previous stage we obtain a set of examples of each feature with measured shape characteristics, each of them consisting of the same set of dimensions, where every dimension is an anthropometric measurement. As each measurement has a different average and variation, it is prohibitive to define the measurement axis by directly using the measurements unit. Instead, the example measurements are normalized. Generally, we assume that an example model of a facial feature has n dimensions, where each dimension is represented by a value in the interval (0,1]. A value of 1 corresponds to the maximum measurement value of the dimension. The measurements of the example model can then be represented by a n-dimensional vector. This is equivalent to projecting each example model into a measurement space spanned by the n selected anthropometric measurements.
    With the input shape control thus parameterized, our goal is to generate a new deformation of the generic mesh by computing the corresponding eigenmesh coordinates with control through the measurement parameter. Given an arbitrary input measurement in the measurement space, such controlled deformation should interpolate the example models. To do this we interpolate the eigenmesh coordinates of the example models and obtain smooth range over the measurement space. The RBFs are employed in our shape interpolation scheme. Given the input anthropometric control parameters, a novel output model with the desired shapes of facial features is obtained in run time by blending the example models.

    After the shape interpolation procedure, the surrounding facial areas should be blended with the deformed features to generate a seamlessly smooth face mesh. We apply a gradual geometric blending along the subregion boundaries.

    Left: without shape blending, the obvious geometric discontinuities around boundary of the nose region impair realism of the morphing to a large extent. Right: using our approach, geometries of the feature region and surrounding area are smoothly blended around their boundary.

Results

    Our method has been implemented in an interactive system with C++/OpenGL. Our system starts with a mean model which is computed as the average of 186 meshes of the RBF-warped models. In addition to starting with the average head, the user may also select the desired head model of a specific person from the example database for further editing. The following figure illustrates a number of synthesized face shapes. A close view of each feature reveals a wide range of variations across the generated faces; for example, clear differences are found in the width of the nose alar wings, the straightness of the nose bridge, the inclination of the nose tip, the roundness of eyes, the distance between eyebrows and eyes, the thickness of mouth lips, the shape of the lip line, the sharpness of the chin, etc.

    Automatically generated face models by synthesizing the shape of four facial features on the average model (framed red) according to the input anthropometric parameters.

    Close view of synthesized shapes of individual features.
    The global shape control together with local feature synthesis allows us to generate individualized faces which exhibit a wide range of geometric variation. A dynamic morphing can be generated by varying the shape parameters continuously.

    New faces synthesized from the average model (leftmost) with global and local shape variations.

    Face shape morphing (left to right in each example).
    In addition to starting with the mean model, the user may also select the desired head model of a specific person from the example database for further editing.

    Feature-based face editing on the models of two individuals. In each example, the original model is shown in the top-left.
    In order to quantify the performance, we arbitrarily selected ten examples in the database for the cross validation. Each example has been excluded from the example database in training the face synthesis system and its shape measurements were used as a test input to the system. The output model was then compared against the original model. Following figure shows a visual comparison of the result. We assess the reconstruction by measuring the maximum, mean, and root mean square (RMS) errors from the feature regions of the output model to those of the input model. The 3D errors are computed by the Euclidean distance between each vertex of the ground truth and synthesized model. The average errors measured for the ten reconstructed models can be found in our paper.

    Comparison of the ground truth (left in each view) and the synthesized model (right in each view) in cross validation.

Other Applications

    This is useful for classification of face scans. We wish to gain insight into how facial features change with personal characteristics by comparing statistics between groups of faces. We calculate the mean and standard deviation statistics of anthropometric measurements for each facial feature of different groups. The morphometric differences between groups are visualized by comparing the statistics of each facial feature in a diagram. We have followed this approach to study the effects of race and gender.
    In the applications of creating virtual characters for entertainment production, sometimes it is desirable to adjust the face so that it has certain facial features similar to those of a particular person. Therefore, it is useful to be able to transfer desired facial feature(s) between different human subjects. One might wish, given a database of example faces, to select a face or multiple faces to which to adjust facial features. Our high-level facial feature control framework allows the transfer of desired facial features from example faces to a source model in a straightforward manner. We can alter the feature of the source model with a feature-adjustment step which coerces the anthropometric measurement vector to match that of the target feature of an example face. The new shape of the selected feature is reconstructed on the source model and can be further edited if needed.

    Transfer of facial features. We start with a source model (a), and synthesize facial features of the eyes (c), nose (d), mouth (e) and chin (f) on it by coercing shape and texture control parameters to match those of the target face (b).

    Facial features of four example faces (b) in the database are transferred to the source model (a) to generate a novel composite face (c).
    Adapting the model to local populations falls neatly into our framework. The problem of automatically generating a population is reduced to the problem of generating the desired number of plausible sets of control parameters. It is convenient to generate each parameter value independently as if sampled from the Gaussian normal distribution with its mean and variance. The generated control parameter values both respect a given population distribution.

    Left to right: (a) average face; (b), (c) and (d) synthesized faces with the ethnicity of Caucasian, Mongolian and Negroid, respectively; (e) and (f) synthesized male and female faces, respectively.
    For the face synthesis based on a large example data set, the ability to organize examples into database, compress, and efficiently transmit them is a critical issue. The example face meshes used for this paper are restricted from being transmitted in their full resolution because of their dense-data nature. In our method, we take advantage of the fact that the objects under our consideration are of the same class and that they lie in correspondence to compress data very efficiently. Instead of storing instances of geometry data for every example, we adopt a compact representation obtained by extracting the statistics with PCA, which are several orders of magnitude smaller than the original 3D scans. This accounts for the space gain from M times the dimensionality of high-resolution 3D scans (hundreds of thousands), to K (K<<M) times the dimensionality of an eigenmesh (several thousands), with M and K being the number of examples and eigenmeshes respectively.
Papers:

Copyright 2005-2013, Yu Zhang.
This material may not be published, modified or otherwise redistributed in whole or part without prior approval.

Back to my page