Facial modelling techniques
A major unsolved problem in computer graphics is the construction and animation of realistic human facial models. Traditionally, facial models have been built painstakingly by manual digitization and animated by ad hoc parametrically controlled facial mesh deformations or kinematic approximation of muscle actions. Fortunately, animators are now able to digitize facial geometries through the use of scanning range sensors and animate them through the dynamic simulation of facial tissues and muscles. However, these techniques require considerable user input to construct facial models of individuals suitable for animation.
Facial animation is primarily an area of computer graphics that encapsulates models and techniques for generating and animating images of the human head and face.
Facial animation has been a research topic for more than 25 years, aiming at building models of human faces which can be animated and used to produce facial expressions reflecting emotions and mouth movements for spoken text.
2. Facial Modelling Techniques
Face modelling is an important part in the creation of a character because the appearance of the character is a large part of its identity, and the face is probably the first contact users have with him/her.
Face modelling can be considered in two proposes: to create a new face or clone a real face. In the first case, an artist will do the design of the face because there is no data available for a face that does not exist. The designer can start from a generic basic three-dimensional face and use modelling software to modify it.
In case of cloning a real face different techniques are available, which can be split into two parts in regard of the data input used: three-dimensional and two dimensional input.
Polygonal modeling specifies exactly each 3d point, which connected to each other as polygons. This is an exacting way to get topology (points) where you need it on a face and not where you don't.
Advantages: Specific control and placement (of topology and mapping).
Disadvantages: Hard to work with polygon data as you must work with it directly.
2.2 Patches (NURBs)
Patches (or a set of splines) indirectly defines a smooth curve surface from a set of control points. A small amount of control points (called CVs in Maya) can define a complex surface. One type of spline is called NURBs which stands for Non-Uniform Rational B-Splines. This type of batch allows each control point to have its own weight that can affect the "pinch" of the curve at the point. So they are considered the most versatile of batches. They work very well for organic smooth objects so hence they are well suited for facial modeling however several issues arise.
To create a face from NURBS you must use topology (control points) evenly across the face but the face tends to have areas of high detail (eyes, mouth, ears) and areas of low detail (forehead, cheeks, ...). Modeling with batches deals with this in two (not perfect) ways: either you 1) make one large patch with high density everywhere because it is needed in the high detail areas or 2) make a low density patch for the face and use higher density patches specific to facial areas ( ie. Eyes) and stitch them in to the overall face batch.
The two online tutorials demonstrate modeling heads with patches. The first shows creating a even density head gravitating the extra splines into the mouth for its needed detail and the stitching in eyes, at the end it shows how to make one (lame) patch over the whole head.
Other issues are that with faces have many creases (around the eyes, mouth, ears, nose, and wrinkles) and patches present a challenge to do creases in a non-adhoc way.
Sub-Division surfaces is a fairly new modeling technique that gives you the control and flexibility of polygons with the ease of use and smoothness of patches. Tony DeRose (who wrote the paper on Sub-D surfaces and created a working version for Pixar, first used in Geri's Game) has slides on the advantages on sub-d surface
Sub-D surfaces gives you the detail only where you need it.. Paul Aichele discussed this on our Pixar trip with Geri's head. "Rather then fighting the curvature with NURBs", he was able to put the topology where the curvature of the sculpture called for it. See. Sub-D surfaces also gives you better control of texturing, creases and local sharpness. Maya now has Sub-D surfaces in the unlimited version and Sub-D emulation (with polygons) in there complete version.
Facial models can be created by digitizing live humans or physical models. There are several techniques. One that does need an expensive digitizer is using fudical points to reconstruct 2D photographs into a 3D model such as I did for the Kraftwerk music video animation.. Now however, automatic digitizing equipment like that from CyberWare is regularly used to create high resolution 3D models of live human models complete with color data While digitizing models are very useful for many application such as Stanford's Digital Michelangelo Project, typically the data from these systems are too high resolution and not semantically setup for facial animation.
Web based avatar companies and others are using techniques that take a photograph of a human face (sometimes from the front and side) and map it onto a pre-made 3D model that animates by going through a registration process, where key points on the photograph (corners of: the eyes, eyebrows, mouth, ..) are picked via the mouse to register the image with the model. Several researchers are working on automatic registering techniques but lighting conditions on a live face or photograph and other standardization issues plagues the process.
Face generation systems genetically evolve the type of face you want or have you search or surf through a theoretical space of faces as with the program I developed for The Sims and we used on our assignment
Still other face generation systems allow you to pick from a database of facial parts to create a head as you might with a police artist sketch kit such as the Faces system we used as an assignment.
The facial animation is a complex task due to the real structure of the face composed of muscles, bones and skin. The motion of this complex structure is difficult to simulate because small changes make different expressions and humans are used to reading them intuitively.
In different facial animation techniques a scheme is used which describes the relation of the muscle actions and their effects on the facial expression. Ekman and his colleagues developed this scheme, called Facial Action Coding System (FACS), in 1978. This describes the Action Units (AU), which are the smaller visible changes on the human face, in order to associate them with muscles responsible of these changes.
The following describes the facial animation techniques.
The most widely used animation technique is key-framing, where the animator creates key poses of an articulated model and the animation system interpolates the "in between" frames from that set of key-frame data. Typically the data being keyed and interpolated are transformations (move, rotation, scale) of rigid objects such as the hierarchical parts of human body. The problem with facial animation is there really are no rigid parts that move in relation to each other to key-frame. Hence there is a myriad of facial animation techniques based on what you are actually key-framing/interpolating to get smooth animation of a flexible surface of the face.
One widely used basic technique is to create a model of a face in a rest position. Then using essentially modeling techniques, edit the points of a copy of that face to make other faces (typically with the same topology hence the copying of the rest face) in different phoneme and expression states. Then animate a facial animation sequence by morphing (point interpolation) between this set of like-minded faces.
The disadvantage with this technique is that the animator is only picking from a set of pre-made face morphs and thereby limited to the expressions possible from that set in the final animated face sequence. There are several variants of this morphing technique - most notable compound or hierarchical morph targets which allow the animator to blend several faces together with differing weights and/or only morph specific areas of the face. Again, all versions of this technique limit the creative process by only allowing you to pick from a pre-made set of expressions (or force you to stop the animation process to create additional morph targets).
Character animation tools
Tools that were made and are useful for character and organic animation have been used to animate the face. Typically these techniques are not straightforward and are cumbersome because they are not very well suited for a flexible face. They include free form deformations, "bones", point cluster techniques and others.
Fred Parke's early work " Parameterised Model for Facial Animation" in facial animation lead to the first facial animation system, in 1982. It used a control parameterized method where the animation becomes a process of specifying and controlling parameter set values as a function of time. Parameter systems are able to animate the facial expressions as well as the facial types. The FaceLift program used for The Sims uses a simplified version of a production parameterization system. Most parameter systems use Paul Ekman's FACS ( Facial Action Coding System) which describes facial muscle movement as a basis or starting point for specifying and defining the range of parameters. Ken Perlin's web Java-based system also uses a simple but very effective parameterized technique Parameter systems create a very compact specification for facial animation and are therefore ideally suited for the web and games.
Muscle Simulation Systems
Keith Waters work uses a simple simulation of muscle deformation to animate the face. It uses two types of muscles: linear muscles that pull and sphincter muscles that squeeze. He uses a mass and spring technique to animate or deform the skin. The muscle control system has a one to one correspondence to know face muscles and to Ekman's FACS. An extended, fast and open source version of Waters technique with applications for games and our real-time systems is called Expressions
Facial Motion Capture
Facial animation has also been achieved via performance systems where a live performance is digitalized and applied to the facial model rather than created by an animator. Motion capture is the most widely used performance technique. Systems typically track via one or several cameras, small point-like reflective stickers attached in strategic positions on the performers face.
This method is used for animation as well as modelling. A patch could be a Bezier Patch or a B-spline patch, and the surface defined by this patch could be modified through control points. In fact the patch can be used also to control the deformation of polygon mesh, where each vertex is a point of the patch. Because the number of control points is inferior to the number of vertices covert by the patch it is easier to control the deformation in a smoother way.
Speech Generated Systems
Another form of performance animation system uses the voice only to create not only the synched lip movement, but the movement of the other parts of the face as well including the eyebrows, blinks and eye movement, and head movement (neck rotation). These systems analyze the voice to get lip syncing phoneme positions and also use pitch, volume, sentence semantics (dividing speech into sentence sections based on pauses) and other cues to approximate the animation of a faces. These systems in standalone form or combined with photogrametric techniques are being used both in linear animation systems and in real-time web-based applications.
The human face consists of a biological tissue layer with nonlinear deformation properties, a muscle layer knit together under the skin, and an impenetrable skull structure beneath the muscle layer.