Face Morphing Jingyi Li, cs194-bx

In this assignment, I implemented smooth face morphing between two images. I also calculated the mean face of the Danish, extrapolated it for a caricature, played with the perceived gender of my face, and made a smashing music video.


Face Morphing

As pixel coordinates span a linear subspace, we may change their relationships and create "warped" images (in this case, used for morphing faces to each other.)

The first step is to select a bunch of matching image correspondance pairs. Here, we will use Bjoern and Eric as our guinea pigs. After we've selected the points and add in the corner points of the image, we can define an alpha to interpolate to a common point location (such as alpha=.5, a midpoint). We then use Delaunay triangulation on these points to get a nice mesh of triangles to manipulate: since it's the dual of Voronoi cells, it makes triangles fat to help prevent non-local warping (aka ugly results).

Initial Point Selection

Delaunay triangulation overlayed

Once we get the triangles, we can use some linear algebra to compose their affine transformation matrix (in homogenous coordinates, which means the last row is 0s except a 1 in the right corner). As we learned in class, going from the source image to the target image might result in some sampling issues and blank pixels: instead, we will take the inverse of this transformation matrix and apply it to our desired image pixel locations, to get the pixel locations in the original image where we should sample our colors from.

Since the results of the matrix multiplication usually were not integer values like pixel locations, I just rounded to the nearest neighbor. Sadly, this occasionally produces some visible artifacts.

After we have the images warped, we can cross-fade their colors (again with some alpha) to get a combined photo! Were we in want of an animation, we can get multiple frames, and vary alpha from 0 to 1 in frame number of steps.

Bjoern warped to Eric's geometry (midpoint)

With colors crossfaded

Eric warped to Bjoern's geometry (midpoint)

My point selection probably wasn't the best as Bjoern had an ear and Eric didn't, so there are some artifacts.

Below are some example animations, their original photos, and their middle frames. If you want to see all 45 frames of the Bjoern/Eric transition, check out this Dropbox folder.


Prof. Bjoern Hartmann

Less terrifying than their Proj3 spawn.

Prof. Eric Paulos

Hettienne Park

Asian American women in media!

Lucy Liu

Sherlock aka Benedict Cumberbatch

Johnlock aka white men who cares

John aka Martin Freeman

Hugh Dancy aka Will Graham

Madancy/Hannigram (is canon)

I have a lot of feelings about this. Come talk to be about Hannigram.

Mads Mikkelsen aka Hannibal Lecter

It's young me (at 7)!

13 years, and not much has changed.

It's current me (at 20)!

It's me.

I could also probably achieve this by lifting and reading the Bible.

It's Sufjan Stevens.

Middle Images


Mean of Danes

I used the 37 faces provided in this dataset to calcualte the mean face of some Danish people. I did this by first calculating the average shape, and warping each individual person's face into that shape. You can see all 37 here, and below are 3 I found interesting. Note that the control points were only defined on the face, so there are a lot of strange forehead warps.


After that, it was a simple matter of averaging. I warped my face shape into that of the Danish, and then the average Danish person me.


Averages remove individual blemishes and present us with nice, generic symmetry.

Me with the average Danish facial structure.

The average Danish person with my facial structure.

Caricatures

To produce caricatures of people, I first calculated the difference between someone's face shape and the average face shape from a population. I then weighted this by an alpha and added it back onto the original face shape, and morphed to this new shape.

Our Danish friend Mads Mikkelsen again.

The average dane.

Mads Mikkelsen 80% more Danish.

Me.

The average Chinese face.

Me 50% more Chinese.

Me 80% more Chinese.

Bell & Whistle: Changing My Gender

On the same vein as caricatures, I took the average mainlander Chinese man and Chinese woman to see what I would look like if my face weren't so deliberately gender neutral.

Me.

The average Chinese male.

The average Chinese female.

Point definition.

Color morph only (m)

Color morph only (f)

Shape morph only (m).

Shape morph only (f).

Color & shape morph 50% (m)

Male me = bushier eyebrows...

Color & shape morph 80% (m)

Color & shape morph 50% (f)

Female me = higher cheekbones...

Color & shape morph 80% (f)

Bell & Whistle: Music Video

This is probably the pinnacle of my CS career. I took Neil Cicierega's mash up of a John Lennon and Smash Mouth song and, well...just watch it. If you would like to the video in gif form, it's here.

Bell & Whistle: Class Music Video

I participated in it by putting my face in the pool.

Takeaways

This project was super cool, and not just because I could become Sufjan Stevens. It's incredibly beautiful how we can triangulate things relatively arbitrarily yet their boundaries will be continuous. If I had more time, I would try to make my code faster with smarter matrix masking--it took almost an hour to "render" the 55 frames for the video. Solidarity to the rendering folks out there.