Fun with Frequencies Jingyi Li, cs194-bx

In this assignment, I sharpened images, made hybrid images, calculated gaussian and laplacian stacks, and implemented multiresolution blending. You can click on the photos to view them fullsize.

Part 0: Image Sharpening

I took a lot of concert photos in high school. To clear them up, I used the unsharp mask method we talked about in class: I subtracted the gaussian of an image from itself for clear "edges" (also a high pass filter), and added it onto the original image. Parameters are a sigma for the gaussian and an alpha for the blending coefficient; my defaults are 7 and .75.

Annie Clark (St. Vincent)

Sharp Annie

Thom Yorke (Radiohead)

Sharp Thom

Andrew Bird

Sharp Bird

Since this image was kind of blurry to begin with, sharpening does not help that much.

Part 1: Hybrid Images

If images are far away, we see their low frequencies (aka the blurry stuff). If they're close by, we see their high frequencies (aka the sharp edge stuff). Using this logic, you can create hybrid images as detailed as in this paper! The low pass filter is a gaussian and the high pass filter is the original image minus its gaussian. Parameters that are changable are the window size of the gaussian filter kernel and the sigma of the filter; my default values are 21 and 5e-3, respectively, for the low pass, and 15 and 5e-4 for the high pass. This is how I get results which look nice.

Original: Nicki

Original: Beyonce

Nicki low pass filtered

Beyonce high pass filtered

Hybrid image!

FFT visualization of each image

It's clear from this analysis that the spectrum of the hybrid image is a multiplication of the spectrums of the LPF and HPF images.

Bells & Whistles: Color

For the following examples, I implemented color hybrid images. In my opinion, all that matters is that the low pass filtered image is in color: the color effect of the high frequencies is often trivial since we perceive them as "details". (That being said, both images are in color here).

Derek & Nutmeg, both black/white

Derek & Nutmeg, Derek (low) color

Derek & Nutmeg, Nutmeg (high) color

Derek & Nutmeg, both color

Hybrid Niyonce

Hybrid Becki

Original: My dad Jay

Original: My kitten Emmet

Hybrid CatDad!

Original: Trump

A lot of my classmates enjoy more meme-like Trump morphs (corn, troll dolls, etc), but here is my vision for what America will become under his leadership.

Original: Hitler

Hybrid Trump Hitler!

Original: Ruby

Original: Sapphire

Garnet?

This is a failed example. Hybrid images don't work well with cartoons since so few frequencies exist to begin with, as they're flat shaded. (Also, everyone should watch Steven Universe.)

Part 2: Gaussian & Laplacian Stacks

Stacks are created when users specify an image to create a stack of, and N, a depth, and optionally a window size for the filter. Depth is calculated by power of 2 sigams. Gaussian stacks are progressively increasing blurs, while Laplacian stacks are the gaussian at each current point minus the gaussian of the last point. For the images below, I used depth=5 and a window size of 10. I implemented it in color (is that a bell & whistle?)

For Lincoln portrait painted by Dali, you can clearly see the woman at Laplacian level 1 (high frequencies). At level 4, it's clearly a picture of Lincoln (low frequencies). For the Mona Lisa, the different frequencies reveals how Da Vinci played with peripheral vision to keep things interesting.

Original: Lincoln

Gaussian Lvl 1

Gaussian Lvl 2

Gaussian Lvl 3

Gaussian Lvl 4

Gaussian Lvl 5

Original: Lincoln

Laplacian Lvl 1

Laplacian Lvl 2

Laplacian Lvl 3

Laplacian Lvl 4

Laplacian Lvl 5


Original: Mona Lisa

Gaussian Lvl 1

Gaussian Lvl 2

Gaussian Lvl 3

Gaussian Lvl 4

Gaussian Lvl 5

Original: Mona Lisa

Laplacian Lvl 1

Laplacian Lvl 2

Laplacian Lvl 3

Laplacian Lvl 4

Laplacian Lvl 5


Original: Becki

Gaussian Lvl 1

Gaussian Lvl 2

Gaussian Lvl 3

Gaussian Lvl 4

Gaussian Lvl 5

Original: Mona Lisa

Laplacian Lvl 1

Laplacian Lvl 2

Laplacian Lvl 3

Laplacian Lvl 4

Laplacian Lvl 5

Part 3: Multiresolution Blending

As detailed in this paper, multiresolution blending takes advantage of stacks of frequencies and combines images across each of their Laplacian stacks. Using a black/white mask as a weighted guide, final image = sum((m)a + (1-m)b) for each for the two images a and b. You also have to add the lowest gaussian back in, too, since a laplacian is a subtraction of two gaussians. In normal people talk, you blend each frequency separately (so you add the blurry parts of one image to the blurry part of the other image, the sharp parts to the sharp parts, etc) and sum them up at the end.

For each image, I played around with two parameters: size of the gaussian filter and stack depth. (Default ones were window 4, depth 6). I also did color as a bell & whistle; it's not under a separate heading because all my results are in color. (This was also my biggest challenge: when I was combining all the channels in the end, my colors end up "muted"--I think it deals with slight grey variations in the Laplacians.)

The Apple.

The Orange.

The Orapple mask.

The Orapple, in black and white.

The Orapple in color.

A tweak: to get more of the details in, I multiplied Laplacian levels 1 and 2 by 1.25.

Professor Eric Paulos.

Professor Bjoern Hartmann.

Their non-trivial mask.

Together, they probably push 100 CHI papers each year.

Bjoern's Laplacian stack of depth 6.

Eric's Laplacian stack of depth 6.

I don't know why the first image is so dark...

The masks's Gaussian stack of depth 6.

The frequencies to be added, of depth 7.

7 since you also need to add back in the last gaussian. You can see Eric's dark eyes on depth 1.

Robert Pattinson.

A foot.

Their non-trivial mask.

Suddenly, we're back in 2008.

Hannibal Lecter.

Please watch Hannibal.

A stag.

The Windigo, but really less terrifying.

Ooh, contrast.

A (kind of) bell and whistle: automatic histogram leveling, so it's even more ~HDR~

Soda Hall.

A fire.

Fire mask.

A bad blend.

The fire looks really bad because the background of the two images do not match up at all. Gradient domain blending would be better here. Or just setting the image on screen in Photoshop.

My sister.

Shrek.

Face mask.

Nice.

Takeaways

As someone who has been using Photoshop since they were 12, learning the DSP side behind images is super neat. The coolest thing I learned from the assignment, besides the general knowledge that images may be decomposed and manipulated via their frequencies instead of just their raw pixels (as Photoshop's interface implies), was probably the warm up: the unsharp mask. It's a super simple idea, but implemented in many phones because of its simplicity. It's amazing how we can "trick" ourselves to perceive an image "sharper" than it actually is, but not actually gain anything information wise.