Message Boards

GROUPS:

Staff Picks Image Processing Recreation Visual Arts Graphics and Visualization Wolfram Language Know-How

Silvia Hao

Simulating Out-of-focus Effect from Depth Map

Silvia Hao, CTO at Glimscape Technology

Posted 9 months ago

3669 Views

4 Replies

17 Total Likes

Follow this post

out-of-focus effect

In the past Wolfram China Virtual Conference I talked about a simple method to simulate the out-of-focus effect in image processing way. The slideshow was brief with some overlooked issues, so I think it would be nice to refine it a bit, explain in more details and share it to the community here.

Helper functions

ClearAll[pipe,branch,branchSeq]pipe=RightComposition;branch=Through@*{##}&;branchSeq=pipe[branch[##],Apply@Sequence]&;

A question

Given an everywhere clear image (Fig 1) and corresponding depth map (Fig 2), how can one compute an out-of-focus effect (like Fig 3) from them?

Out-of-focus effect, or Bokeh, usually comes from a non-zero lens pupil diameter. In computer graphics field, render engine can produce it naturally. If done in image processing way, it’s a classical spatially varying convolution. Either way it’s a compute-intensive effect. In the following text we will try realizing the effect with pure image processing functions.

Scene setup

As shown in the title image, our basic scene setup includes 2 background walls, a foreground column and the Enneper’s minimal surface in the middle:

In[]:=

Inactive[ParametricPlot3D](*Enneper'sminimalsurface:*)u-

,-v-

,(*background-redwall:*)

Style[

]

,(*background-greenwall:*)

Style[

]

,(*foreground-bluecolumn:*)

SequenceIcon

,{u,-2,2},{v,-2,2},

SequenceIcon

//Activate

Out[]=

But instead of the original 3D model, we only have access to a “color photo” of it:

In[]:=

img=ImportByteArray

orig.png

,"PNG"//Image[#,ImageResolution120]&

Out[]=

and a “depth-map photo” of it:

In[]:=

depthMask=ImportByteArray

depthmap.png

,"PNG"//Image[#,ImageResolution120]&

Out[]=

Thin lens with shallow field of view

The effect comes from defocus aberration. For a perfect lens and shallow enough field of view, the size of the defocus disk (

radius

) is approximately proportional to the distance

δu

from the object to the focal plane.

In[]:=

Graphics(*lens:*)

SequenceIcon

,(*objectsandimages:*)

SequenceIcon

,(*lightrays:*)

SequenceIcon

,(*defocusregions:*)

SequenceIcon

,(*textandmarkers:*)

SequenceIcon



Out[]=

This can be shown easily with the thin lens equation (

is the focus length of the lens):

In[]:=



u0+δu

v0+δv

,(1-λ){0,1}+λ{v0+δv,0}-{v0,radius}//RightComposition[Flatten,Thread[#0]&,Solve[#,{v0,δv,λ,radius}]&,radius/.#&]

Out[]=



fδu

(f-u0)(u0+δu)



So the radius of the defocus disk is:

radius=

f(u-

)

(f-

When

δu≪

, we have

radius∝δu

In[]:=

Series

fδu

(f-u0)(u0+δu)

,{δu,0,2}

Out[]=

fδu

(f-u0)u0

δu

(f-u0)

O[δu]

PSF scaling factor according to radius

Now we have a depth-map, which is an image encoding the value of

per-pixel -- the distance from the camera to an object point:

In[]:=

Manipulate[Rescale[depthMask,{0,1},{umin,umax}],{{umin,.5},0,1},{{umax,2},umin,10},ControlPlacementTop]

Out[]=

So if

is specified, we can compute

δu

easily: (an extra gamma correction is performed to make the result easy to eyes.)

In[]:=

With[{umin=10,umax=20},With[{u=Rescale[depthMask,{0,1},{umin,umax}]},Manipulate[Abs[u-u0]//ImageAdjust[#,{0,0,.2}]&,{{u0,12},umin,umax},ControlPlacementTop]]]

Out[]=

Or just compute the radius directly:

In[]:=

With{umin=10,umax=20,f=4},With{u=Rescale[depthMask,{0,1},{umin,umax}]},ManipulateAbs

1-u0/u

1-u0/f

//ImageAdjust[#,{0,0,.2}]&,{{u0,12},umin,umax},ControlPlacementTop

Out[]=

From the radius map, we will rescale it to

[0,1]

, use it as a scaling factor to get the PSF (point spread function).

In[]:=

{u0,umin,umax,fLens}={5.5,5,7,4};

Image in Wolfram language support “overloading” some numerical functions, so we can write the following code, using the formula directly:

In[]:=

scalingMap=With{u=Rescale[depthMask,{0,1},{umin,umax}]},Abs

1-u0/u

1-u0/fLens

//ImageAdjust

Out[]=

Group image pixels by scaling factor

For scaling map with 8 bit color depth, there are at most

levels of shades:

0≤d≤255

. Using Binarize , we can select regions of any certain shade, which corresponds to the set of pixels with a certain PSF scaling:

In[]:=

ManipulateBinarizescalingMap,

{d,d+1}

255

,{{d,11},0,255-1,1},ControlPlacementTop

Out[]=

Filtering out the relevant shade levels (those correspond to non-empty regions), we can see how the scaling map groups original image img into different partitions:

In[]:=

activeLevelSet=Functiond,BinarizescalingMap,

{d,d+1}

255

//ImageData[#,"Byte"]&//Flatten//Total//{d,#}&/@Range[0,255]//DeleteCases[{_,0}];

In[]:=

Functiond,BinarizescalingMap,

{d,d+1}

255

/@activeLevelSet〚;;,1〛〚1;;-1;;2〛//Accumulate//Mean//ImageAdjust

Out[]=

In[]:=

ManipulateRemoveAlphaChannel[img]+AlphaChannel[img]BinarizescalingMap,

{d,d+1}

255

//0.7#&,{{d,11},activeLevelSet〚;;,1〛,ControlTypeManipulator},ControlPlacementTop

Out[]=

Trick of blurring image with transparency

Now we can apply different scale of blurring effect to different regions. But before that, one thing worth pointing out is functions like ImageConvolve, Blur, GaussianFilter, etc. will NOT apply the effect to the alpha channel:

In[]:=

Moduleblur=GaussianFilter[#,5]&,orig=

//Image[#,ImageResolution36]&,orig//pipe[branch[Identity,pipe[blur,branch[Identity,pipe[ColorSeparate,{Characters@"RGBA",#}&,Grid]],{StringSplit["Full image,Channels",","],#}&,Grid]],{StringSplit["Original image,Blurred",","],#}&,Grid[#,FrameAll,AlignmentCenter]&]

Out[]=

Original image

Blurred

Full image

Channels

R	G	B	A

One solution is re-combining the channels as a general multi-channel image before blurring filter, and converting it back afterwards:

In[]:=

Moduleblur=pipe[ColorSeparate,ColorCombine,GaussianFilter[#,5]&,ColorSeparate,ColorCombine[#,"RGB"]&],orig=

//Image[#,ImageResolution36]&,orig//pipe[branch[Identity,pipe[blur,Image[#,ImageResolution36]&,branch[Identity,pipe[ColorSeparate,{Characters@"RGBA",#}&,Grid]],{StringSplit["Full image,Channels",","],#}&,Grid]],{StringSplit["Original image,Blurred",","],#}&,Grid[#,FrameAll,AlignmentCenter]&]

Out[]=

Original image

Blurred

Full image

Channels

R	G	B	A

Blurring image regions according to PSF scaling

The region-wise blurring filter is straightforward, but please do note the Padding0 option in GaussianFilter , which will play an important roll in alpha blending operation:

In[]:=

AbsoluteTimingresult=Block{rMax=20,img=img//ColorSeparate//ColorCombine,alpha=AlphaChannel@img},Functiond,GaussianFilterimgBinarizescalingMap,

{d,d+1}

255

,

255

rMax,Padding0/@activeLevelSet;;,1//Total//ColorSeparate//ColorCombine[#,"RGB"]&;

Out[]=

{14.6528,Null}

In[]:=

result

Out[]=

The trick of alpha blending

Readers might have noticed the blackish silhouette in our result :

⟹

That is due to Padding0 option in GaussianFilter, which effectively causes an alpha blending with black background across all channels.

In[]:=

RemoveAlphaChannel@result

Out[]=

For the same reason, ImageCompose won’t give correct result if we try to overlay result on top of a background image:

In[]:=

checkerboardBackground=ImportByteArray

checkerboard.png

,"PNG"//Image[#,ImageResolution120]&

Out[]=

In[]:=

ImageCompose[checkerboardBackground,result]

Out[]=

Note the blackish silhouette is still there.

Nor will Overlay help:

In[]:=

Overlay[Image[#,ImageResolution120]&/@{checkerboardBackground,result}]

Out[]=

To correctly perform the overlay, we need to understand what is happening during the blurring filter:

So for each channel of each pixel, the value

is already a pre-blended one between

′

and the Padding value

, with a fraction equal to its alpha channel:

C=α

′

+(1-α)P

To overlay

on top of another pixel

from another image, we need to re-blend

′

with

to get the final color

D=α

′

+(1-α)B=C+(1-α)(B-P)

So the correct way to overlay result on top of a white background should be:

In[]:=

Module[{α=AlphaChannel@result,rgb=RemoveAlphaChannel@result,ℬ=1,=0},rgb+(1-α)(ℬ-)]

Out[]=

Note we should NOT add an alpha channel to aboving result, or else we will get over-blending along the silhouette region:

In[]:=

ImageDifference[%,SetAlphaChannel[%,AlphaChannel@result]//Rasterize[#,RasterSize1000]&]//ColorSeparate//Total//ImageAdjust//

Out[]=

The same operation can be done with any background image:

In[]:=

Module[{α=AlphaChannel@result,rgb=RemoveAlphaChannel@result,ℬ=checkerboardBackground,=0},rgb+(1-α)(ℬ-)]

Out[]=

Possible improvement

When computing the scaling map we used pure image processing functions, which are responsible for the artificial streaks in final result:

To reduce this kind of artifact, we can start from a finer depth map (with larger size and/or larger color bit-depth), to reduce lost of precision due to color quantization.

POSTED BY: Silvia Hao

Answer

4 Replies

Sort By:

Silvia Hao

Silvia Hao, CTO at Glimscape Technology

Posted 8 months ago

I did it wrong on the color and alpha blending in this post. Please refer to [Notebook] [GIF] Color Notes - Alpha blending done properly for more details!

POSTED BY: Silvia Hao

Answer

Silvia Hao

Silvia Hao, CTO at Glimscape Technology

Posted 9 months ago

Thank you for your kind words! The `(* foreground - blue column: *)` part, which is an "iconized" `Sequence[...]`, seems to cause some warning messages, so I inactivate `ParametricPlot3D` to allow the iconized form expanding first. A simple demonstration:

POSTED BY: Silvia Hao

Answer

Moderation Team

Moderation Team, WOLFRAM

Posted 9 months ago

-- you have earned *Featured Contributor Badge* Your exceptional post has been selected for our editorial column *Staff Picks* http://wolfr.am/StaffPicks and Your Profile is now distinguished by a *Featured Contributor Badge* and is displayed on the Featured Contributor Board. Thank you!

POSTED BY: Moderation Team

Answer

Updating Name

Posted 9 months ago

Nice!! Great read and I learned a lot. Just curious about the inactivate/activate use of parametricplot3d. Why do you use that. I'll read your post more because there's a lot to digest.

POSTED BY: Updating Name

Answer

Reply to this discussion

Reply Preview

Attachments

Remove Add a file to this post

Follow this discussion

or Discard

Group Abstract

Feedback