# [WSS18] Image Morphing Using Semantic Segmentation and Shape Interpolation

Posted 1 year ago
4051 Views
|
5 Replies
|
13 Total Likes
|

# INTRODUCTION

Morphing or "metamorphosis" is a technique used to generate a sequence of images that smoothly transform a source image into a target image. This technique is often used to create special effects for motion pictures or television commercials. A conventional method used for generating smooth transformations is by first warping the source and the target images so that the features on both images are aligned to each other and then cross-dissolving the colors of the warped images.

There are two main hurdles to overcome in this approach: -

1. Extracting features from the source and the target images and establishing a correspondence between them: this concerns the amount of effort required by the user to specify the correspondence between the source and the target images.
2. Interpolating image key points to generate features for the intermediate image: this becomes complicated especially when rotation is involved.

Here we attempt to find a solution to the feature interpolation problem. The goal of this project is to create a morph between two images using semantic segmentation and geometric interpolation.

# METHODS

We approached this problem by breaking it into three steps: -

## Segmentation

The first step was to create a segment of the region of interest. This helps isolate feature detection only to the concerned object without picking up the background noise. After unsuccessfully segmenting images using Wolfram Language's image processing features, we turned our attention towards semantic segmentation. For this, we implemented the Ademxapp Model A1 which is a convolution neural network model that performs semantic pixel-wise segmentation. We selected this model because of its high IoU accuracy on the PASCAL VOC2012, a rich classification and segmentation dataset.

NetEvaluate[img_, device_: "CPU"] := Block[
{net, resized, encData, dec, mean, var, prob},
net = NetModel[
"Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data"];
resized = ImageResize[img, {504}];
encData = Normal@NetExtract[net, "Input"];
dec = NetExtract[net, "Output"];
{mean, var} = Lookup[encData, {"MeanImage", "VarianceImage"}];
prob = NetReplacePart[net,
{"Input" ->
NetEncoder[{"Image", ImageDimensions@resized,
"MeanImage" -> mean, "VarianceImage" -> var}],
"Output" -> Automatic}
][resized, TargetDevice -> device];
prob = ArrayResample[prob,
Append[Reverse@ImageDimensions@img, 21]];
dec[prob]
]

# FUTURE WORK

Many different adaptations, tests, and experiments were left out due to lack of time. Some of the ideas that we would have liked to try during the development of this project are: -

1. Implement a shape interpolation scheme to produce results with a bounded amount of conformal distortion
2. Extract matching features for image alignment and local warping
3. Develop a morphing sequence from automatically generated sparse correspondence points

# REFERENCES

• Wolfram Neural Net Repository
• H. Johan, Y. Koiso, T. Nishita, "Morphing using curves and shape interpolation techniques", Proceedings the Eighth Pacific Conference on Computer Graphics and Applications, pp. 348-454, 2000.
Attachments:
5 Replies
Sort By:
Posted 1 year ago
 This is a really nice post. Thank you for sharing. I am looking forward to playing with your code.Thanks,Marco
Posted 1 year ago
Posted 1 year ago
 Did you play around with Inpaint and texture synthesis at all? I feel like the natural next step for this is to remove the old fish, fill in the background with texture synthesis and then put the new fish into the picture. Either way, I'm really excited to work thru and understand your RadialBasisInterpolation function. I've needed this for my own projects and the interpolating function I've made haven't been satisfactory.