Message Boards

GROUPS:

Staff Picks Data Science Image Processing Graphics and Visualization Wolfram Language Machine Learning Wolfram Summer School Video Processing Neural Networks

Jiya Singh

[WSS21] Finding ways to pick optimal frames for video summarization

Jiya Singh

Posted 1 year ago

1955 Views

1 Reply

6 Total Likes

Follow this post

video-summary

Finding ways to pick optimal frames for video summarization
by Jiya Singh

With the growing amount of data on the web, it is hard to process and understand the information we need in short time. Video summarization process plays an important role in helping us with this problem. Video summarization is a procedure of finding key frames that highlight the most informative and representative parts of a video. In this project we focus on finding the best techniques to pick optimal frames for video summarization.
We explore different feature-based approaches and give a comparison between them using the Wolfram Language. This notebook highlights namely two feature extraction methods i.e. image feature extraction and video feature extraction using pretrained neural network models.

Image feature extraction

Video extraction

We use a “Live CEOing Ep 448: Geography Design Review for Wolfram Language 12.3” https://www.youtube.com/watch?v=xVqtBY-sT2w YouTube video in this notebook. We select 1 frame for every 10 seconds from the original video to work on.

In[]:=

video=Video[Import["/Users/jiya/projects/liveceo1.mkv"],RasterSize300,AudioTrackSelection->None]

In[]:=

n=10;dur=QuantityMagnitude[Duration[video],"Seconds"];frames=VideoExtractFrames[video,Range[0,dur,n]];

In[]:=

Length[frames]

Out[]=

418

In[]:=

ImageDimensions[frames[[1]]]

Out[]=

{300,169}

Video frames preprocessing

After getting a total of 418 frames from the video, we use different metrics to remove the similar frames without losing information and compare their performance.

Using ImageDistance

ImageDistance function is a metric used to find distance between two images according to different distance methods. We explore four of these distance methods CosineDistance, NormalizedSquaredEuclideanDistance, RootMeanSquare and CorrelationDistance for reducing the frames by eliminating similar ones.

We randomly pick 200 pairs frames from the video to find the heuristics of the above mentioned distance methods to eliminate similar frames. The threshold for each metric is chosen as 25% quantile of the image distances found between the randomly selected pairs of frames.

In[]:=

distFunctions={CosineDistance,NormalizedSquaredEuclideanDistance,RootMeanSquare,CorrelationDistance};

In[]:=

perm=Permutations[frames,{2}];Length@perm

Out[]=

174306

In[]:=

images=RandomChoice[perm,200];

Cosine Distance

In[]:=

cosDist=ImageDistance[Sequence@@#,DistanceFunction->CosineDistance]&/@images;

In[]:=

Histogram[cosDist]

Out[]=

In[]:=

Quantile[NumericalSort[cosDist],1/4]

Out[]=

0.0163821

NormalizedSquaredEuclidean Distance

In[]:=

normSEDist=ImageDistance[Sequence@@#,DistanceFunction->NormalizedSquaredEuclideanDistance]&/@images;

In[]:=

Histogram[normSEDist]

Out[]=

In[]:=

Quantile[NumericalSort[normSEDist],1/4]

Out[]=

0.221483

RootMeanSquare Distance

In[]:=

rmsDist=ImageDistance[Sequence@@#,DistanceFunction->RootMeanSquare]&/@images;

In[]:=

Histogram[rmsDist]

Out[]=

In[]:=

Quantile[NumericalSort[rmsDist],1/4]

Out[]=

0.176038

Correlation distance

In[]:=

corrDist=ImageDistance[Sequence@@#,DistanceFunction->CorrelationDistance]&/@images;

In[]:=

Histogram[corrDist]

Out[]=

In[]:=

Quantile[NumericalSort[corrDist],1/4]

Out[]=

0.436426

Finding reduced frames using different distance functions of ImageDistance

Using the above thresholds, we reduce the previously obtained 418 frames of the video to reduce the number of frames.

In[]:=

reducedCosDist=DeleteDuplicates[frames,ImageDistance[#1,#2,DistanceFunction->CosineDistance]<0.02&];reducedNSE=DeleteDuplicates[frames,ImageDistance[#1,#2,DistanceFunction->NormalizedSquaredEuclideanDistance]<0.2&];reducedRMS=DeleteDuplicates[frames,ImageDistance[#1,#2,DistanceFunction->RootMeanSquare]<0.2&];reducedCorDist=DeleteDuplicates[frames,ImageDistance[#1,#2,DistanceFunction->CorrelationDistance]<0.4&];

In[]:=

Length/@{reducedCosDist,reducedNSE,reducedRMS,reducedCorDist}

Out[]=

{33,21,19,21}

Using ColorDistance

ColorDistance is another metric used for the process of reducing frames on their similarity. This distance function compares the colors distances in the given images. We again use the randomly selected pairs of frames and run the function.

In[]:=

Histogram[Sqrt@Total[ImageData[ColorDistance@@#]^2,2]&/@images]

Out[]=

In[]:=

colDist=Sqrt@Total[ImageData[ColorDistance@@#]^2,2]&/@images;

In[]:=

Quantile[NumericalSort[colDist],1/4]

Out[]=

43.1604

In[]:=

reducedFramesColDist=DeleteDuplicates[frames,Sqrt@Total[ImageData[ColorDistance@@{#1,#2}]^2,2]<43&];

The following table highlights the performance of above mentioned metrics, i.e. ImageDistance function with four different methods and the ColorDistance function, along with the selected thresholds. These reduced number of frames help in removing the redundancy of similar frames in the video and help in further steps.

In[]:=

table={{"ColorDistance",48,Length[reducedFramesColDist]},{"CosineDistance",0.02,33},{"NormalizedSquaredEuclideanDistance",0.2,21},{"RootMeanSquare",0.2,19},{"CorrelationDistance",0.4,21}};

In[]:=

Grid[Prepend[table,{"Distance Functions","Thresholds","Number of reduced frames"}],Background->{None,{Lighter[Yellow,.9],{White,Lighter[Blend[{Blue,Green}],.8]}}},Dividers->{{Darker[Gray,.6],{Lighter[Gray,.5]},Darker[Gray,.6]},{Darker[Gray,.6],Darker[Gray,.6],{False},Darker[Gray,.6]}},Alignment->Left,Spacings->{4,2}]

Out[]=

Distance Functions	Thresholds	Number of reduced frames
ColorDistance	48	35
CosineDistance	0.02	33
NormalizedSquaredEuclideanDistance	0.2	21
RootMeanSquare	0.2	19
CorrelationDistance	0.4	21

Feature Extractor using NN

Using Inception V3 feature extractor on the reduced number of frames to compare different frames

In[]:=

model=NetModel["Inception V3 Trained on ImageNet Competition Data"];extractor=Take[model,{1,-4}]

Out[]=

NetChain



Input port:	image
Output port:	vector (size: 2048)



Clustering

Using clustering algorithms without predefined number of clusters

Clustering the reduced frames obtained after applying ImageDistance and ColorDistance functions.

In[]:=

fclusters1=FindClusters[extractor[reducedFramesColDist]->reducedFramesColDist,Method->#]&/@{"Agglomerate","GaussianMixture"};fclusters2=FindClusters[extractor[reducedCosDist]->reducedCosDist,Method->#]&/@{"Agglomerate","GaussianMixture"};fclusters3=FindClusters[extractor[reducedNSE]->reducedNSE,Method->#]&/@{"Agglomerate","GaussianMixture"};fclusters4=FindClusters[extractor[reducedRMS]->reducedRMS,Method->#]&/@{"Agglomerate","GaussianMixture"};fclusters5=FindClusters[extractor[reducedCorDist]->reducedCorDist,Method->#]&/@{"Agglomerate","GaussianMixture"};

In[]:=

Length[fclusters1[[#]]]&/@Range[2]

Out[]=

{10,5}

In[]:=

imgCol1=RandomChoice[fclusters1[[1,#]],1]&/@Range[10];imgCol2=RandomChoice[fclusters1[[2,#]],1]&/@Range[5];

In[]:=

Length[fclusters2[[#]]]&/@Range[2]

Out[]=

{8,5}

In[]:=

imgCol3=RandomChoice[fclusters2[[1,#]],1]&/@Range[8];imgCol4=RandomChoice[fclusters2[[2,#]],1]&/@Range[5];

In[]:=

Length[fclusters3[[#]]]&/@Range[2]

Out[]=

{5,5}

In[]:=

imgCol5=RandomChoice[fclusters3[[1,#]],1]&/@Range[5];imgCol6=RandomChoice[fclusters3[[2,#]],1]&/@Range[5];

In[]:=

Length[fclusters4[[#]]]&/@Range[2]

Out[]=

{7,5}

In[]:=

imgCol7=RandomChoice[fclusters4[[1,#]],1]&/@Range[7];imgCol8=RandomChoice[fclusters4[[2,#]],1]&/@Range[5];

In[]:=

Length[fclusters5[[#]]]&/@Range[2]

Out[]=

{5,5}

In[]:=

imgCol9=RandomChoice[fclusters5[[1,#]],1]&/@Range[5];imgCol10=RandomChoice[fclusters5[[2,#]],1]&/@Range[5];

Equally spaced five frames from the original video for comparison with the clustered frames.

In[]:=

seqFrames1=VideoExtractFrames[video,Range[1,dur,dur/5]];

The following table gives a summary of the used metrics and their performance in video summarization process. The last column also gives a comparison of our methods with selecting equally spaced frames from the video for this task.

In[]:=

table={{"Color Distance",48,Length[reducedFramesColDist],10,5,ImageCollage[Length[fclusters1[[2,#]]]&/@Range[Length[fclusters1[[2]]]]->Flatten[imgCol2],BackgroundWhite],ImageCollage[seqFrames1,BackgroundWhite]},{"CosineDistance",0.02,33,8,5,ImageCollage[Length[fclusters2[[2,#]]]&/@Range[Length[fclusters2[[2]]]]->Flatten[imgCol4],BackgroundWhite],ImageCollage[seqFrames1,BackgroundWhite]},{"Normalized Squared Euclidean Distance",0.2,21,5,5,ImageCollage[Length[fclusters3[[2,#]]]&/@Range[Length[fclusters3[[2]]]]->Flatten[imgCol6],BackgroundWhite],ImageCollage[seqFrames1,BackgroundWhite]},{"RootMeanSquare",0.1,60,7,5,ImageCollage[Length[fclusters4[[2,#]]]&/@Range[Length[fclusters4[[2]]]]->Flatten[imgCol8],BackgroundWhite],ImageCollage[seqFrames1,BackgroundWhite]},{"Correlation Distance",0.4,21,5,5,ImageCollage[Length[fclusters5[[2,#]]]&/@Range[Length[fclusters5[[2]]]]->Flatten[imgCol10],BackgroundWhite],ImageCollage[seqFrames1,BackgroundWhite]}};Grid[Prepend[table,{"Distance Functions","Thresholds","Number of reduced frames","Cluster Agglomerate","ClusterGMM","Clustered GMM Images","Equally spaced frames"}],ColumnWidths->{8,6,6,8,6,25},Background->{None,{Lighter[Yellow,.9],{White,Lighter[Blend[{Blue,Green}],.8]}}},Dividers->{{Darker[Gray,.6],{Lighter[Gray,.5]},Darker[Gray,.6]},{Darker[Gray,.6],Darker[Gray,.6],{False},Darker[Gray,.6]}},Alignment->Left,Spacings->{4,2}]

Out[]=

Final Summary by combining above methods

We finally use the frames obtained above after all frame reduction techniques and clustering using GaussianMixture method to give the best representation of the video.

In[]:=

allClusteredImages=Flatten[{imgCol2,imgCol4,imgCol6,imgCol8,imgCol10}];

In[]:=

clusters=FindClusters[allClusteredImages,4,Method->"KMeans"];length=Length[clusters[[#]]]&/@Range[4];

In[]:=

images=RandomChoice[clusters[[#]],1]&/@Range[4];

We also use the length of clusters as weights to make a keyframe summary of the video using ImageCollage.

In[]:=

ImageCollage[length->Flatten[images],BackgroundWhite]

Out[]=

Using clustering algorithms with predefined number of clusters

We apply the same techniques on another video “Live CEOing Ep 446: Discrete Computation Design Review for Wolfram Language 12.3” https://www.youtube.com/watch?v=2v7AGAt0Pw4.
However, this time we use clustering algorithms that require predefined number of clusters as an input which are KMeans and KMedoids clustering algorithms.

In[]:=

video2=Video[Import["/Users/jiya/projects/liveceo2.mkv"],RasterSize300,AudioTrackSelection->None]

In[]:=

n=10;dur=QuantityMagnitude[Duration[video2],"Seconds"];frames2=VideoExtractFrames[video2,Range[0,dur,n]];

In[]:=

Length[frames2]ImageDimensions[frames2[[1]]]

Out[]=

353

Out[]=

{300,169}

Selecting 200 random pairs of frames to determine the heuristics for reducing frames using different distance functions.

In[]:=

perm2=Permutations[frames2,{2}];Length@perm2

Out[]=

123553

In[]:=

images2=RandomChoice[perm2,200];

In[]:=

cosDist2=ImageDistance[Sequence@@#,DistanceFunction->CosineDistance]&/@images2;normSEDist2=ImageDistance[Sequence@@#,DistanceFunction->NormalizedSquaredEuclideanDistance]&/@images2;rmsDist2=ImageDistance[Sequence@@#,DistanceFunction->RootMeanSquare]&/@images2;corrDist2=ImageDistance[Sequence@@#,DistanceFunction->CorrelationDistance]&/@images2;colDist2=Sqrt@Total[ImageData[ColorDistance@@#]^2,2]&/@images2;

In[]:=

Grid[{Histogram/@{cosDist2,normSEDist2,rmsDist2,corrDist2,colDist2}},FrameAll,ItemSize15]

Out[]=

In[]:=

Quantile[NumericalSort[#],1/4]&/@{cosDist2,normSEDist2,rmsDist2,corrDist2,colDist2}

Out[]=

{0.00927195,0.240082,0.129888,0.475991,30.7653}

Finding the reduced frames using different methods and their thresholds

In[]:=

reducedCosDist2=DeleteDuplicates[frames2,ImageDistance[#1,#2,DistanceFunction->CosineDistance]<0.01&];reducedNSE2=DeleteDuplicates[frames2,ImageDistance[#1,#2,DistanceFunction->NormalizedSquaredEuclideanDistance]<0.2&];reducedRMS2=DeleteDuplicates[frames2,ImageDistance[#1,#2,DistanceFunction->RootMeanSquare]<0.1&];reducedCorDist2=DeleteDuplicates[frames2,ImageDistance[#1,#2,DistanceFunction->CorrelationDistance]<0.4&];reducedFramesColDist2=DeleteDuplicates[frames2,Sqrt@Total[ImageData[ColorDistance@@{#1,#2}]^2,2]<30&];

Clustering using KMeans and KMedoids clustering algorithms

In[]:=

clusters1=FindClusters[extractor[reducedNSE2]->reducedNSE2,6,Method->#]&/@{"KMeans","KMedoids"};clusters2=FindClusters[extractor[reducedCosDist2]->reducedCosDist2,6,Method->#]&/@{"KMeans","KMedoids"};clusters3=FindClusters[extractor[reducedRMS2]->reducedRMS2,6,Method->#]&/@{"KMeans","KMedoids"};clusters4=FindClusters[extractor[reducedCorDist2]->reducedCorDist2,6,Method->#]&/@{"KMeans","KMedoids"};clusters5=FindClusters[extractor[reducedFramesColDist2]->reducedFramesColDist2,6,Method->#]&/@{"KMeans","KMedoids"};

In[]:=

imgColKMeans1=RandomChoice[clusters1[[1,#]],1]&/@Range[6];imgColKMeans2=RandomChoice[clusters2[[1,#]],1]&/@Range[6];imgColKMeans3=RandomChoice[clusters3[[1,#]],1]&/@Range[6];imgColKMeans4=RandomChoice[clusters4[[1,#]],1]&/@Range[6];imgColKMeans5=RandomChoice[clusters5[[1,#]],1]&/@Range[6];

In[]:=

imgColKMed1=RandomChoice[clusters1[[1,#]],1]&/@Range[6];imgColKMed2=RandomChoice[clusters2[[1,#]],1]&/@Range[6];imgColKMed3=RandomChoice[clusters3[[1,#]],1]&/@Range[6];imgColKMed4=RandomChoice[clusters4[[1,#]],1]&/@Range[6];imgColKMed5=RandomChoice[clusters5[[1,#]],1]&/@Range[6];

Selecting equally spaced frames for getting a video summary

To give a comparison between using above methods and selecting equally spaced frames from the video for video summarization.

In[]:=

dur=QuantityMagnitude[Duration[video2],"Seconds"];seqFrames2=ImageCollage[VideoExtractFrames[video2,Range[1,dur,dur/6]]];

The following table gives a summary of our results for different frame reduction techniques and the KMeans and KMedoids clustering algorithms. The last column also gives a comparison of our methods with selecting equally spaced frames from the video for this task.

In[]:=

table={{"Color Distance",30,Length[reducedFramesColDist2],ImageCollage[Length[clusters1[[1,#]]]&/@Range[6]->Flatten[imgColKMeans1],BackgroundWhite],ImageCollage[Length[clusters1[[2,#]]]&/@Range[6]->Flatten[imgColKMed1],BackgroundWhite],seqFrames2},{"CosineDistance",0.01,Length[reducedCosDist2],ImageCollage[Length[clusters2[[1,#]]]&/@Range[6]->Flatten[imgColKMeans2],BackgroundWhite],ImageCollage[Length[clusters2[[2,#]]]&/@Range[6]->Flatten[imgColKMed2],BackgroundWhite],seqFrames2},{"Normalized Squared Euclidean Distance",0.2,Length[reducedNSE2],ImageCollage[Length[clusters3[[1,#]]]&/@Range[6]->Flatten[imgColKMeans3],BackgroundWhite],ImageCollage[Length[clusters3[[2,#]]]&/@Range[6]->Flatten[imgColKMed3],BackgroundWhite],seqFrames2},{"RootMeanSquare",0.1,Length[reducedRMS2],ImageCollage[Length[clusters4[[1,#]]]&/@Range[6]->Flatten[imgColKMeans4],BackgroundWhite],ImageCollage[Length[clusters4[[2,#]]]&/@Range[6]->Flatten[imgColKMed4],BackgroundWhite],seqFrames2},{"Correlation Distance",0.4,Length[reducedCorDist2],ImageCollage[Length[clusters5[[1,#]]]&/@Range[6]->Flatten[imgColKMeans5],BackgroundWhite],ImageCollage[Length[clusters5[[2,#]]]&/@Range[6]->Flatten[imgColKMed5],BackgroundWhite],seqFrames2}};Grid[Prepend[table,{"Distance Functions","Thresholds","Number of reduced frames","Clustered KMeans Frames","Clustered KMedoids Frames","Equally spaced frames"}],ColumnWidths->{8,6,6,25,25},Background->{None,{Lighter[Yellow,.9],{White,Lighter[Blend[{Blue,Green}],.8]}}},Dividers->{{Darker[Gray,.6],{Lighter[Gray,.5]},Darker[Gray,.6]},{Darker[Gray,.6],Darker[Gray,.6],{False},Darker[Gray,.6]}},Alignment->Left,Spacings->{4,2}]

Out[]=

Using a general video

To see the performance of our model on a general video, we implement our techniques on a YouTube short film and compare our summary with that using equally spaced frames from the video.

In[]:=

generalVideo1=Video[Import["/Users/jiya/projects/shortmov.mkv"],RasterSize300,AudioTrackSelection->None]

In[]:=

VideoExtractFrames[generalVideo1,{10,20,185}]

Out[]=





In[]:=

duration=QuantityMagnitude[Duration[generalVideo1],"Seconds"];movFrames=VideoExtractFrames[generalVideo1,Range[0,duration-1]];

In[]:=

Length[movFrames]

Out[]=

186

In[]:=

images3=RandomChoice[Permutations[movFrames,{2}],300];thresh1=Quantile[NumericalSort[ImageDistance[Sequence@@#,DistanceFunction->NormalizedSquaredEuclideanDistance]&/@images3],1/4];

In[]:=

movRedFrames=DeleteDuplicates[movFrames,ImageDistance[#1,#2,DistanceFunction->NormalizedSquaredEuclideanDistance]<thresh1&];Length[movRedFrames]

Out[]=

In[]:=

movClusters=FindClusters[extractor[movRedFrames]->movRedFrames,Method"GaussianMixture"];

In[]:=

movImg1=RandomChoice[movClusters[[#]],1]&/@Range[Length[movClusters]]



,

