Message Boards

GROUPS:

Staff Picks Data Science Image Processing Graphics and Visualization Wolfram Language Machine Learning Wolfram Summer School Neural Networks

José Antonio Fernández

[WSS20] Curve OCR for "AP Calculus"-like "sketched" curves

José Antonio Fernández

Posted 2 years ago

5742 Views

2 Replies

11 Total Likes

Follow this post

enter image description here

Introduction and aim

The goal of my project for this year’s Wolfram Summer School was to perform function recognition of curves of the type that can appear in the AP tests (i.e., rough sketches described qualitatively), and be able to compute the typical results that are generally asked in these exam questions: integrals and derivatives, limits, extremum seeking, etc. The methods we use are image processing and neural networks of different kind, in an attempt to obtain automatically the relevant parts of the image for this goal.

The problem in essence reduces to retrieving the image pixels that belong to the function curve. Those pixels will not match the coordinates of the actual graph, and the coordinate system in the image (with origin at the lower-left corner) will be in a different scale, thus we will also need to apply a transformation afterwards. This defines the second task: identify two points in the picture which belong to the graph, along with their transformed counterparts into plot coordinates. First, those will be considered an input to the problem (as well as the breakpoints), and then, attempts to extract them automatically will be explored.

Image Processing approach

The graphical transformations applied here aim at discarding (or hiding) non-relevant information of the image, letting us choose the important parts out of what remains. These methods do not need tedious manual work, but has one limitation: they are image-dependent, meaning they need fine parameter-tuning for every different picture encountered, which further requires a certain technical knowledge from the user. Two examples are analyzed here using this approach. We will first extract two points of the graph by extracting information for the axes labels. (They need not be curve points, we just need to know their coordinates in both systems.) Afterwards, we will retrieve the curve pixel locations.

Example 1

In[]:=

img=

;

We will first invert the colors and standardize the image margins to reduce the differences that can be presented between input images:

plot=ImagePad[ImageCrop@ColorNegate@RemoveAlphaChannel@img,8,Black]

Out[]=

In this particular case, axes labels are well delimited and are identified as independent image components:

components=MorphologicalComponents[Binarize@plot];components//Colorize

Out[]=

We need to set the size of the components we are looking for (estimated to be between 2x10 and 18x24 in size):

axesDigits=ImageClip@Image@SelectComponents[components,(2<#Width<18∧10<#Length<24)&](*Wedon'tsettheminimumwidthhighersincewecouldlosedigits1,whicharenarrow.*)

Out[]=

We will need their image representation (in order to have a network recognize them) and their centroids (in order to reconstruct points in the graph in image coordinates):

In[]:=

digits=Values@ComponentMeasurements[Binarize@plot,{"Image","Centroid"},(2<#Width<18∧12<#Length<24)&];

We will assume there is at least one portion of the graph in the first quadrant and that every digit in the right half of the plane is an x axis label. That implies the points identified in that part of the graph are the actual ones, without negative signs missed in the process.

{xDigits,yDigits}=With[{d=#},Select[digits,#〚2,d〛>Part[ImageDimensions@img/2,d]&]]&/@{1,2}(*Pointsfallingontherightandupperhalfoftheimage,respectively.*)

Out[]=



,{266.722,60.8611},

,{313.863,60.275},

,{360.925,60.9},

,{112.972,223.181},

,{112.355,177.37}

We revert them back to a normal appearance before passed to a pre-trained net for recognition:

In[]:=

clearedDigits=Map[ColorNegate[ImageCrop[#,{28,28}]]&,{xDigits〚All,1〛,yDigits〚All,1〛},{2}];recognisedDigits=NetModel["LeNet Trained on MNIST Data"]/@clearedDigits;

Associate each x axis and y axis digit to its horizontal and vertical coordinate respectively (note we have assumed the numbers are aligned with the axes’ ticks):

In[]:=

digitPositions=Thread/@Thread[recognisedDigits{xDigits〚All,2,1〛,yDigits〚All,2,2〛}];

We take two axes labels only (e.g. the first two), which will allow us to reconstruct the two necessary points:

{{p1Hat,p2Hat},{p1,p2}}=Thread/@(Through[{Keys,Values}[digitPositions〚All,;;2〛]])

Out[]=

{{{3,3},{4,2}},{{266.722,223.181},{313.863,177.37}}}

where Keys gives us the axes labels, and Values the position in the image; by threading them (respectively) we obtain the coordinate pairs.

In order to extract the curve points, we need to get rid of everything but the curve itself, and that is the next task.

The Top Hat Transform, with a thin horizontal row as structuring element, extracts white horizontal peaks:

TopHatTransform[plot,BoxMatrix[{0,16}]]

Out[]=

The same transform, with a thin vertical row instead, extracts white vertical peaks:

TopHatTransform[plot,BoxMatrix[{16,0}]]

Out[]=

By multiplying these two images (pixel-wise), and the following one, we will make 0 every pixel but those of the curve itself:

ColorNegate[axesDigits]

Out[]=

curve=Binarize[TopHatTransform[plot,BoxMatrix[{0,3}]]TopHatTransform[plot,BoxMatrix[{3,0}]]ColorNegate[axesDigits]]

Out[]=

To retrieve the curve points we can just pattern-match the white pixels, who have a value of 1. That gives array indexes, which need to be converted to image coordinates. These two steps are performed by the function labeledPoints (see end):

(points=labeledPoints[ImageDimensions@#,ImageData@#,1]&@curve;HighlightImage[curve,points])

Out[]=

Furthermore, in cases where the breakpoints have a distinctive feature from the background, they can be extracted. Here we have used an opening operation, which basically deletes thin white lines (in our case, everything but the breakpoints):

breakPointComponents=Binarize@Opening[plot,DiskMatrix[4]]

Out[]=

breakPoints=First/@convert[Values@ComponentMeasurements[breakPointComponents,"Centroid"],p1,p2,p1Hat,p2Hat](*Thelastfourargumentsdonotchange*)

Out[]=

{-2.0085,-0.021943,2.00652,5.02115}

Once the points’ coordinates are stored in a list we can use the function convert (see ref [1]; adapted), which receives a list of point coordinates (in image axes) as first argument, the two retrieved required graph coordinate pairs p1 and P2, and their transformed pairs as the rest of arguments:

pHat=convert[points,p1,p2,p1Hat,p2Hat];

We can attempt to retrieve the symbolic expression using the built-in function FindFormula. As of now, this is expected to work well only if applied piece-wise,

findFormulaPiecewise[pHat,breakPoints,SpecificityGoal"High",TargetFunctions{Times,Sqrt,Plus},PerformanceGoal"Quality"]

Out[]=

-1.1558-1.04444x,-1.12861+2.07011x,-1.20044+

20.9952

4.61868

6.72622

1.19828

,



Typically, AP-level functions usually include polynomials, so one choice initially considered to fit the points was spline interpolation, fitting each piece separately. However interpolation parameters do not work equally well for the different functions that can appear in each interval, so it requires user attention to select them appropriately. We will thus use FindFormula from now on.

Example 2

The second example has as purpose show the obstacles that are present in our path towards automation.

In[]:=

img=ImageResize

,500;

In[]:=

plot=ImagePad[ImageCrop@ColorNegate@RemoveAlphaChannel@img,8,Black]

Out[]=

As we start getting the components, we see there are numerous difficulties: double-digit numbers are segmented separately, the axes titles might mistakenly be taken as the digits, ... :

In[]:=

components=MorphologicalComponents[Binarize@plot];axesDigits=ImageClip@Image@SelectComponents[components,(2<#Width<18∧5<#Length<26)&];ImageResize[GraphicsRow[{components//Colorize,axesDigits}],1000]

Out[]=

so in this case it is left as manual input.

Attempting again to isolate the curve (note we have needed to significantly change the parameters):

In[]:=

curve=Binarize[TopHatTransform[plot,BoxMatrix[{0,100}]]TopHatTransform[plot,BoxMatrix[{20,0}]]ColorNegate[axesDigits],.4]

Out[]=

This has not been enough on its own this time, so we manage by extracting the larger components of that result:

In[]:=

SelectComponents[MorphologicalComponents[Binarize@curve],20<#Length&]//Colorize

Out[]=

Another technique that might be attempted is Region Growing, directly over the cropped (not color-negated) image in this case:

In[]:=

plot=ImagePad[ImageCrop@RemoveAlphaChannel@img,8,White];

In[]:=

curve=SetAlphaChannel[plot,ColorNegate[RegionBinarize[plot,{{230,120}},1.65]]]

Out[]=

In[]:=

SelectComponents[MorphologicalComponents[ColorNegate@Binarize[Rasterize@curve,.999]],20<#Length&]//Colorize

Out[]=

In this case, to extract the coordinates, we would need to manually select two points whose transformation is known, since as we have seen retrieving information from the axes has proved difficult.

Region Growing works very well here, but only because the line weight of the function curve is notably thicker and darker than the background items, and we need to make sure we do not put the seed points right onto the function curve (otherwise that will be the first thing to disappear). It can in any case help us remove background elements so that the image components are more clearly defined, to then process them with the other techniques more effectively.

Neural Networks approach

We have generated synthetic datasets, each with different types of training rules for the task towards each is directed, in order to teach some networks different specialized tasks, breaking down the problem into more simple steps. We first thus focus on data generation, and later we will show the different models explored.

Training Data Generation

To train the network we need example graphs, representative of those that it will receive in operation. A training rule is an association between the original example image, and what we target as desired predicted output. For example in the segmentation methods we need to tell the network what class every pixel within an input graph is; therefore the target consists of another image (or more precisely, array, called mask) with the class number in its corresponding pixel place.

The process has been randomized as much as possible in order to create as varied data as possible, and the method used will allow to generate specific types of graphs, and different kinds of masks, for these and other potential future methods.

Define image resolution:

In[]:=

imgSize={240,160}(*forimagesegmentation*);

In[]:=

imgSize={384,384}(*forobjectdetection*);

A rich list of function families has been defined, from which we will sample later pieces to plot. They can be specified with parameters “xL” and “xR” as domain limits, so that they be defined within the interval they later happen to fall in:

In[]:=

functions[xL_,xR_]:=

ListIcon

;

In[]:=

numberFunc=Length@functions[0,0]+1(*numberoffunctionfamiliescontainedinthelist"functions"*);

Let’s associate a class number (integer) to the strings that designate each function family in the list of functions:

In[]:=

funcMap=AssociationThread[Range[2,numberFunc]DownValues[functions]〚1,2,All,1〛];

The number of function pieces that can appear in a single graph, and the number of graphs, are defined a priori; respectively:

In[]:=

p=3;q=300;

The domain of definition:

In[]:=

d=2(*Minimumdomainlengthofeachpiece.*);x0=RandomInteger[{-3,1},q];xn=x0+d+RandomInteger[{0,5},q];

For some methods, we’ll need to know which family each laid-out piece is:

In[]:=

funcLayouts=Table[RandomChoice[Range[2,numberFunc],p],q];

The divisions were the pieces join together are also chosen randomly within the interval, with some care so as to avoid tiny stretches:

In[]:=

d=(xn-x0)/(RandomReal[{1,1.5},q]p)(*Minimumseparationbetweendivisions.Adenominatorofpimpliesevenly-spaceddivisionsbyadistanced*);divisions=With[{i=#},FoldList[RandomReal@{#1+d〚i〛,xn〚i〛-d〚i〛#2}&,x0〚i〛,Reverse@Range[p-1]]~Join~{xn〚i〛}]&/@Range[q](*Inspiredfromanswersin[2].Reverse@Range[(p-1)-t]]isasequenceofthedivisionsremainingtobecomputedwhenthefunctioninthefirstargumentisbeingappliedforthet-thtime.Weneedtoleaveroomintheintervalforthese*);

and those divisions are partitioned into the domain of definition of each piece function:

In[]:=

partitions=First@Flatten[Partition[divisions,{1,2},1],{3}](*intervalsinwhicheachpieceisdefined*);

The actual piecewise functions are laid out, picked from the corresponding family :

In[]:=

pieceLayouts=MapThread[RandomChoice@functions[#2〚1〛,#2〚2〛]〚#1,2〛&,{funcLayouts-1,partitions},2];

We’ve laid out the functions randomly, and as a result most functions are discontinuous at all their breakpoints. Typical AP-test functions indeed have some discontinuities, but not to this extent. We’ll fix this by shifting the laid-out pieces by adding constants appropriately, in order to make them continuous at some randomly-selected breakpoints; this is done by the function joinPiecewiseAt, defined at the end.

In[]:=

If[p≠1,pieceLayouts=MapThread[joinPiecewiseAt[#1,#2,Sort@RandomSample[Range[p-1],RandomInteger@{1,p-1}]]&,{pieceLayouts,divisions}]];

The following variables are used in the generation of the training plots:

In[]:=

{yMin,yMax}={Min/@#,Max/@#}&@MapThread[Table[#1,{x,#2〚1〛,#2〚2〛,.01}]&,{pieceLayouts,partitions},2](*verticalaxis'limits*);t1=Table[RandomChoice@Range[.0023,.011,.001],{q}](*thicknessesofthefunctions'curves*);

We have defined some plotting options common to all graph styles, as randomized as possible, with tried and tested acceptable ranges in order for them to be resemblant to those that typically appear in plots. The list is ever under construction and has come to grow so long that has better been placed at the end of the document.

To enrich the training data, some plots will be generated featuring large dots at the breakpoints, and some others even “callouts”. As preparations :

In[]:=

q1=Floor[q/3];q2=Floor[2q/3]-q1;q3=q-(q2+q1)(*there'llbeq1simplegraphs,q2withbreakpoints,andq3withcallouts*);breakPoints=DeleteDuplicates/@Map[Point,Flatten[MapThread[{{#2〚1〛,#1/.x#2〚1〛},{#2〚2〛,#1/.x#2〚2〛}}&,{pieceLayouts〚#〛,partitions〚#〛}]&@#,1]&/@Range[q],{2}];callouts=Apply[Callout[#,"("<>ToString@Round@#〚1〛<>", "<>ToString@Round@#〚2〛<>")",AppearanceNone,BackgroundNone,CalloutMarkerNone]&,Take[breakPoints,-q3],{2}];callouts=DeleteDuplicatesBy[#,#〚2〛&]&/@callouts(*same-labeledcalloutsaretreatedasduplicates*);

Generation of the simplest graphs of the training set :

In[]:=

options1={commonOptions@#,PlotTheme{"Grid"},GridLinesStyleDirective[RandomChoice[{.8,.2,.1}{Plain,Dashed,Dotted}],RandomChoice[{.8,.2}{Black,GrayLevel@RandomReal@.8}],Thickness[t1〚#〛/RandomInteger@{3,6}]]}&(*optionsspecificforthesimplestgraphs*);graphsSimplest=Show[ListLinePlot[MapThread[Table[{x,#1},{x,#2〚1〛,#2〚2〛,.001}]&,{pieceLayouts〚#〛,partitions〚#〛}],PlotStyleDirective[Black,Thickness@t1〚#〛],options1@#],Method{"AxesInFront"axesInFront}]&/@Range[1,q1];

Out[]=





Generation of the graphs featuring highlighted breakpoints:

In[]:=

options2={options1@#,Epilog{Directive@PointSize[RandomReal@{.008,.022}+t1〚#〛],breakPoints〚#〛}}&(*optionsspecificforthegraphsfeaturingbreakpoints:samestyleassimplest,plusbreakpointsontop*);graphsBreakPoints=Show[ListLinePlot[MapThread[Table[{x,#1},{x,#2〚1〛,#2〚2〛,.001}]&,{pieceLayouts〚#〛,partitions〚#〛}],PlotStyleDirective[Black,Thickness@t1〚#〛],options2@#],Method{"AxesInFront"axesInFront}]&/@Range[q1+1,q1+q2];

Out[]=





Generation of the graphs featuring both highlighted breakpoints and “callouts”, instead of grids:

In[]:=

options3={commonOptions@#,Epilog{Directive@PointSize[RandomReal@{.008,.022}+t1〚#〛],breakPoints〚#〛}}&(*optionsspecificforthegraphsfeaturingcallouts*);auxCallouts=Show[ListLinePlot[MapThread[Table[{x,#1},{x,#2〚1〛,#2〚2〛,.001}]&,{pieceLayouts〚#〛,partitions〚#〛}],PlotStyleDirective[Black,Thickness@t1〚#〛],options3@#],Method{"AxesInFront"axesInFront}]&/@Range[q1+q2+1,q](*auxiliarygraphswiththecalloutsnotyetinplace-usedtoconstructthe"templates"below*);graphsCallouts=MapIndexed[Show[{#1,ListPlot[callouts〚First@#2〛,First@*Cases[Rule[TicksStyle,Directive@OrderlessPatternSequence[___,Rule[FontSize,s_],sr:Rule[FontSlant,_],fr:Rule[FontFamily,_]]]LabelStyleDirective[IntegerPart@s,sr,fr]]@*Options@#1(*calloutfontsizeandfontslantmatchingtheticks'*),PlotMarkersGraphics@{}]}]&,auxCallouts](*thegraphswiththecalloutsinplace*);

Out[]=





All graphs are put together in one variable and they are rasterized:

In[]:=

graphs=Rasterize/@Flatten@{graphsSimplest,graphsBreakPoints,graphsCallouts};

Since borders contain no information, we crop them. (We will need to do this too as a preprocessing step for new images fed into the network.)

In[]:=

cropAmount=BorderDimensions/@graphs(*amounttocropthegraphs.Savedseparatelytocropthemasksidentically.*);graphs=MapIndexed[ImagePad[#1,-cropAmount〚#2〚1〛〛]&,graphs];

The images are resized to network dimensions:

In[]:=

graphs=ImageResize[#,imgSize]&/@graphs;

For our aim to mask the images, we will need some auxiliary images (or “templates” ):

In[]:=

templates=Show[#,Options@#/.Directive[d__]Directive[d,Opacity@0]]&/@Flatten@{graphsSimplest,graphsBreakPoints,auxCallouts}(*we'rejustaddinganextraopacitydirectivetoturnthebackgroundtransparent(everythingexceptthefunctioncurveswearetoidentify)*);

Binary curve mask generation

The first task we will consider is segmenting the curve pixels out of the graphs, i.e. classifying each single pixel in the image as being background (class 1), or curve (class 2). Here are generated these targeted outputs for our training examples.

The mask templates we have defined go through the same three steps as the graphs above; now in one go:

In[]:=

curveBinaryMasks=ImageResize[#,imgSize]&/@MapIndexed[ImagePad[#1,-cropAmount〚#2〚1〛〛]&,Rasterize/@templates];

They need additionally to be binarized, prior to putting the actual class number on each curve pixel:

In[]:=

curveBinaryMasks=Binarize/@curveBinaryMasks;

Out[]=





In[]:=

curveBinaryMasks=((#/.02)&@ImageData@#)&/@curveBinaryMasks;

Multiclass curve mask generation

The second approach explored to identify the piece functions is trying to predict what family of functions each pixel point belongs to. Here we are not just told whether it is background or not, but we are seeking to have the pixels labeled with their corresponding function family out of those defined in the array “functions” above at the beginning. We would afterwards just be left to find the parameters of the family.

The approach here to construct the masks here is to plot every single piece separately, on the exact same background as originally (still made transparent); this will allow us to put the different classes on the different stretches, to then reconstruct the full curve masked with its corresponding label per stretch as we target.

In[]:=

multiclassTemplates=With[{i=#},MapThread[Show[ListLinePlot[Table[{x,#1},{x,#2〚1〛,#2〚2〛,.001}],Options[Flatten[{graphsSimplest,graphsBreakPoints,auxCallouts}]〚i〛]/.Directive[d__]Directive[d,Opacity@0],PlotStyleDirective[Black,Thickness@t1〚i〛]],Method{"AxesInFront"False}]&,{pieceLayouts〚#〛,partitions〚#〛}]]&/@Range[q](*We'replottingeverypieceseparately,onthesamebackgroundastheoriginalpicturenowmadetransparent.*);

We use the same three functions as for the original graphs and the binary masks; however here we have “p” separate templates per graph, so we need to map each of them at one deeper level:

In[]:=

curveMulticlassMasks=Binarize/@#&/@(ImageResize[#,imgSize]&/@#&/@MapIndexed[ImagePad[#1,-cropAmount〚First@#2〛]&,Rasterize/@#&/@multiclassTemplates,{2}]);

To replace the curve pixels with the corresponding function class in every separated curve stretch, we first obtain (via ArrayRules) the position of the curve pixels (by selecting the 0-valued pixel positions). Afterwards these positions are associated with their corresponding class (family) value, what will constitute the new array rules. Doing that for every piece leaves us with as many lists of array rules (or equivalently, images) as pieces. Joining these lists will give a single array (or image), with all positions of the different stretches in it, and thus the full curve masked.

In[]:=

curveMulticlassMasks=SparseArray[Join@@MapThread[(Thread[Cases[ArrayRules[ImageData@#1,1],Rule[{x_,y_},0]{x,y}]#2])&,{curveMulticlassMasks〚#〛,funcLayouts〚#〛}],Reverse@imgSize,1]&/@Range[q];

In[]:=

colorRules=Thread[Range@numberFuncPrepend[RandomColor[numberFunc-1],White]];

Out[]=





Doing this in this way solves the problems that would otherwise arise when curves overlap a bit at the breakpoints, since repeated array rules are all but the first ignored by SparseArray. One of these two curves will then just override the other on these tiny overlapping regions around the breakpoints.

Masks for the breakpoints

The next segmentation task will be an attempt to predict the breakpoints. We note it is not an usual segmentation problem and we could not try to classify this way a handful of pixels out of the tens of thousands that are in the image. Thus, in order to place more importance in those tiny areas of interest (scarce, single pixels) and prevent them from being ignored, we have artificially placed large (pixel-richer) dots on them, regardless of how they look in the original plot.

We could argue this is in reality not an actual “OCR” task, and more of an analytical one. We will see the points here are going to be detected for how they look to the eye, and though this is in principle undesirable, they come in handy and FindFormula will welcome them to do a better job approximating the curves.

Breakpoint masks will be created by laying the breakpoint dots after the whole graph has been made transparent (or “masked” out). So we need to to turn the curve line from the templates transparent as well:

In[]:=

transparentGraphs=templates/.{options:Except[_Line]..,curves:__Line}{options,Opacity@.0,curves};(*Thecurve'soptionscannotbeaccessedviaOptions,andarelocatedtogetherwiththecurveprimitivesthemselves.ThisworksbecauseourgraphshavenootherexpressionswithheadLine,otherwisewe'dneedtobemorespecificonwherewe'redoingthereplacement.*);

The large dots are placed on the breakpoint locations:

In[]:=

breakPointMasks=Show[transparentGraphs〚#〛,Epilog{Directive@PointSize@0.05,breakPoints〚#〛}]&/@Range[q];

And finally they go through the same steps as the curve masks above. In one go:

In[]:=

breakPointMasks=Binarize/@(ImageResize[#,imgSize]&/@MapIndexed[ImagePad[#1,-cropAmount〚First@#2〛]&,Rasterize/@breakPointMasks]);

Out[]=





By having created the image masks including all the original background elements, made transparent, we ensure pixels will accurately match those they correspond to in the original image. (Callouts are not in there but are not a problem, since they’re superimposed onto the graph after the actual plot is generated.)

In[]:=

breakPointMasks=((#/.02)&@ImageData@#)&/@breakPointMasks;

Masks for the axes tick labels

We can further mask out everything in the plots except the boxes bounding the axes’ digits, giving us an easy chance to perform what we have come to call “naive detection”, still sticking to image segmentation. This may well not be a proper “object detection” method, but it happens to give quite very good results with a reasonably small-sized network that does not require heavy training. Furthermore, it still captures a feature of proper “single-shot” detectors (as the result is also given by a single forward pass over the whole image): the network reasons about the whole image and thus objects are detected considering context. This is very important as we do not want to detect any digits that may happen to be laid out around on the image, but we want strictly just those that are tagged to the axes.

To mask the tick labels (i.e. the axes digits) we need to override the opacity and color directives in the fully-transparent graphs we made above specifically for the tick labels:

In[]:=

tickLabelMasks=Show[#,Options@#/.Rule[TicksStyle,Directive@OrderlessPatternSequence[d__,Rule[Background,_]]]Rule[TicksStyle,Directive[d,Opacity@1,BackgroundBlack,FontColorBlack(*blackdigits*),White(*whiteticks*)]]]&/@transparentGraphs;

They undergo the same processing as the rest of masks:

In[]:=

tickLabelMasks=Binarize/@(ImageResize[#,imgSize]&/@MapIndexed[ImagePad[#1,-cropAmount〚First@#2〛]&,Rasterize/@tickLabelMasks]);

Out[]=





In[]:=

tickLabelMasks=((#/.02)&@ImageData@#)&/@tickLabelMasks;

Segmentation Networks Training


Image Segmentation Results

Binary segmentation tasks

In[]:=

colorRules={1White,2Black};

Out[]=

NetChain



Input port:	image
Output port:	array (size: 160×240×2)



Out[]=

NetChain



Input port:	image
Output port:	array (size: 160×240×2)



Out[]=

NetChain



Input port:	image
Output port:	array (size: 160×240×2)



Two segmentation network architectures have been tried: a vanilla one form the documentation, and a UNET architecture, already implemented and generated using the code from [3]. Performance on the synthetic datasets is already very good, so we skip these and jump directly to some real-world data examples.

The UNET has performed better in all cases than the vanilla version. (The comparison is not fully fair since it has been trained for longer and in an updated dataset, but it showed more learning potential, taking more time but showing more steady learning with already quite better results on the old datasets.)

Figures in order: exam plot, prediction using a vanilla net, prediction using UNET architecture.

Out[]=

