https://community.wolfram.com RSS Feed for Wolfram Community showing any discussions tagged with Artificial Intelligence with no replies sorted by active. https://community.wolfram.com/groups/-/m/t/3577512 ![enter image description here][1] Hi everybody! In just about six weeks, our online [Neural Networks Boot Camp][2] will begin. This two-week boot camp will feature lectures from Wolfram instructors and developers from the machine learning team. You'll learn what neural networks are, how you can use pre-trained networks and how to build and train your own models. Explore ways that AI and deep learning can be applied to text, image and audio analysis. Earn certification for: - **Program completion** (by attending and actively participating in the sessions) - **Level 1 proficiency in neural networks** (by completing graded exercises) - **Level 2 applied expertise in neural networks** (by building and training your own neural networks) This boot camp has discounted pricing for students, educators and even hobbyists. > [**REGISTER HERE**][3] ![enter image description here][4] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot_2025-11-1816.21.53.png&userId=1711324 [2]: https://www.wolfram.com/wolfram-u/boot-camp-neural-networks/ [3]: https://www.wolfram.com/wolfram-u/boot-camp-neural-networks/ [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=WolframUBanner.jpeg&userId=1711324 Arben Kalziqi 2025-11-18T22:25:23Z https://community.wolfram.com/groups/-/m/t/3323587 ![enter image description here][1] Hi everybody! In just about six weeks, our online [Neural Networks Boot Camp][2] will begin. This two-week boot camp will feature lectures from Wolfram instructors and developers from the machine learning team. These lectures will cover topics ranging from what a neural network even is all the way to describing generative pre-trained transformers (GPTs) and neural network applications in image, audio and video analysis. We will also cover much of the new LLM-based functionality that has been integrated into Wolfram Language, including some functionality which is yet to be announced :). Earn certification for: - **Program completion** (by attending and actively participating in the sessions) - **Level 1 proficiency in neural networks** (by completing graded exercises) - **Level 2 applied expertise in neural networks** (by building and training your own neural networks) > [**REGISTER HERE.**][2] ![enter image description here][3] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot_2024-11-1816.39.10.png&userId=1711324 [2]: https://www.wolfram.com/wolfram-u/boot-camp-neural-networks/ [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=WolframUBanner.jpeg&userId=1711324 Arben Kalziqi 2024-11-18T22:43:26Z https://community.wolfram.com/groups/-/m/t/3098690 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=9760image.jpeg&userId=911151 [2]: https://www.wolframcloud.com/obj/fa97fc66-dfc6-42d2-ad39-b5c612ca0097 Wolfram Education Programs 2024-01-09T16:08:37Z https://community.wolfram.com/groups/-/m/t/3094975 ![lottery neural nets][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=9580image.jpeg&userId=911151 [2]: https://www.wolframcloud.com/obj/63013bc5-ddfa-4ca1-9f29-a70a3efe4506 Wolfram Education Programs 2024-01-02T19:16:32Z https://community.wolfram.com/groups/-/m/t/3094693 ![language learning][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=7107image.jpeg&userId=911151 [2]: https://www.wolframcloud.com/obj/b692ee5f-d732-4087-a163-ddcef47c9244 Wolfram Education Programs 2024-01-02T17:13:48Z https://community.wolfram.com/groups/-/m/t/3094670 ![machine learning][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=4889image.jpeg&userId=911151 [2]: https://www.wolframcloud.com/obj/29e809c2-c42c-4491-a69e-c1ca763ab6a1 Wolfram Education Programs 2024-01-02T17:07:26Z https://community.wolfram.com/groups/-/m/t/2818058 ![classifier ][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2023-01-27at10.31.22AM.png&userId=911151 [2]: https://www.wolframcloud.com/obj/a266d746-090f-445a-a10f-7113b3e76767 Wolfram Education Programs 2023-01-27T16:33:55Z https://community.wolfram.com/groups/-/m/t/2144200 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/d34dbf01-6652-4598-9cff-6aad9b015495 Wolfram Education Programs 2020-12-21T18:20:08Z https://community.wolfram.com/groups/-/m/t/2144104 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/723658f2-7fbd-40e9-b430-1f8271427990 [Original]: https://www.wolframcloud.com/obj/4cbab921-c1df-4a1b-be33-dfd6289fe860 Wolfram Education Programs 2020-12-21T16:24:15Z https://community.wolfram.com/groups/-/m/t/1872736 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/7552c952-8321-4ea8-804a-b835876bcaa1 Wolfram Education Programs 2020-02-04T19:43:47Z https://community.wolfram.com/groups/-/m/t/1379517 ![enter image description here][1] #Rooftop Recognition for Solar Energy Potential The aim of this project is to detect the rooftop of buildings to determine the available area at different locations and to identify the most suitable ones for solar energy application such as solar PV using Neural Networks and satellite imagery. Github link for files and notebooks: https://github.com/enricocg/Project The Dataset =========== The Inria Aerial Image Labeling addresses a core topic in remote sensing: the automatic pixelwise labeling of aerial imagery. Dataset features: - Coverage of 810 km (405 km for training and 405 km for testing) - Aerial orthorectified color imagery with a spatial resolution of 0.3 m - Ground truth data for two semantic classes : building and not building (publicly disclosed only for the training subset) [https : // project.inria.fr/aerialimagelabeling/][2] Select the images for the input and those for the results: trainFilesInput = Select[FileNames[ "image/*.tif"], ! StringMatchQ[#, ___ ~~ "._" ~~ ___] &]; trainFilesResult = StringReplace[#, "image/" -> "mask/"] & /@ trainFilesInput; Import an image to test: imgInput = Import[trainFilesInput[[1]]]; imgResult = Import[trainFilesResult[[1]]]; Partition of the images in 100 from a 5000x5000 to images 500x500 : splicesInput = Join @@ ImagePartition[imgInput, 500]; splicesResults = Join @@ ImagePartition[imgResult, 500]; Assemble Images --------------- Assemble images in sets of ten to verify: assambleImage = ImageAssemble[splicesInput[[41 ;; 50]]] ![Image assembly for row 4][3] Assemble mask images in sets of ten to verify: assambleMask = ImageAssemble[ Image /@ Round[ImageData /@ splicesResults[[41 ;; 50]]]] ![Mask assembly for row 4][4] Compose ------- Compose the images to verify image matching: ImageCompose[assambleImage, {assambleMask, 0.5}] ![Image composition of row 4 with mask][5] rand = RandomInteger[{1, 100}]; ImageCompose[splicesInput[[rand]], {splicesResults[[rand]], 0.5}] ![Image composition of a random image][6] Association ----------- mxTrain = Thread[splicesInput -> splicesResults]; ImageCompose[Keys[mxTrain[[rand]]], {Values[mxTrain[[rand]]], 0.4}] ![Image association][7] Export the MX file Export["File.mx", mxTrain] Export ------ **mxFileCreator** The first approach to organize the data was to make MX files, one per image, each file contain the 100 images with their respective mask. In order to do that a function mxFileCreator was build. Function that creates a MX file per each 5000x5000 image: mxFileCreator[trainFilesInput_,trainFilesResult_,i_]:=Block[ {Flag,imgInput,imgResult,splicesInput,splicesResults,mxTrain}, Flag=TextString[i]; imgInput=Import[trainFilesInput]; imgResult=Import[trainFilesResult]; splicesInput=Join@@ImagePartition[imgInput,500]; splicesResults=Round[ImageData/@(Join@@ImagePartition[imgResult,500])]; mxTrain=Thread[splicesInput-> splicesResults]; Export["MXFiles/File"<>Flag<>".mx",mxTrain] ] Test the function mxFileCreator[trainFilesInput[[1]], trainFilesResult[[1]], 1] "MXFiles/File1.mx" file = Import["MXFiles/File1.mx"]; file[[RandomInteger[{1, 100}]]] ImageCompose[Keys[file[[rand]]], {Image[Values[file[[rand]]]], 0.5}] ![mxFileCreator test][8] To Map all the images and convert them into association in a MX files : MapIndexed[mxFileCreator[#[[1]], #[[2]], Echo[#2[[1]]]] &, Transpose[{trainFilesInput, trainFilesResult}]] The net ======= The net selected for this project was at Wolfram Neural Net Repository for Semantic Segmentation. Released in 2016 by the University of Adelaide, Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data was modify to identify two classes instead of 21. Take the net model for Semantic Segmentation from the repositories: netModel = NetModel["Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO \ Data"] ![Net model for Semantic Segmentation from the repositories:][9] Net surgery ----------- **Modify the Input for the size of the images** netModel500 = NetReplacePart[netModel, "Input" -> NetEncoder[{"Image", {500, 500}, "MeanImage" -> {0.485, 0.456, 0.406}}] ] ![Replace the encoder to accept images 500x500][10] **Resize the layer** Drop the last three layer to modify the net: firstPartNet = NetDrop[netModel500, -3] **Convolution layer** Add a convolution layer to have two outputs: convLayer=NetChain[{firstPartNet, ConvolutionLayer[2, {3, 3}, "Stride" -> {1, 1}, "PaddingSize" -> {12, 12}, "Dilation" -> {12, 12}], ResizeLayer[{500, 500}]}]; Take the last two layers from the one with the new encoder: lastPartNet = NetReplacePart[NetTake[netModel500, -2], "Input" -> Automatic] **SoftmaxLayer** Append the last layers and Softmax Layer with a net decoder of classes: finalNet = NetAppend[convLayer, {lastPartNet[[1]], SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", {0, 1}, "InputDepth" -> 3}]] **Initialize the net** Initialize the final net to check for errors: iniFinalNet = NetInitialize[finalNet] ![Final net][11] Net to train ------------ **Loss Function** Connect the final net to a loss function: LossNet = NetGraph[<|"eval" -> finalNet, "loss" -> CrossEntropyLossLayer["Index"]|>, {"eval" -> "loss"} ] Initialize the LossNet: iniLossNet = NetInitialize[LossNet] Export the complete net for training: Export["iniLossNet.wlnet", iniLossNet] Generator function ================== Reduce Dataset Size ------------------- From the MX files created in section. Set the file path for the MX files: path = "MXFiles/"; mxFiles = FileNames[path <> "*.mx"]; RandomSample[mxFiles, 1][[1]] **Binary Files** Binary files are a way to reduce of the size while importing the train set. Once the files are binarize the file size reduces and ones read in can be deserialize. At the same time TIF images were converted to JPG to remove unnecessary information while reducing size. Binary write: binaryWrite[file_,expr_]:=With[{bytes=BinaryWrite[file,BinarySerialize[expr]]}, Close[file]; bytes] Function to convert TIF images to JPG and matrix to Binary: importAndReExport[path_,folder_]:=Block[{imported,imgs,masks,hashes}, imported=Import[path]; imgs=imported[[All,1]]; masks=imported[[All,2]]; hashes=Hash/@imgs; Print[MapThread[Export[folder<>"/"<>ToString[#1]<>".jpg",#2]&,{hashes,imgs}];//AbsoluteTiming]; Print[MapThread[binaryWrite[folder<>"/"<>ToString[#1]<>".bin",#2]&,{hashes,masks}];//AbsoluteTiming]; Clear[imported,imgs,masks,hashes] ] Take each MX file and create the correspondent JPG and BIN file : Table[importAndReExport[mxFiles[[i]], "binFiles"], {i, 1, Length[mxFiles], 1}] **Binary Read** Read the binary file and Deserialize the file: BinaryDeserialize[ReadByteArray["/Users/enricocastro/Documents/GitHub/Project/binFiles/\74004748475675200.bin"]] // Dimensions Training Out of Core ==================== Partitional Function -------------------- **Data from MX files** Take the name of the JPG and BIN files: trainingDataSet = {FileNames["binFiles/*.jpg"], FileNames["binFiles/*.bin"]}; Set the data to train: imageDataSet = Table[File[trainingDataSet[[1, i]]], {i, 10}]; maskDataSet = Table[ReadByteArray[trainingDataSet[[2, i]]], {i, 10}]; Associate the image with his respective mask and take a random sample: data = RandomSample@Thread[imageDataSet -> maskDataSet]; data[[2]] Generator --------- In order to train the net with a big amount of data a generator function was created. The generator function load a single batch of data from an external source to train each time. NetTrain[net, f, \[Ellipsis]] calls f at each training batch iteration, thus only keeping a single batch of training data in memory. The function can depend on the batch size, which can be set or set automatic by the computer, the absolute batch that is the number of batches load during the training and the round. partGenerator associate and load a batch of size BatchSize into the net training. In order to do so it takes partitions of the whole dataset in batches of size BatchSize and load the element of this partition taking the module of the AbsoluteBatch. The function also adds one to the mask matrix to get the right values for the net: partGenerator=Function[Block[{batch,dataSet,batchData}, If[!ValueQ[partitionedData],partitionedData=Partition[Range[Length[data]],#BatchSize,#BatchSize,1]]; Print[Association@Thread[Keys[#]->Values[#]]]; batch=Mod[#AbsoluteBatch,Floor[Length@data/#BatchSize]]; batchData=data[[partitionedData[[Echo@(batch+1)]]]]; Thread[Keys[batchData]->1+BinaryDeserialize/@Values[batchData]] (*<|"Input"\[Rule]Keys[batchData],"Target"\[Rule]BinaryDeserialize/@Values[batchData]|>*) ] ]; **Test of partGenerator** partGeneratorData = partGenerator[<|"BatchSize" -> 2, "Round" -> 0, "AbsoluteBatch" -> 1|>] ![<|BatchSize->2,Round->0,AbsoluteBatch->1|>][12] Train test ---------- Before training for a lot of data is always good to try first with a small set. Clear partitionedData for different batch sizes: ClearAll[partitionedData]; iniLossNet = Import["iniLossNet.wlnet"] Set the checkpoint directory: checkpointDir = "checkpoint" Net training with partGenerator: iniLossTrainNet = NetTrain[iniLossNet, {partGenerator, "RoundLength" -> 3}, All, MaxTrainingRounds -> 3, TrainingProgressCheckpointing -> {"Directory", checkpointDir}] Trained Net =========== Once trained one can look at the relevant information of the training process. Import the trained net as an object: **Encoder & Decoder** Set the encoders and decoders for further tests. Set the encoder for image size 500x500 and a MeanImage: enc = NetEncoder[{"Image", {500, 500}, "MeanImage" -> {0.485, 0.456, 0.406}}]; Set the decoder for the classe which 0 means no rooftop and 1 means rooftop: dec = NetDecoder[{"Class", {0, 1}, "InputDepth" -> 3}]; **Training information** ![finalTrainedNet["LossEvolutionPlot"]][13] ![finalTrainedNet["ErrorRateEvolutionPlot"]][14] ![Properties][15] Extract the trained net: finalTrainedNetEval = NetExtract[finalTrainedNet["TrainedNet"], "eval"] Set the encoder: finalTrainedNetEvalEnc = NetReplacePart[finalTrainedNetEval, "Input" -> enc ] Set the decoder: finalTrainedNetEvalDec = NetReplacePart[finalTrainedNetEvalEnc, "Output" -> dec ] Export the net to test: Export["FinalTrainedNetEval.mx", finalTrainedNetEvalDec] Net Evaluation ============== netEvaluation = Import["/Volumes/ECG/ProjectWSS/AerialImageDataset/\ FinalTrainedNetEval.mx"] Test Files ---------- Take a random file from the training set: randomFile = RandomSample[FileNames["image/*.tif"], 1]; TestImage = Import[randomFile[[1]]]; TestMask = Import[StringReplace[randomFile[[1]], "image/" -> "mask/"]]; Take 100 partition of the image to feed into the net: imageToTest = ImagePartition[TestImage, {500, 500}]; maskToTest = ImagePartition[TestMask, {500, 500}]; rand1 = RandomInteger[{1, 10}]; rand2 = RandomInteger[{1, 10}]; ImageCompose[ imageToTest[[rand1, rand2]], {Image[netEvaluation[imageToTest[[rand1, rand2]]]], 0.5}] ![Image test][16] ImageCompose[ maskToTest[[rand1, rand2]], {Image[netEvaluation[imageToTest[[rand1, rand2]]]], 0.5}] ![Mask comparision][17] Compare the percentage of building in the area with the net and the mask: {{"Net", "Mask"}, {ImageMeasurements[ Image[netEvaluation[imageToTest[[rand1, rand2]]]], "Mean"], ImageMeasurements[maskToTest[[rand1, rand2]], "Mean"]}} // Dataset ![Percentage comparision][18] GeoImage -------- testImage[latitud_, longitud_, net_] := Module[{geoImage, geoImageResize}, geoImage = GeoImage[GeoPosition[{latitud, longitud}], GeoRange -> Quantity[75, "Meters"], GeoProjection -> "Mercator", ImageSize -> Small]; geoImageResize = ImageResize[geoImage, {500, 500}]; {ImageCompose[geoImageResize, {Image[net[geoImageResize]], 0.3}] , Image[net[geoImageResize]]}] **IER-UNAM** IER = testImage[18.839940, -99.235635, netEvaluation] ![IER-UNAM, Temixco, Morelos][19] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ImageCollage.jpeg&userId=1363133 [2]: https://project.inria.fr/aerialimagelabeling/ [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=A1.jpeg&userId=1363133 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=A2.jpeg&userId=1363133 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=C1.jpeg&userId=1363133 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=C2.jpeg&userId=1363133 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Association.jpeg&userId=1363133 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=mxFileTest.jpeg&userId=1363133 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=netModel.jpeg&userId=1363133 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=netModel500.jpeg&userId=1363133 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=finalNet.jpeg&userId=1363133 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=partGenerator.jpeg&userId=1363133 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=FinalLoss.jpeg&userId=1363133 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ErrorRate.jpeg&userId=1363133 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Properties.jpeg&userId=1363133 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Test1.jpeg&userId=1363133 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MaskTest.jpeg&userId=1363133 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Percentage.jpeg&userId=1363133 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=IER.jpeg&userId=1363133 Enrico Castro 2018-07-11T19:56:30Z https://community.wolfram.com/groups/-/m/t/1137429 ![Main picture][1] # Introduction The aim of this project is to train a Deep Reinforcement Learning agent to bring a smile to your face. The first part of the project was to train a Deep Convoluted Neural Network to predict one of five (happy, neutral, sad, angry, surprised) facial emotions. The input to this classifier is taken directly from the user's camera. Then, using this prediction as an input, a reinforcement learning agent was trained to make you smile. To do that a customizable emoticon was used. ## Why Laetitia? Laetitia is the Roman Goddess of joy, gaiety, and celebration, and is especially linked with holidays and festivals. # Phase one: facial emotion detector The first phase of the project was to implement the facial emotion detector. ## Dataset The very first part of this phase was dataset selection. I based my study on the Fe.r2013 database. It is free and available online on [Kaggle platform][2]. The reason standing behind my choice were small size of training example (pictures 48x48 pixels), huge number of labeled training O(20k) and test O(7k) examples. Each of database element belongs to one of following classes happy, neutral, sad, angry and surprised. Unfortunately, the database is contaminated by noised images or animated faces. It needed lot of afford to manually remove all the useless examples. ![training examples][3] ## Model training At the beginning of my study I trained classical Machine Learning models, such as Supported Vector Machine, Random Forest, k Nearest Neighborhood. These models were used as a performance baseline for further studies based on Deep Convoluted Neural Networks. My analysis shows that models that contain two hidden convoluted layers are the best classifiers. A selected model achieved a performance of 91% in happiness recognition, measured as area under Receiver Operating Characteristic curve. As mentioned before, the best facial emotion detector is a Deep Convolution Neural Network. Above cell presents architecture (upper right plot) of selected network and training progress related plots. ![CNN training][4] ## Convoluted Neural Network performance Classifier performance measurement was performed on O(7k) testing images. At first I present a confusion matrix which is a specific table layout that allows visualization of the performance of an algorithm. Each column of the matrix represents the instances in a predicted class while each row represents the instances in an actual class. The name stems from the fact that it makes it easy to see if the system is confusing two classes (i.e. commonly mislabelling one as another). ![cm][5] Another widely used classifier performance measuring metrics is the area under the Receiver Operating Characteristic curve, which can be interpreted as equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one. The obtained results for particular classes: ![rocs][6] ## Classifier evaluation The crucial project constraint is classifier evaluation time. The classifier need to be able to return response in real time. Here you find sample. The delays are caused by gif segmentation. ![sample][7] ## Transfer learning approach I also tried to use idea of transfer learning. Roughly speaking, we take pre-trained model such as [VGG16](http://www.robots.ox.ac.uk/~vgg/research/very_deep/) or [Inception](https://arxiv.org/abs/1512.00567) and then replace two last layers. emotionTypes = {"happy", "neutral", "sad", "angry", "surprise"} deepCNN = NetModel["Wolfram ImageIdentify Net for WL 11.1"] truncatedNetwork = Take[deepCNN, {1, -3}]; deepCNNInceptionV3 = NetInitialize @ NetChain[{truncatedNetwork, LinearLayer[Length[emotionTypes]], SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", emotionTypes}] ] Unfortunately, due to technical limitation of my notebook (no GPU unit) I was not able to finish training that kind of model. It probably will perform better than shallow version. This idea will be implemented in the future release of the project. # Phase Two: Reinforcement Learning In this phase, based on chosen model prediction, the policy to find the funniest emoticon was obtained. To do it the Monte Carlo based searching method was applied. At first I would like to present tunable emoticon that was used to make you smile. The animation was taken from [Wolfram Demonstration Project site](http://demonstrations.wolfram.com/SmileyChanger/). ![emoticon][8] This element will be replaced into auto generated pictures or jokes in the future release. ## Monte Carlo optimization Monte Carlo methods (or Monte Carlo experiments) are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. Their essential idea is using randomness to solve problems that might be deterministic in principle. From the point of view this work I am using numerical simulation to find the "funniest" emoticon configuration. Please take a look at the simulation sudo code. nFrames = 10; currentHappines = 0.; previousHappines = 0.; happinessesList = {{r, t , s, a , b, c , ey , er, currentHappines}}; ResetParameters[]; Dynamic[ parameter = SelectParameter[]; previousParamValue = KeepPreviousParameterValue[parameter]; update = CalculateParameterUpdate[parameter, step]; UpdateParmeter[parameter, update]; frame = 0; currentHappines = 0.; Column[{ animation, While[frame < nFrames, image = CurrentImage[]; greyFace = ExtractFaceFromImage[image]; currentHappines += PredictHappinessProbability[greyFace]; frame++; ]; currentHappines /= nFrames; image, greyFace, currentHappines, If[currentHappines > previousHappines , previousHappines = currentHappines, ReloadParameterValue[parameter, previousParamValue]]; happinessesList = Join [happinessesList, {{r, t , s, a , b, c , ey , er, currentHappines}}];} ] ] This simulation allows to obtain learning curve like plot and the from them funniest emoticon configuration. ![learningCurveMC][9] #Future goals From the future perspective the Q-Learning approach will be implemented. A starting point of the further studies will be Monte Carlo tuned policy mentioned before. The second idea is to replaced shallow version of CNN into pre-trained model. # Github Please take a look at the project source code on [github](https://github.com/adendek/DeepLaetitia). ####Stay tuned for further improvements! ####I’m happy about any kind of feedback. Please upvote if you like it ;-) [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Classifier.gif&userId=1075716 [2]: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge "Kaggle" [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=images.png&userId=1075716 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=trainingResult.png&userId=1075716 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=CM.png&userId=1075716 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Rocs.png&userId=1075716 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Reaction.gif&userId=1075716 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Emoticon.png&userId=1075716 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MonteCarlo.png&userId=1075716 Adam Dendek 2017-07-05T22:32:13Z https://community.wolfram.com/groups/-/m/t/1137055 ## Introduction **Instance Segmentation** is one of many interesting computer vision tasks that combines **object detection** and **semantic segmentation**. It detects the object, and at the same time, generates a segmentation mask, which you can think as classifying each pixel - whether it belongs to an object or not. Therefore, solving object detection and semantic segmentation together is a good approach to solve instance segmentation. In the summer school, we borrowed the framework of Mask R-CNN, to combine object detection and semantic segmentation in one pipeline, and produced some promising results. Mask R-CNN is developed like this: R-CNN -> Fast R-CNN -> Faster R-CNN -> Mask RCNN. In each stage, researchers solved some bottleneck problems to get faster and better performance. R stands for the region based, so R-CNN a region based convolutional neural network. Mask R-CNN has two stages, the first stage is trying to produce valid bounding box, you can think it as "blobby" image regions because "blobby" regions are likely to contain objects. In the early stage, researchers feed these warped image regions into a convolutional network, and in the output, they put two head, one regression head to produce the bounding-box, another head like SVM to do classify. And peoples kept working on it and make the network more efficient using some tricks like switch the proposal layer and convolutional layer to avoid unnecessary computations. ![enter image description here][1] There is a Github repo [FastMaskRCNN][2], several machine learning enthusiasts are trying to reproduce this paper in Tensorflow. I already obtained >400k epoch weights, but haven't tested yet, I will keep working on it after summer school. This Mask R-CNN model a too big to finish within 2 weeks, especially since I am new to the TensorFlow framework. Here is the graph visualization from TensorBoard, which looks really complicated and I haven't figure out the pipeline yet. ![enter image description here][3] ## My Project Because building Mask R-CNN network in Mathematica side turned out to be too complicated to finish before the deadline. So in the last two days of summer school, I did something simpler, but it's inspired by Mask R-CNN's framework, which uses the bounding-box region and corresponding mask to train a network to produce a binary mask (pixel-to-pixel). And this process is like semantic segmentation. In order to get this "pixel-to-pixel" trianing dataset, I wrote a script to process 24k [COCO][4] train2014 dataset and to crop the bounding box region based on the annotation `json` file. Here is how information encoded for object instance annotations: annotation{ "id" : int, "image_id" : int, "category_id" : int, "segmentation" : RLE or [polygon], "area" : float, "bbox" : [x,y,width,height], "iscrowd" : 0 or 1, } The original annotation `json` data looks like this: {"area" -> 54653., "segmentation" -> {{312.29, 562.89, 402.25, 511.49, 400.96, 425.38, 398.39, 372.69, 388.11, 332.85, 318.71, 325.14, 295.58, 305.86, 269.88, 314.86, 258.31, 337.99, 217.19, 321.29, 182.49, 343.13, 141.37, 348.27, 132.37, 358.55, 159.36, 377.83, 116.95, 421.53, 167.07, 499.92, 232.61, 560.32, 300.72, 571.89}}, "iscrowd" -> 0, "bbox" -> {116.95, 305.86, 285.3, 266.03}, "image_id" -> 480023, "category_id" -> 58, "id" -> 86} The code is very simple, we just need to do the following things: - Use the bounding-box `{{x, y}, {dw, dh}` to trim the image into bounding-box region image. - Use the list of points (vertices of the polygon (encoding ground truth segmentation mask) ) and bounding-box image dimensions to produce the corresponding mask for the region image. convertAnnotationBBoxIntoImageCoord[bbox_List, h_]:= Module[ {x1, y1, dw, dh}, {x1, y1, dw, dh} = bbox; Transpose[{0, h} + {1, -1} Transpose[Partition[{x1, y1, x1 + dw, y1 + dh}, 2]]] ] getMask[pts_List, w_, h_]:= Module[ {vtx}, vtx = convertAnnotationMask[pts, h]; Binarize @ Rasterize[Graphics[{White, Polygon@@vtx}, PlotRange->{{0, w}, {0, h}}, Background->Black],"Image", ImageSize->{w, h}] ] extractBBoxAndMask[img_, bbox_, segPts_List]:= Module[ {bboxLocal, w, h, boxRegion, maskRegion}, {w, h} = ImageDimensions[img]; bboxLocal = convertAnnotationBBoxIntoImageCoord[bbox, h]; boxRegion = ImageTrim[img, bboxLocal]; maskRegion = ImageTrim[getMask[segPts, w, h], bboxLocal]; {boxRegion, maskRegion} ] imgsAndMasks = extractBBox[imgs[[#]], bboxes[[#]], maskCoord[[#]]] & /@ Range[10] ![enter image description here][5] For the network I used to train, I used ENet, which is a very fast and an efficient network. The Mask R-CNN paper used FCN, which is known as standard network to perform semantic segmentation. I also construct this network in Mathematica and I will try it later as well. The ENet architecture is as followings: ![enter image description here][6] ![enter image description here][7] The output is a 256 * 256 * 2 tensor produced by a softmax layer, so it encoded the mask as the Pr[ this pixel belongs to object]. Therefore, I also need to convert the mask image from {0,1} binary into {1,2} as the class label and save it as `.dat` as training labels. Because I was running out of the time, I just trained with this simple input and output. A better way Etienne suggested is to extract the output from the final convolutional layer of yolo as an input feature to feed in near the output, which I will definitely try soon. I trained my network only for 9 hours on a single Tesla K80 GPU, and already got very promising results. The way my `instanceSegmentation[image, net, detectionThreshold, overlapThreshold]` work is as followings: 1. Use YOLO network as detector to produce labels, bounding-boxes, and probabilities 2. Use bounding-boxes to crop the image object region and feed it to our trained network 3. Take the output tensor, convert it to binaryImages, resize it back to bounding box dimensions by using `ImagePad`. padRegionMask[trimmedMask_, bboxLocal_, w_, h_]:= Block[ {x1,y1,x2,y2}, {{x1,y1},{x2,y2}} = bboxLocal; ImagePad[trimmedMask, {{x1, w - x2}, {y1, h - y2}}] ] instanceSegmentation[img_, ennet_, detectionThreshold_, overlapThreshold_]:= Module[ {labels, bboxes, probs, masks, coloredMasks, yoloVis, yoloRes, rectangles, centers}, yoloRes = detection[img, detectionThreshold, overlapThreshold]; rectangles = Transpose[yoloRes][[2]]; {labels, bboxes, probs} = convertYoloResult[yoloRes]; centers = Mean/@ bboxes; masks = produceMask[ennet, img, bboxes]; coloredMasks = Flatten[{RandomColor[], Opacity[.45], #}&/@ masks]; yoloVis = Transpose @ MapThread[ { {Darker @ Green,Opacity[0],#2}, Style[ Inset[#1<>" \n("<>ToString@Round[#3, .01]<>")", #4, {Center, Center} ], FontSize -> Scaled[.03], FontColor -> White, GrayLevel[0,1], Background -> GrayLevel[1,0] ] }&, {labels, rectangles, probs, centers}]; HighlightImage[img, Join[{coloredMasks,yoloVis}], ImagePadding -> Scaled[.02]] ] Ok, here are some results, I only started training today Many cute dogs ![dogs][8] Me and my mentor ![Me with my mentor ][9] Me and my phone ![Me with my phone][10] Me and my coffee ![Me with my coffee][11] Me and my handbag ![Me with my handbag][12] Me and my classmates and his phone ![Me with my classmates with his phone][13] ## Some personal reflection I enjoyed the summer school overall. I have been hoping to explore the Tensorflow framework and watch Stanford CS231n class for few months, but I was always very occupied with school's classes, other projects, lab's assignments, coding interviews, etc. I finally find some peaceful time to sit down and learn things I had always to learn. And in the process, I am also very amazed by the neural network framework Wolfram people developed. This is a very powerful and user-friendly framework that inherits Wolfram Language's elegant syntax and interactive property. I still have some questions about this framework and plan to learn more about it. ## Future Direction 1. Use FCN to do mask semantic segmentation 2. After obtained Mask R-CNN trained network, deploy it on a server and build an interesting iOS application. 3. Collaborate with Medical school people and apply Mask R-CNN to some medical imaging problems. ## Reference 1. Mask R-CNN Paper: https://arxiv.org/abs/1703.06870 Code (under testing): https://github.com/CharlesShang/FastMaskRCNN 2. ENet: https://arxiv.org/abs/1606.02147 3. YOLO2: https://arxiv.org/abs/1506.02640 4. Fully Convolutional Network (FCN): https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf 5. Project repo: https://github.com/zhuwenzhen/InstanceSegmentation [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Framework.png&userId=524853 [2]: https://github.com/CharlesShang/FastMaskRCNN [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=png.png&userId=524853 [4]: http://mscoco.org/dataset/#download [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0515.14.08.png&userId=524853 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Enet.png&userId=524853 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0515.22.14.png&userId=524853 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.02.58.png&userId=524853 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.20.56.png&userId=524853 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.20.47.png&userId=524853 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.21.35.png&userId=524853 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.25.30.png&userId=524853 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.25.38.png&userId=524853 Wenzhen Zhu 2017-07-05T21:17:23Z https://community.wolfram.com/groups/-/m/t/1137044 The following is a short summary of a project I did at the Wolfram Summer School, 2017. In this project we aim to convert any given mathematical expression (printed or or even handwritten) into LaTeX syntax. We implement the algorithm in the Wolfram Language using the built-in Neural Network functionality. We "loosely" follow the algorithm proposed in the paper https://arxiv.org/pdf/1609.04938v1.pdf . The dataset is imported from https://zenodo.org/record/56198#.WVzy-caZORt. It includes a total of approximately 100k formulas and images split into train, validation and test sets. The file "im2latex_formulas.lst" contains 103558 latex formulae separated by "\n" which we import in Mathematica formulae = StringSplit[Import["/Users/Himanshu/Desktop/Wolfram Assignments/Wolfram \Project/im2latex_formulas.lst", "String"], "\n"]; Using just `Import["/Users/Himanshu/Desktop/Project/im2latex_formulas.lst", "String"]` leads to incorrect number of lines 104563. We then process this data. First we strip off unnecessary elements in the LaTeX formulae ( like \label{eqn} ) and white-spaces as follows: formulae = StringSplit[ Import["/Users/Himanshu/Desktop/Wolfram Assignments/Wolfram \ Project/im2latex_formulas.lst", "String"], "\n"]; labels = StringTrim[ StringReplace[ StringReplace[#, {"\\label{" ~~ ShortestMatch[___] ~~ "}" :> "", "\t" | "\\," | WhitespaceCharacter | "\\:" | "\\;" -> "~"}], "~" .. -> "~"], "~"] & /@ formulae; The formulae, stored in the folder "formula_images", are in png format and rendered as latex expressions on a transparent background. Although they occupy a small size on the disk, while importing in the Mathematica kernel the size of the cropped images increases. fileDataset[s_String, import_?BooleanQ, folder_String: ""] := Dataset[<|"Input" -> Last@#, "Target" -> labels[[First@# + 1]]|> & /@ ({ToExpression@#[[1]], If[import, Import, Identity]@ File[folder <> #[[2]] <> ".png"]} &~ParallelMap~ StringSplit[ReadList[s, String]][[;; , {1, 2}]])] The following code generates the training, test and validation dataset. WARNING: Time consuming and requires at least 20 GB of disk space. trainDataset = fileDataset["im2latex_train.lst", False, "formula_images/"]; // AbsoluteTiming testDataset = fileDataset["im2latex_test.lst", False, "formula_images/"]; // AbsoluteTiming validateDataset = fileDataset["im2latex_validate.lst", False, "formula_images/"]; // AbsoluteTiming We also pad the formula images in order to get images of the same size. This is to avoid distortion. The final training / validation / test data is `Dataset` ![enter image description here][1] which is nothing but a list of rules of the following form ![enter image description here][2] The length of the training data is 83884, test data is 10355, validation data is 9320. After processing, they occupy a disk size of approximately 20 GB. Since the Dataset generated is huge in size it is not optimal for uploading on the GPU machine. Therefore instead of creating, saving and then uploading the dataset we generate the dataset from the raw data and save it in a temporary variable in memory which is fed into the network for training. Next we turn to the neural network architecture. The neural network architecture is divided in to three stages. 1. **Convolutional Network**: The visual features of an image are extracted with a multi-layer convolutional neural network (CNN) interleaved with max-pooling layers. The CNN takes the raw input and produces a feature grid $\tilde{V}$ of size $D \times H \times W $ where D denotes the number of channels and H and W are the resulted feature map height and width. 2. **Row Encoder**: The feature grid $\tilde{V}$ produced by the CNN is fed into a Row Encoder that localizes its input by running a Recurrent Neural Network (RNN) over each of the rows of CNN feature grid $\tilde{V}$ and produces a new feature grid V. For OCR, it is important for the encoder to localize the relative positions within the source image. 3. **Decoder**: The target markup tokens {$y_t$} are then generated by a decoder based on the row encoded feature grid V. The decoder (equipped with an attention mechanism) is trained to calculate the conditional probability of a token $y_{t+1}$ appearing at position $t+1$ given the sequence ${y_0,y_1,...y_t }$ and the feature grid V. However in my implementation of the network I have not incorporated the attention mechanism. The final network that was used for training is schematically shown below ![enter image description here][3] where the encoder is a `NetChain` that consists of CNN interleaved with Pooling and Batch Normalization layers and decoder is a `NetGraph` which is schematically shown below ![enter image description here][4] In this work we have been able to implement the first two stages of the algorithm and a slightly modified version of the third stage that cuts away the attention mechanism. We have trained the network on a 12 GB NVIDIA GPU. Training batch size is set to 16 due to the size limit of GPU memory. After 7 rounds the loss drops to 1.48. It is expected that on adding attention layer in the network, the loss should improve. For original and previous work in this direction please refer to the following links - https://github.com/harvardnlp/im2markup - https://github.com/ritheshkumar95/im2latex-tensorflow ## Acknowledgements ## I would like to thank Giulio Alessandrini, Matteo Salvarezza and Daniel George for useful discussions and helping me out in various stages of the project. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at3.47.49PM.png&userId=1123238 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at3.48.08PM.png&userId=1123238 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at4.45.12PM.png&userId=1123238 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at4.52.36PM.png&userId=1123238 Himanshu Raj 2017-07-05T21:16:09Z https://community.wolfram.com/groups/-/m/t/1136941 # Abstract The problem of scene parsing is a difficult one. Despite this complexity, the ability to correctly analyse the environment is critical for the development of autonomous systems such as self-driving cars. Here I describe the procedure to develop a model for semantically segmenting urban street scenes. The aim of my project is to implement a deep neural network in the Wolfram Language which can perform a pixel-wise semantic segmentation of the images contained in the [Cityscapes dataset](https://www.cityscapes-dataset.com/). The dataset used is a challenging one consisting of a comprehensive set of images from various kinds of urban settings. ### Note: I have tried to write this post for 2 kinds of readers - those who want a high level overview of what this project does, and those who want to implement a similar model on their own and are looking for guidelines. To cater to both these kinds of readers, I have included as little code in this post as possible, but have referenced the relevant files on my [GitHub repo](https://github.com/saxenarohan97/WolframResearch) as and when required. So if you want to see how I did any of the things I've shown here, just scroll around and you will find a link to the relevant file on GitHub. # Project ## Neural Netwok Design Overview Here let's set some ground rules for the design of our network. It doesn't have to be very detailed, but we must know what we are doing since the preprocessing of our data will depend upon the network design. On the [Cityscapes benchmarks](https://www.cityscapes-dataset.com/benchmarks/#scene-labeling-task) page, there is a list of papers which have achieved a high accuracy on this dataset. As of this writing, the paper with the highest accuracy (with accompanying code) is [PSPNet](https://hszhao.github.io/projects/pspnet/). It uses the 101-layer pretrained [ResNet](https://arxiv.org/abs/1512.03385) network, appended with a custom pyramid pooling scheme. ## Data Preparation We must first prepare the data to be fed into the neural network of our design. We will be passing in images of urban scenes into the network. However, they are too large (2048 x 1024 pixels) to be processed easily. PSPNet crops the images to ~720 pixels for this reason. Since we will be working under limited time and a modest GPU, we shall crop the images to 256 x 256. There is a problem with this, however. In an image of 2048 x 1024, a crop of 256 x 256 classes will contain a very small subset of the image, which will in turn contain very few of the classes we want to classify. So our network will not be able to see many classes at once, in the same image. To address this issue, let's resize the images to 1/4 of the original resolution (that is, to 1024 x 512 pixels). ![enter image description here][1] The cropped images look like this: ![enter image description here][2] We will feed in the `ImageData` of the input images and get the output mask as an array (which will be the `ImageData` of the output mask). The masks are also resized and cropped in the same way. The code for resizing and cropping the input images and masks is in the file `GetData.nb` on my [GitHub repo](https://github.com/saxenarohan97/WolframResearch). Since we will be using the pretrained weights of ResNet, we must also include the preprocessing that ResNet uses. It subtracts the mean of all images in its dataset from each image before feeding the image into the network. We must extract this mean image, as shown below. ## Neural Network Design Details This is what the entire ResNet looks like in Mathematica: ![enter image description here][3] Phew, that's deep! By the way, ```meanImg = Normal[NetExtract[resnet, "Input"]]["MeanImage"]``` gives us the mean image of ResNet, which we'll subtract from each input image before sending it into the network (as I'd said above). As the PSPNet paper says, I will chop off the last few layers from layer 5a onwards. Then, I will add a pyramid pooling module which is a series of pooling layers of various sizes. The PSPNet authors have upsampled the images a bit too much for my taste (this could be because they train on the coarse annotations of the Cityscapes dataset). Since we want to train on the fine annotations, I replaced some of the upsampling by deconvolutions (which is basically a learned upsampling). The entire design code of the neural network is in the file `network.nb` on my [GitHub repo](https://github.com/saxenarohan97/WolframResearch). ![enter image description here][4] The final network isn't much to look at because the entire ResNet, the pyramid pooling and other layers have each been compressed to a single line (a `NetGraph` or a `NetChain`). ## Train the network Now let's begin the most fun part of the project - actually training the neural network. For this, I am using an AWS instance with the NVIDIA Tesla K80 GPU. I uploaded all my files onto the remote machine and have written a script to train the network. This script can be found as (surprise!) `script.wl` on my [GitHub repo](https://github.com/saxenarohan97/WolframResearch). A convenient way I like to view my training (on a MacBook) is as such: - Open 2 terminal windows and log into the AWS instance on both - Set them up in OS X Split View - Run the training script in a `tmux` session on one, and a `watch nvidia-smi` on the other - This shows us the training progress on one side, and the GPU memory usage on the other. Remember that a batch size must be chosen such that it is the largest number of files which can comfortably fit on the GPU/CPU. Here is a screenshot of the network training: ![enter image description here][5] We can see the training loss, time spent, etc on the left, and GPU information on the right. Here is the output of `watch nvidia-smi` a bit more clearly: ![enter image description here][6] When the training error is reasonable or when you are bored out of your mind waiting for the network to finish training, download the network to your local machine and test it on images! ## Test Performance For my test file, see `test performance.nb` on [GitHub](https://github.com/saxenarohan97/WolframResearch). I have organised the paths on my machine such that I can randomly pick an image to visualise it with this code: i = RandomInteger[{1, Length @ testPics}]; input = testPics[[i]]; pixelResult = pixelPredict[net, input]; ImageCollage[ { ArrayPlot[pixelResult, ColorRules -> colorRules], fip @ trialFiles[[i]], ArrayPlot[importer @ coarseMaskFiles[[i]], ColorRules -> colorRules], ArrayPlot[importer @ maskFiles[[i]], ColorRules -> colorRules] } ] This is the output I get from the above piece of code: ![enter image description here][7] In clockwise direction starting from top left, these images are: - Output of my neural network - Input image - Fine annotation of the Cityscapes dataset - Coarse annotation of the Cityscapes dataset As we can see, the network output is better than the coarse annotation, but not as good as the fine annotation. Moreover, it has some difficulty detecting small or thin objects such as poles, signs, etc. I suspect this is because of the large amount of upsampling in the network. If we add more deconvolutions instead, I am confident that the accuracy will further increase. Let's see the incorrectly classified pixels in white: ![enter image description here][8] Yup, as I said, small / thin objects are not so easily spotted by this guy. If we measure the accuracy for this image (percentage of correctly classified pixels): (1. - Mean @ Flatten[ Unitize[pixelResult - testMasks[[i]]] ]) * 100 We get `83.6151`. Hey, not too bad for a day's work! In my [GitHub repo](https://github.com/saxenarohan97/WolframResearch), I have included the code to test the network on the entire test set (3,800 images). But beware! Running it on a CPU takes forever, it is much better to run it on a GPU. # References: - Zhao, Hengshuang, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. "Pyramid Scene Parsing Network." [1612.01105] Pyramid Scene Parsing Network. CVPR (IEEE), 27 Apr. 2017. Web. 25 June 2017. - He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep Residual Learning for Image Recognition." [1512.03385] Deep Residual Learning for Image Recognition. CVPR, 10 Dec. 2015. Web. 25 June 2017. - Cordts, Marius, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. "The Cityscapes Dataset for Semantic Urban Scene Understanding." [1604.01685] The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR, 07 Apr. 2016. Web. 25 June 2017. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=resize.jpg&userId=1098614 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=parts.png&userId=1098614 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=net.png&userId=1098614 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=myNet.png&userId=1098614 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Training.jpeg&userId=1098614 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=nvidia-smi.jpeg&userId=1098614 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=9248output.png&userId=1098614 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=incorrect.png&userId=1098614 Rohan Saxena 2017-07-05T21:12:46Z https://community.wolfram.com/groups/-/m/t/1136102 ![Overall steps of the process][1] # Abstract # **Goal of the project:** To output the value of a resistor given a photo of it. The photos are intended to be taken in a white background, like from a white paper sheet and with a cellphone camera. **Summary of work:** Many samples are used in a Classifier, leaving as result a logistic regression, all training and test samples were rotated and cropped automatically. The resistor position is found using morphological components, then the picture is cropped and rotated leaving the resistor to cover the majority of the picture, using the Classifier we obtain the final result. **Results and future work:** The project is limited to a small set of resistors to which it can calculate the value. The next step is to either to crop the color bands of the resistor, and use machine learning in a similar way to determine the color of each of the bands of the resistor, or improve the current machine learning methods to raise the extent of the project. # First approach # For the first attempt to solve this problem the method suggested in the Lecture Notes in Computer Science where a similar problem is being solved. The difference was that the resistor photos were taken in an environment where the light is controlled, reducing the glare and shadows. The first step was to obtain the background color of the image, this was originally done calculating the mean of the pixels near the edges of the picture, in this case it was done obtaining a mask to identify the background and calculating the mean of all the pixels in it. img = ![initial image][2] ``` img = ImageAdjust[ImageResize[img, {500}]]; img = ColorConvert[RemoveAlphaChannel[img],"RGB"]; ``` ``` meanColorMask[image_,mask_ ]:= Module[ {toRemove,imgList}, toRemove = PixelValuePositions[mask, Black]; imgList = Transpose @ ImageData[image]; imgList = Delete[imgList, toRemove]; RGBColor[Mean[Catenate @ imgList]] ] ``` ``` getBackColor[image_] := meanColorMask[image, ColorNegate @ AlphaChannel[RemoveBackground[image]]]; backColor = getBackColor[img] ``` ![backcolor][3] This color is extracted from the image, then the result of that is converted to grayscale. ``` absoluteSubstract[image_, color_] := ImageApply[ MapThread[ Abs[#1 - #2]&, {Flatten[#],Flatten[List @@ color]} ]&, image ] substracted = absoluteSubstract[img, backColor] monochromatic = ColorConvert[substracted,"Grayscale"] ``` ![enter image description here][4] ![enter image description here][5] From here we calculate a threshold we are going to use to separate the resistor from the background. A threshold that maximizes the variance between the rows and columns lower and higher than the threshold. Then crop the image based on the values that were higher than it. This method is further explained in \[1] and \[2]. ``` threshold[n_] := Module[ {num, p, \[Omega], \[Mu], \[Mu]T, maxK, max\[Sigma], \[Sigma]2B, temp, k}, num = Total[n]; p = n / num; \[Omega] := Sum[p[i],{i,0,#}]&; \[Mu] := Sum[(i+1) * p[i], {i, 0, #}]&; \[Mu]T = \[Mu][255]; \[Sigma]2B[x_Integer] := Module[ {\[Mu]k, \[Omega]k}, \[Omega]k = \[Omega][x]; \[Mu]k = \[Mu][x]; If[ \[Omega]k != 0 && \[Omega]k != 1, ((\[Mu]T*\[Omega]k - \[Mu]k)^2) / (\[Omega]k * (1 - \[Omega]k)), -1 ] ]; max\[Sigma] = 0; maxK = 255; For[k = 0, k < 255, k++, temp = \[Sigma]2B[k]; If[ NumberQ[temp] && temp > max\[Sigma], max\[Sigma] = temp; maxK = k; ] ]; maxK ] ``` ``` matr = ImageData[monochromatic]; verMean = Map[ Mean[Flatten[#]]&, matr ]; verMean = Round[verMean * 255]; horMean = Map[ Mean[Flatten[#]]&, Transpose[matr] ]; horMean = Round[horMean * 255]; horFrequency = Counts[Catenate[{horMean,Range[0,255]}]] - 1; verFrequency = Counts[Catenate[{verMean,Range[0,255]}]] - 1; verThreshold = threshold[verFrequency]; horThreshold = threshold[horFrequency]; verPlot = DiscretePlot[ verMean[[x]], {x, 1, Length[verMean]}, PlotRange->All, ImageSize->Medium ]; verThresholdPlot = Plot[ verThreshold, {x, 1, Length[verMean]}, PlotRange->All, ImageSize->Medium, PlotStyle->{Red} ]; horPlot = DiscretePlot[ horMean[[x]], {x, 1, Length[horMean]}, PlotRange->All, ImageSize->Medium ]; horThresholdPlot = Plot[ horThreshold, {x, 1, Length[horMean]}, PlotRange->All, ImageSize->Medium, PlotStyle->{Red} ]; xPos = Values @ Cases[ MapIndexed[#1->First[#2]&, horMean], (x_->y_)/;x>= horThreshold ]; xStart = Min[xPos]; xEnd = Max[xPos]; yPos = Values @ Cases[ MapIndexed[#1->First[#2]&, verMean], (x_->y_)/;x>= verThreshold ]; yStart = Min[yPos]; yEnd = Max[yPos]; segmented = ImageTake[ img, {yStart, yEnd},{xStart, xEnd} ]; xStartLine = Graphics @ Line[{{xStart, 0},{xStart, 255}}]; xEndLine = Graphics @ Line[{{xEnd, 0},{xEnd, 255}}]; yStartLine = Graphics @ Line[{{yStart, 0},{yStart, 255}}]; yEndLine = Graphics @ Line[{{yEnd, 0},{yEnd, 255}}]; Grid[{ { Show[{horPlot, horThresholdPlot, xStartLine, xEndLine}], Show[{verPlot, verThresholdPlot, yStartLine, yEndLine}] }, {"Horizontal mean", "Vertical mean"} }] segmented ``` ![enter image description here][6] ![enter image description here][7] The graphs represent the average color of each of the columns and rows of the pixels of the monochromatic image. The threshold for each of them is represented with the red line and is calculated using the Otsu's method. The black lines represent the left-most, right-most columns, and top and bottom rows where the mean color is over the threshold. With those values we can cut the resistor of the image. Finally, in this approach we extract an image with a height of 15 pixels. ``` extracted = ImageCrop[segmented, {ImageDimensions[segmented][[1]], 15}, Center] ``` ![enter image description here][8] To get the color bands, the image is subtracted of the back color and made grayscale, then clustering the bars with the K-means algorithm. However, this method did not result convenient for our use case, as kept marking the glare in the resistor as part of the band. ``` extracted = ImageResize[extracted, {500}]; exBackColor = ColorConvert[ Hue @@ ImageMeasurements[ColorConvert[extracted, "HSB"], "Mean"], "RGB" ] exSubs = TotalVariationFilter @ absoluteSubstract[ extracted, ColorConvert[exBackColor,"RGB"] ] exMono = ColorConvert[exSubs, "Grayscale"] exBinary = Binarize[exMono,Method->"Cluster"] allX = Map[ Interval[{Ramp[Min[#]],Ramp[Max[#]]}]&, Values[ComponentMeasurements[ MorphologicalComponents[exBinary], "MinimalBoundingBox" ]][[All, All, 1]] ]; allX = List @@ (IntervalUnion @@ allX); squaresX = Polygon @ Map[ Module[ {min, max, height}, min = #[[1]]; max = #[[2]]; height = ImageDimensions[extracted][[2]]; {{min, 1}, {min, height}, {max, height}, {max, 1}} ]&, allX ]; HighlightImage[extracted, {Red, squaresX}] ``` ![enter image description here][9] ![enter image description here][10] ![enter image description here][11] ![enter image description here][12] ![enter image description here][13] # Second approach # The next approach was to use machine learning to obtain the values of the resistors. Several samples were made taking pictures from resistors on a white background, making sure they had different positions and angles. However, the training was taking too long and causing the Mathematica program to crash. Later it was concluded that the results would improve if the resistor in the photo was cropped, removing several pixels without any information the total memory needed to process the images would be lower. For cropping the resistor from the photo, the following was made: * Pass the photo through ImageAdjust and BrightnessEqualize rise the contrast and remove shadows from the image. * Used the function RemoveBackground to obtain from the alpha channel a mask that would separate the resistor from the background. * Erode said mask, so the wires in the resistor are eliminated from the mask. * Use RegionBinarize to recover the parts of the resistor that were removed from the mask. * Obtain the minimal bounding box and orientation angle from this resistor mask. * Rotate and crop the image according with the previous values. ``` resistorCrop[image_] := Module[ {preprocess, angle, rotated, newPoints, allX, allY, minX, minY, maxX, maxY, processed, adjusted}, adjusted = ImageAdjust @ BrightnessEqualize[image]; preprocess = SelectComponents[Erosion[AlphaChannel[RemoveBackground[adjusted]],10],"Count",-1]; preprocess = RegionBinarize[adjusted,preprocess,0.15]; processed = MorphologicalComponents[preprocess]; angle = First @ Values[ ComponentMeasurements[processed, "Orientation"] ]; rotated = ImageRotate[image, -angle, Masking->Full]; newPoints = First @ Values[ComponentMeasurements[processed, "MinimalBoundingBox"]]; newPoints = RotationTransform[-angle,ImageDimensions[image]/2][newPoints]; newPoints = TranslationTransform[(ImageDimensions[rotated] - ImageDimensions[image])/2][newPoints]; allX = newPoints[[All, 1]]; allY = newPoints[[All, 2]]; minX = Min[allX]; minY = Min[allY]; maxX = Max[allX]; maxY = Max[allY]; ImageTake[rotated, Ramp[ImageDimensions[rotated][[2]]-{maxY, minY}],{minX,maxX}] ] ``` From this function the samples would be cropped to allow the training to run faster. Some images were not cropped correctly and had to be removed. This method can not be used for the final product though, as it takes too much time to filter the images for cropping. One way to make a faster method is to rescale the initial image smaller before obtaining the mask and at the end rescale the cropping coordinates accordingly, reducing the total pixels the program would work on. Naturally this method is less precise. ``` resistorLightCrop[image_] := Module[ {preprocess, angle, rotated, newPoints, allX, allY, minX, minY, maxX, maxY, processed, adjusted, resized, prop, fullRotated}, resized = ImageResize[image, {500}]; prop = ImageDimensions[image][[1]] / ImageDimensions[resized][[1]]; adjusted = ImageAdjust @ BrightnessEqualize[resized]; preprocess = SelectComponents[Erosion[AlphaChannel[RemoveBackground[adjusted]],10 / prop],"Count",-1]; preprocess = RegionBinarize[adjusted,preprocess,0.15]; processed = MorphologicalComponents[preprocess]; angle = First @ Values[ ComponentMeasurements[processed, "Orientation"] ]; rotated = ImageRotate[resized, -angle, Masking->Full]; newPoints = First @ Values[ComponentMeasurements[processed, "MinimalBoundingBox"]]; newPoints = RotationTransform[-angle,ImageDimensions[resized]/2][newPoints]; newPoints = TranslationTransform[(ImageDimensions[rotated] - ImageDimensions[resized])/2][newPoints]; allX = newPoints[[All, 1]]; allY = newPoints[[All, 2]]; minX = Min[allX]; minY = Min[allY]; maxX = Max[allX]; maxY = Max[allY]; fullRotated = ImageRotate[image, -angle, Masking->Full]; ImageTake[fullRotated, Ramp[ImageDimensions[fullRotated][[2]]-{maxY * prop, minY * prop}],{minX * prop,maxX * prop}] ] SetDirectory[NotebookDirectory[]]; DumpSave["resistorLightCrop.mx", resistorLightCrop]; ResetDirectory[]; ``` Once the images of the dataset are loaded we need to create the training set and the test set. As the cropping method used in the training is more accurate than the one used for the final product, adding noise to the samples is needed to improve the results. This is because the method used has shown to fail sometimes in rotating the resistor in a horizontal position, especially if the wires in the resistor are bended, so more samples were the resistor is rotated to random angles were added to make the Classifier be prepared if the cropping function fails to make the rotation. Also, blurry samples using a GaussianFilter were added, to compensate loss of focus in that can be done in the photo. As for the Classifier method, a simple Classify function with a quality performance goal is used because it gave better results than forcing it to be a neural network or using a Predict function. ``` newBlurry[data_, n_] := Module[ {toBlur}, toBlur = RandomSample[data, n]; toBlur/.(x_->y_):>GaussianFilter[x,15]->y ] ``` ``` newRotated[data_, n_] := Module[ {toRotate}, toRotate = RandomSample[data, n]; toRotate/.(x_->y_):>ImageRotate[x,RandomReal[{0, \[Pi]}], Masking->Full]->y ] ``` ``` newBlurry[trainingSet, 1] ``` ![enter image description here][14] ``` newRotated[trainingSet, 1] ``` ![enter image description here][15] This improved not only the accuracy for the given samples, but also the results on the final product. ``` cNoParameters = Classify[trainingSet, PerformanceGoal->"Quality"] ``` ![enter image description here][16] ``` ClassifierInformation[cNoParameters] ``` ![enter image description here][17] ``` cNoParametersM = ClassifierMeasurements[cNoParameters, testSet] ``` ![enter image description here][18] ``` cNoParametersM["ConfusionMatrixPlot"] ``` ![enter image description here][19] Trying other methods did not give better results ``` cNeuralNet = Classify[trainingSet, Method->"NeuralNetwork", PerformanceGoal->"Quality", FeatureExtractor->"ImageFeatures"]; cNeuralNetM = ClassifierMeasurements[cNeuralNet, testSet] ``` ![enter image description here][20] ``` cNeuralNetM["ConfusionMatrixPlot"] ``` ![enter image description here][21] # Microsite link # https://wolfr.am/mUtXyBEe # Code # https://github.com/carolinadp/WSS2017 # References # \[1] Mitani Y., Sugimura Y., Hamamoto Y. (2008) A Method for Reading a Resistor by Image Processing Techniques. In: Lovrek I., Howlett R.J., Jain L.C. (eds) Knowledge-Based Intelligent Information and Engineering Systems. KES 2008. Lecture Notes in Computer Science, vol 5177. Springer, Berlin, Heidelberg \[2] Otsu N. A Threshold Selection Method from Gray-Level Histograms. IEEE Transactions on Systems, Man and Cybernetics. 1979 [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=teaserImage.png&userId=1092998 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img1.png&userId=1092998 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img3.png&userId=1092998 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img4.png&userId=1092998 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img5.png&userId=1092998 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img6.png&userId=1092998 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img7.png&userId=1092998 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img8.png&userId=1092998 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img9.png&userId=1092998 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img10.png&userId=1092998 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img11.png&userId=1092998 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img12.png&userId=1092998 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img13.png&userId=1092998 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img14.png&userId=1092998 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img15.png&userId=1092998 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at4.54.44PM.png&userId=1092998 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at4.56.40PM.png&userId=1092998 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at5.00.28PM.png&userId=1092998 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img16.png&userId=1092998 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at5.08.42PM.png&userId=1092998 [21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img17.png&userId=1092998 Itzel Delgadillo Perez 2017-07-05T19:14:05Z