https://community.wolfram.com RSS Feed for Wolfram Community showing any discussions tagged with Image Processing with no replies sorted by most likes. https://community.wolfram.com/groups/-/m/t/1893009 According to the [official announcement][1]: > For the first time in its 174-year history, the Smithsonian has released 2.8 million high-resolution two- and three-dimensional images from across its collections onto an open access online platform for patrons to peruse and download free of charge. Featuring data and material from all 19 Smithsonian museums, nine research centers, libraries, archives and the National Zoo, the new digital depot encourages the public to not just view its contents, but use, reuse and transform them into just about anything they choose—be it a postcard, a beer koozie or a pair of bootie shorts. *COLLECTION is HERE*: https://www.si.edu/openaccess There are a lot of interesting and images and 3D models from very diverse corners of life, - here are a few samples: ![enter image description here][2] One of coolest thing is that some of 3D models are given in parts that can be 3D printed separately and then assembled together, thus making cool DIY project. Here is an example you can try with a 3D model of [Mammuthus primigenius / Blumbach][3]. Below the model find and copy the download link: ![enter image description here][4] Import data from the collection (I shortened the URL with `URLShorten`): data = Import["https://wolfr.am/KQ6NWYff", "*"] ![enter image description here][5] You can see these are the pieces that should make up a whole object: Show[data] ![enter image description here][6] It is easy to make 3D printouts with the Wolfram Language -- the parts Printout3D /@ data or whole thing: Printout3D[RegionUnion @@ data] ![enter image description here][7] It is a really neat toy and reminds me of 3D puzzles: you get the parts and have to figure out what the whole is by building it. Could be a fun time with kids :-) If you can get your hands on 3D printer it might be better than a toy store. There are very different types too. Here is, for instance for space-dreamers, the [Apollo 11 Command Module][8] -- apart and together. apollo = Import["https://wolfr.am/KQLE0LOx", "*.obj"] ![enter image description here][9] Show[apollo] ![enter image description here][10] [1]: https://www.smithsonianmag.com/smithsonian-institution/smithsonian-releases-28-million-images-public-domain-180974263/ [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-03-06at9.47.27AM.png&userId=11733 [3]: https://3d.si.edu/object/3d/mammuthus-primigenius-blumbach:341c96cd-f967-4540-8ed1-d3fc56d31f12 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-03-06at10.37.24AM.png&userId=11733 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=esgasd4q3Untitled.png&userId=11733 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=fdsfsdfsdfs.png&userId=11733 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=gfxgfdtey35Untitled.png&userId=11733 [8]: https://3d.si.edu/object/3d/command-module-apollo-11:d8c6457e-4ebc-11ea-b77f-2e728ce88125 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-03-06at10.14.41AM.png&userId=11733 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-03-06at10.25.19AM.png&userId=11733 Vitaliy Kaurov 2020-03-06T05:56:34Z https://community.wolfram.com/groups/-/m/t/1137429 ![Main picture][1] # Introduction The aim of this project is to train a Deep Reinforcement Learning agent to bring a smile to your face. The first part of the project was to train a Deep Convoluted Neural Network to predict one of five (happy, neutral, sad, angry, surprised) facial emotions. The input to this classifier is taken directly from the user's camera. Then, using this prediction as an input, a reinforcement learning agent was trained to make you smile. To do that a customizable emoticon was used. ## Why Laetitia? Laetitia is the Roman Goddess of joy, gaiety, and celebration, and is especially linked with holidays and festivals. # Phase one: facial emotion detector The first phase of the project was to implement the facial emotion detector. ## Dataset The very first part of this phase was dataset selection. I based my study on the Fe.r2013 database. It is free and available online on [Kaggle platform][2]. The reason standing behind my choice were small size of training example (pictures 48x48 pixels), huge number of labeled training O(20k) and test O(7k) examples. Each of database element belongs to one of following classes happy, neutral, sad, angry and surprised. Unfortunately, the database is contaminated by noised images or animated faces. It needed lot of afford to manually remove all the useless examples. ![training examples][3] ## Model training At the beginning of my study I trained classical Machine Learning models, such as Supported Vector Machine, Random Forest, k Nearest Neighborhood. These models were used as a performance baseline for further studies based on Deep Convoluted Neural Networks. My analysis shows that models that contain two hidden convoluted layers are the best classifiers. A selected model achieved a performance of 91% in happiness recognition, measured as area under Receiver Operating Characteristic curve. As mentioned before, the best facial emotion detector is a Deep Convolution Neural Network. Above cell presents architecture (upper right plot) of selected network and training progress related plots. ![CNN training][4] ## Convoluted Neural Network performance Classifier performance measurement was performed on O(7k) testing images. At first I present a confusion matrix which is a specific table layout that allows visualization of the performance of an algorithm. Each column of the matrix represents the instances in a predicted class while each row represents the instances in an actual class. The name stems from the fact that it makes it easy to see if the system is confusing two classes (i.e. commonly mislabelling one as another). ![cm][5] Another widely used classifier performance measuring metrics is the area under the Receiver Operating Characteristic curve, which can be interpreted as equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one. The obtained results for particular classes: ![rocs][6] ## Classifier evaluation The crucial project constraint is classifier evaluation time. The classifier need to be able to return response in real time. Here you find sample. The delays are caused by gif segmentation. ![sample][7] ## Transfer learning approach I also tried to use idea of transfer learning. Roughly speaking, we take pre-trained model such as [VGG16](http://www.robots.ox.ac.uk/~vgg/research/very_deep/) or [Inception](https://arxiv.org/abs/1512.00567) and then replace two last layers. emotionTypes = {"happy", "neutral", "sad", "angry", "surprise"} deepCNN = NetModel["Wolfram ImageIdentify Net for WL 11.1"] truncatedNetwork = Take[deepCNN, {1, -3}]; deepCNNInceptionV3 = NetInitialize @ NetChain[{truncatedNetwork, LinearLayer[Length[emotionTypes]], SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", emotionTypes}] ] Unfortunately, due to technical limitation of my notebook (no GPU unit) I was not able to finish training that kind of model. It probably will perform better than shallow version. This idea will be implemented in the future release of the project. # Phase Two: Reinforcement Learning In this phase, based on chosen model prediction, the policy to find the funniest emoticon was obtained. To do it the Monte Carlo based searching method was applied. At first I would like to present tunable emoticon that was used to make you smile. The animation was taken from [Wolfram Demonstration Project site](http://demonstrations.wolfram.com/SmileyChanger/). ![emoticon][8] This element will be replaced into auto generated pictures or jokes in the future release. ## Monte Carlo optimization Monte Carlo methods (or Monte Carlo experiments) are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. Their essential idea is using randomness to solve problems that might be deterministic in principle. From the point of view this work I am using numerical simulation to find the "funniest" emoticon configuration. Please take a look at the simulation sudo code. nFrames = 10; currentHappines = 0.; previousHappines = 0.; happinessesList = {{r, t , s, a , b, c , ey , er, currentHappines}}; ResetParameters[]; Dynamic[ parameter = SelectParameter[]; previousParamValue = KeepPreviousParameterValue[parameter]; update = CalculateParameterUpdate[parameter, step]; UpdateParmeter[parameter, update]; frame = 0; currentHappines = 0.; Column[{ animation, While[frame < nFrames, image = CurrentImage[]; greyFace = ExtractFaceFromImage[image]; currentHappines += PredictHappinessProbability[greyFace]; frame++; ]; currentHappines /= nFrames; image, greyFace, currentHappines, If[currentHappines > previousHappines , previousHappines = currentHappines, ReloadParameterValue[parameter, previousParamValue]]; happinessesList = Join [happinessesList, {{r, t , s, a , b, c , ey , er, currentHappines}}];} ] ] This simulation allows to obtain learning curve like plot and the from them funniest emoticon configuration. ![learningCurveMC][9] #Future goals From the future perspective the Q-Learning approach will be implemented. A starting point of the further studies will be Monte Carlo tuned policy mentioned before. The second idea is to replaced shallow version of CNN into pre-trained model. # Github Please take a look at the project source code on [github](https://github.com/adendek/DeepLaetitia). ####Stay tuned for further improvements! ####I’m happy about any kind of feedback. Please upvote if you like it ;-) [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Classifier.gif&userId=1075716 [2]: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge "Kaggle" [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=images.png&userId=1075716 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=trainingResult.png&userId=1075716 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=CM.png&userId=1075716 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Rocs.png&userId=1075716 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Reaction.gif&userId=1075716 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Emoticon.png&userId=1075716 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MonteCarlo.png&userId=1075716 Adam Dendek 2017-07-05T22:32:13Z https://community.wolfram.com/groups/-/m/t/2144200 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/d34dbf01-6652-4598-9cff-6aad9b015495 Wolfram Education Programs 2020-12-21T18:20:08Z https://community.wolfram.com/groups/-/m/t/1661864 [![enter image description here][1]][46] We're glad to announce that Version 12.0 of Mathematica and the Wolfram Language is here—and it's a big one. This major update introduces several hundred new functions in a range of areas, including mathematics and geometry, visualization, audio and image processing, machine learning, and more, as well as provides a major overhaul to the Mac and Linux notebook interfaces and numerous systemwide performance enhancements. Some key new features include: Mathematics and Geometry ------------------------------------- - Convex optimization solvers such as [LinearOptimization][2], [QuadraticOptimization][3] and [SemidefiniteOptimization][4], with vector inequalities such as [VectorGreater][5] for modeling vector-valued variables - Improved [NDSolve][6] support for nonlinear PDEs using FEM and automated idealized *n*-body interaction using [NBodySimulation][7] - Use [Around][8] to describe uncertainty in measured values, with built-in support in visualization functions - A host of new functions for generating, analyzing and manipulating polygons and polyhedra ([RandomPolygon][9], [RandomPolyhedron][10], [PolygonDecomposition][11], etc.), with added support for holes/voids - Access built-in axiomatic theories for use in symbolic proofs with [AxiomaticTheory][12] Visualization ---------------- - [GeometricScene][13] and [GeometricAssertion][14] for visualizing Euclidean geometry and its theorems - Visualize different components and properties of complex-valued data and functions with [ComplexListPlot][15], [ReImPlot][16] and [AbsArgPlot][17] - Represent geodetic vectors with [GeoVector][18] and visualize geodetic vector fields using [GeoVectorPlot][19] and [GeoStreamPlot][20] - Updated default appearance for [GraphPlot][21] visualizations, as well as new plot themes and updated properties - Create, analyze and modify chemical species with [Molecule][22] and visualize their 2D and 3D structures with [MoleculePlot][23] and [MoleculePlot3D][24] Audio and Image Processing --------------------------------------- - Color-processing functions such as [FindMatchingColor][25], [ImageRecolor][26] and [ColorDetect][27], all powered by the new [ColorsNear][28] wrapper to represent a color neighborhood - Multi-language support for [SpeechSynthesize][29], as well as improved discovery and searching of available voices through [VoiceStyleData][30] - [AudioAnnotate][31] framework for annotating audio objects - [ShortTimeFourier][32] for computing and storing properties of short-time Fourier transforms and [InverseSpectrogram][33] for approximating a signal from a magnitude spectrogram Machine Learning and Neural Networks ------------------------------------ - High-level recognition functions such as [ImageCases][34], [FacialFeatures][35], [AudioIdentify][36] and [SpeechRecognize][37] - Additional pre-trained models, new layers, new encoders and decoders, and multi-GPU support for the neural network framework - Learn a generalized probability distribution from arbitrary data using [LearnDistribution][38] - Replace missing values in data by generated values using [SynthesizeMissingValues][39] and find anomalous examples in data using [FindAnomalies][40] These are just a few of the highlights from this release, which also includes an experimental [Wolfram Compiler][41], a comprehensive new [Microcontroller Kit][42], a bidirectional [Unity][43] game engine interface and expanded knowledge and functionality for [Entity][44] and [EntityStore][45]. You can find out more about these and other Version 12 additions by reading through [Stephen Wolfram's launch-day blog post][46] and looking at the [Summary of New Features in 12][47]. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=12-0-wordcloud-hero.png&userId=11733 [2]: http://reference.wolfram.com/language/ref/LinearOptimization [3]: http://reference.wolfram.com/language/ref/QuadraticOptimization [4]: http://reference.wolfram.com/language/ref/SemidefiniteOptimization [5]: http://reference.wolfram.com/language/ref/VectorGreater [6]: http://reference.wolfram.com/language/ref/NDSolve [7]: http://reference.wolfram.com/language/ref/NBodySimulation [8]: http://reference.wolfram.com/language/ref/Around [9]: http://reference.wolfram.com/language/ref/RandomPolygon [10]: http://reference.wolfram.com/language/ref/RandomPolyhedron [11]: http://reference.wolfram.com/language/ref/PolygonDecomposition [12]: http://reference.wolfram.com/language/ref/AxiomaticTheory [13]: http://reference.wolfram.com/language/ref/GeometricScene [14]: http://reference.wolfram.com/language/ref/GeometricAssertion [15]: http://reference.wolfram.com/language/ref/ComplexListPlot [16]: http://reference.wolfram.com/language/ref/ReImPlot [17]: http://reference.wolfram.com/language/ref/AbsArgPlot [18]: http://reference.wolfram.com/language/ref/GeoVector [19]: http://reference.wolfram.com/language/ref/GeoVectorPlot [20]: http://reference.wolfram.com/language/ref/GeoStreamPlot [21]: http://reference.wolfram.com/language/ref/GraphPlot [22]: http://reference.wolfram.com/language/ref/Molecule [23]: http://reference.wolfram.com/language/ref/MoleculePlot [24]: http://reference.wolfram.com/language/ref/MoleculePlot3D [25]: http://reference.wolfram.com/language/ref/FindMatchingColor [26]: http://reference.wolfram.com/language/ref/ImageRecolor [27]: http://reference.wolfram.com/language/ref/ColorDetect [28]: http://reference.wolfram.com/language/ref/ColorsNear [29]: http://reference.wolfram.com/language/ref/SpeechSynthesize [30]: http://reference.wolfram.com/language/ref/VoiceStyleData [31]: http://reference.wolfram.com/language/ref/AudioAnnotate [32]: http://reference.wolfram.com/language/ref/ShortTimeFourier [33]: http://reference.wolfram.com/language/ref/InverseSpectrogram [34]: http://reference.wolfram.com/language/ref/ImageCases [35]: http://reference.wolfram.com/language/ref/FacialFeatures [36]: http://reference.wolfram.com/language/ref/AudioIdentify [37]: http://reference.wolfram.com/language/ref/SpeechRecognize [38]: http://reference.wolfram.com/language/ref/LearnDistribution [39]: http://reference.wolfram.com/language/ref/SynthesizeMissingValues [40]: http://reference.wolfram.com/language/ref/FindAnomalies [41]: http://reference.wolfram.com/language/guide/CodeCompilation [42]: http://reference.wolfram.com/language/MicrocontrollerKit/guide/MicrocontrollerKit [43]: http://reference.wolfram.com/language/UnityLink/guide/UnityLink [44]: http://reference.wolfram.com/language/ref/Entity [45]: http://reference.wolfram.com/language/ref/EntityStore [46]: http://wolfr.am/launching-12 [47]: http://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn12.html Brian Wood 2019-04-16T16:11:50Z https://community.wolfram.com/groups/-/m/t/2934668 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=_0confeelezgif.gif&userId=11733 [2]: https://www.wolframcloud.com/obj/e403aedf-46c7-494c-976a-69eb4d5567c5 Vitaliy Kaurov 2023-06-12T05:51:27Z https://community.wolfram.com/groups/-/m/t/1606199 **[Open code in Cloud][1] | Download code to Desktop via Attachments Below** -------------------------------------- ![enter image description here][2] ![enter image description here][3] [@Marco Thiel][at0] pointed me to an interesting project called [The Venice Time Machine][4] (see also a review [video][5] and a [TED talk][6]). The project aims to digitize and algorithmically process a wealth of data packed in ancient manuscripts. Some steps include the following: > **SCANNING**. Paper documents are turned into high-resolution digital images with the help of scanning machines. > **TRANSCRIPTION**. The graphical complexity and diversity of hand-written documents make transcription a daunting task. For the Venice Time Machine, scientists are currently developing novel algorithms that can transform images into probable words. The images are automatically broken down into sub-images that potentially represent words. Each sub-image is compared to other sub-images, and classified according to the shape of word it features. Each time a new word is transcribed, it allows millions of other word transcripts to be recognized in the database. > **TEXT PROCESSING**. The strings of probable words are then turned into possible sentences by a text processor. This step is accomplished by using, among other tools, algorithms inspired by protein structure analysis that can identify recurring patterns. ![enter image description here][7] Because words in the manuscripts are mutually compared to identify similar features and ultimately same words - it reminded me of unsupervised learning. I wanted to play a bit with tech built in in Wolfram Language to see some toy examples at work. (Note, a supervised learning version of this was developed by Marco in a cool project [Classifying Japanese characters from the Edo period][8]). To avoid hunting and cleaning real datasets, lets "simulate handwriting" reflecting on randomness of the character shapes. I also like it because it touches upon some art sides of things, relevant to generation of beautiful random fonts. And actually due to ancient writers' (for example monks) skills and attitudes often handwritten characters will differ less than the typed ones we distort here artificially (see image above). It's a different story of course if we to compare between handwritings of different authors, which is a challenge and what the Venice project envisions ultimately. This becomes even harder if the letters of a handwritten word are seamlessly connected. I deal with single characters, not words. Let's define a function that makes an image from a letter: raster[s_]:=ImagePad[ImageCrop[Rasterize[s,ImageSize->150]],50,White] I padded it to give enough room for random distortions we will apply. Next function looks convoluted, but it is just a transformation formula that sends sin and cos waves over the letters to distort the shapes (think of your reflection in the water of a pond when you drop a stone there). transf[rn_,amp_,fre_]:= #+{ amp Sin[rn[[1]]+fre(#[[1]]+#[[2]])]+amp Cos[rn[[2]]+fre(#[[1]]-#[[2]])], amp Cos[rn[[1]]-fre(#[[1]]-#[[2]])]-amp Sin[rn[[2]]-fre(#[[1]]+#[[2]])]}& The next function actually applies a random transformation to a letter image: randTransf[amp_,fre_][i_]:= With[{rn=RandomInteger[99,2]},ImageTransformation[i,transf[rn,amp,fre]]] Trying this in action for the letters of the word WOLFRAM and "Felipa" font: set:=ImagePad[#,-50,Red]&/@randTransf[.02,20]/@raster/@ (Style[#,FontFamily->"Felipa"]&/@Characters["WOLFRAM"]) set//ConformImages//ImageAssemble ![enter image description here][9] Trying this twice and finding the difference reveals that different random samples differ quite strongly: ImageSubtract[set//ConformImages//ImageAssemble,set//ConformImages//ImageAssemble ![enter image description here][10] Lets generate 6 different random versions (each per a CPU core of my machine for parallel processing): data=Flatten[ParallelTable[set,6]]; Thumbnail[#,50,Padding->None]&/@data ![enter image description here][11] While letters look the same, if you pay attention to the details, the smaller features are different, - that reminds me of ink and hands effects on real paper. Now, to the machine learning. Here is Wolfram Language "out of the box" function, FeatureSpacePlot, that uses machine learning to deduce most relevant features of objects and find similar ones demonstrating the similarity in 2D space. Here the closer objects are more similar. It picks the similarity, correctly gathering letters into 7 different groups - each group for each letter in WOLFRAM: FeatureSpacePlot[data,PlotTheme->"Frame"] ![enter image description here][12] Let's try now another effect. I will increase the frequency of distortion making letters "fuzzy" (I cannot ignore the artistry of the visual :-) ). Repeated generations will give different small-scale features: set:=ImagePad[#,-50,Red]&/@randTransf[.05,100]/@raster/@(Characters["WOLFRAM"]) set ![enter image description here][13] While general shape is the same internal details are very different and noisy. Again generating 6 different sets and applying FeatureSpacePlot we are able to group properly: data=Flatten[ParallelTable[set,6]]; Thumbnail[#,50,Padding->None]&/@data FeatureSpacePlot[data, PlotTheme -> "Frame", PerformanceGoal -> "Quality"] ![enter image description here][14] Of course, with more distortion we would need more sophisticated processing to find clusters of similarity. But this was fun to play with. It would be interesting to come up with something that could cluster same letters even between the 2 fonts shown at the top image in this post. [at0]: https://community.wolfram.com/web/mthiel [1]: https://www.wolframcloud.com/objects/wolfram-community/Random-Fonts-&-Similarities-in-Deformed-Shapes-by-Vitaliy-Kaurov [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=34qtregadsf.png&userId=11733 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=weraw4qtergafdv.png&userId=11733 [4]: http://timemachine.eu/ [5]: https://vtm.epfl.ch/page-110486-en-html/ [6]: https://www.youtube.com/watch?v=b6nfoE9ly90 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-02-05at4.59.26AM.jpg&userId=11733 [8]: https://community.wolfram.com/groups/-/m/t/1221098 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=34qtregadsf.png&userId=11733 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ert43q5etrsbdvzs.png&userId=11733 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=sfaawereg.png&userId=11733 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ertgf34.png&userId=11733 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=weraw4qtergafdv.png&userId=11733 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=sdf3q4tergafsdhng.png&userId=11733 Vitaliy Kaurov 2019-02-05T12:24:26Z https://community.wolfram.com/groups/-/m/t/1383429 ![*Add Cool Image][1] #Microsite# https://one.wolframcloud.com/objects/rohit.panse8/pneumoniadiagnosis #Introduction# Pneumonia is an infection that causes inflammation in one or both of the lungs; it is induced by a variety of organisms: bacteria, viruses, and fungus. Bacterial pneumonia, the more common type, is caused by bacteria that multiplies in the lungs; Viral Pneumonia is caused by an array of viruses, the most common one being Influenza. Although the severity of pneumonia can vary, young children, seniors, and people with a weakened immune system are the most vulnerable.The CDC reports that of the close to 540,000 cases of pneumonia each year, 50,000 people die. Early and proper diagnosis can tremendously decrease mortality rate. Conventional diagnosis methods like examining patient medical history and symptoms have their shortcomings. Dutch researchers, who published their findings in the European Respiratory Journal, found that of 140 patients who had their pneumonia diagnosed by x-ray, doctors initially thought only 41 of them had the severe lung infection. Chest X-rays have long been reliable diagnostic tools for pneumonia. Since pneumonia, is an airspace disease, patients' air spaces are filled with bacteria, viruses, pus, and other microorganisms. A diagnosis is determined by infiltrates, or white spots, present in a chest X-ray. My project uses a convolutional neural network to diagnose the type of pneumonia that a patient has and. Thank you to Daniel Kermany, Daniel Zhang, and Michael Goldbaum for creating and labeling the dataset. Thanks to Paul Timothy Mooney for making the dataset available on Kaggle. <br /> <br /> Below is an example of an infiltrate present in a chest X-ray. ![enter image description here][2] <br /> Below is an example of how chest X-rays of the three classifications generally look. Chest X-rays of pneumonia patients are characterized by loss of diaphramatic shadows, and evidence of pulmonary consolidation(an area of the lung that has filled with liquid instead of air). ![enter image description here][3] #Diagnosing pneumonia# ##Neural Network I used transfer learning to create my neural network. Transfer learning is the process of using a pre-trained model as a starting point as a model for a second task. > "Transfer learning and domain adaptation refer to the situation where > what has been learned in one setting \[Ellipsis] is exploited to > improve generalization in another setting"-(Page 526, Deep Learning, > 2016) I used ResNet 50, an architecture released in 2015 by Microsoft Research Asia; this network was originally trained on 1.2 million images with 1000 classes. The immense computing power, time resources, and knowledge dedicated to perfect this network made it a great neural network to use for transfer learning. <br /> <br /> Below is an example of the ResNet 50 in action. It correctly predicts that the image is a golden retriever. In addition, I set the program to output the top 10 probabilities of what it thought the image was. ![enter image description here][4] There are two notable layers in this network: linear layer and softmax layer. First, the Linear Layer maps the features processed by the other layers to the classifications given. Classifications(classes) are the different objects the neural network is trained to recognize. In the standard ResNet's case the classes are the 1000 different objects that it is trained to recognize. In the case of my project, the classes are a normal, bacterial pneumonia, and viral pneumonia chest X-rays. Second, the Softmax layer determines the probabilities of what the image is. Above the probabilities of the image are displayed; the class with the highest probabilities is what the Neural Net outputs. <br /> <br /> Below, I remove the Linear and the Softmax layer. These layers are currently configured to recognize 1000 classes; I only need them to recognize 3 classes of images(normal, bacterial pneumonia, viral pneumonia). I will add these two layers later on. ![enter image description here][5] <br /> Below, I replaced the "Input" layer to recieve an input of an image of dimensions 512 by 512. I will configure my training images to be of this size. ![enter image description here][6] <br /> I added 3 layers to a new neural net. The above resizedNet will extract features from the dataset. This new neural net that I created will be used to interpret the features and output what it thinks the input image was. The 3 layers that will do this are the "DropoutLayer", "LinearLayer", and "SoftmaxLayer". The "DropoutLayer" is used to reduce overfitting; this is the tendency of a neural network to become extremely specialized to a limited set of data points. Overfitting is not desirable because it makes the neural network very ineffective on data points outside of the training set. The Linear layer maps the features from the above layers to the 3 classes, and the softmax layer determines the probabilities of the features. Additionally, the output, determining what class the chest X-ray is, is specified. 0 means a normal chest X-ray, 1 means virus pneumonia, 2 means bacterial pneumonia. <br /> ![enter image description here][7] <br /> ##Data Importing and Formatting I imported the files from their respective folders. The virus and bacterial pneumonia files were in the same formula, so I used a simple separating sort tool organize only images of a certain class to their corresponding variables. <br /> normalImg = Import /@ FileNames["*.jpeg", NotebookDirectory[] <> "train\\NORMAL"]; virusImg = Import /@ FileNames["*_virus_*.jpeg", NotebookDirectory[] <> "train\\PNEUMONIA"]; bacImg = Import /@ FileNames["*_bacteria_*.jpeg", NotebookDirectory[] <> "train\\PNEUMONIA"]; <br /> I conformed the images to have dimensions of 512 by 512. I did this because I specified the Input layer to take images of only those dimensions. <br /> normNorm = ConformImages[Import /@ normalFiles, {512, 512}]; virusNorm = ConformImages[Import /@ virusFiles, {512, 512}]; bacNorm = ConformImages[Import /@ bacFiles, {512, 512}]; <br /> Below is how conformed images look. <br /> Before: <br /> ![enter image description here][8] <br /> After: <br/> ![enter image description here][9] <br /> Later, I adjusted the way I conformed my images. I augmented my data set to improve my neural net. By altering the data set, my neural net would become adapted to a greater variety of images. <br /> normNorm = ConformImages[ Table[If[RandomReal[] > .8, If[RandomReal[] > .5, rotateImage[i], shearImage[i]], i], {i, Import /@ normalFiles}], {512, 512}]; virusNorm = ConformImages[ Table[If[RandomReal[] > .8, If[RandomReal[] > .5, rotateImage[i], shearImage[i]], i], {i, Import /@ virusFiles}], {512, 512}]; bacNorm = ConformImages[ Table[If[RandomReal[] > .8, If[RandomReal[] > .5, rotateImage[i], shearImage[i]], i], {i, Import /@ bacFiles}], {512, 512}]; <br /> I made 40% of the images undergo a rotation via rotateImage, and I made another 40% of the images undergo a perspective transformation via shearImage. <br /> rotateImage[i_] := ImageRotate[i, RandomReal[{-Pi/27, Pi/27}]]; shearImage[i_] := ImagePerspectiveTransformation[ i, {{1, RandomReal[{-.25, .25}], 0}, {0, 1, 0}, {0, 0, 1}}]; <br /> Below are the images before a rotate and perspective change respectively. ![enter image description here][10] <br /> ##Preparing Images Above, two separate neural networks were created. One, the resizedNet, extracts features. The other, the learnNet, would interpret the features and determine the output. By creating two separate neural networks we can preprocess the data instead of having to process the data repeatedly as we would have to in a prematurely combined network. Below, I process the images to extract features using the resizedNet, a neural network that uses the ResNet 50 framework. <br /> normalFeats = resizedNet[normNorm, TargetDevice -> "GPU"]; virusFeats = resizedNet[virusNorm, TargetDevice -> "GPU"]; bacFeats = resizedNet[bacNorm, TargetDevice -> "GPU"]; <br /> Below, the feature extracted images undergo two processes. First, the images are mapped to certain numerical codes. The normal chest X-rays are mapped to the code 0, the virus chest X-rays are mapped to the code 1, and the bacteria chest X-rays are mapped to the code 2. The images are mapped to codes, so the neural network knows the identity of the images when training the network. In addition, the codes are outputted when the trained network interprets an image to be a certain class. For example if the trained network is given a normal chest X-ray, then it should output the code 0. Second, the processed images are joined into one massive list. <br /> feats = RandomSample[ Join[Map[# -> 0 &, resizedNet[normNorm, TargetDevice -> "GPU"]], Map[# -> 1 &, resizedNet[virusNorm, TargetDevice -> "GPU"]], Map[# -> 2 &, resizedNet[bacNorm, TargetDevice -> "GPU"]]]]; <br /> The massive list of processed images are partitioned into a training set and a test set. The training set will train the network and determine the proper weights needed to recognize the images properly. The test set will be used to check the true progress of the neural network. Since, the neural network is never trained on the test set, it would serve as an objective indicator of progress when training. <br /> trainFeat = feats[[1 ;; 5000]]; testFeat = feats[[5000 ;; 5200]]; ##Training the Neural Net# <br /> The below script was used to train the learnNet with the training data set, trainFeat, and the testing data set. <br /> trainedNet = NetTrain[learnNet, trainFeat, ValidationSet -> testFeat, MaxTrainingRounds -> 35, TargetDevice -> "GPU"] <br /> I trained the neural network 4 separate times each with different settings: 1. Regular data set 35 rounds of training 2. Regular data set 70 rounds of training 3. Augmented data set 35 rounds of training 4. Augmented data set 70 rounds of training <br /> The accuracy results of the neural network are below 1. Regular 35 rounds: about 24.9% error 2. Regular 70 rounds: about 19.9% error 3. Augmented 35 rounds: about 20.9% error 4. Augmented 70 rounds: about 17.4% error <br /> Below is an image loss function and error rate of the Augmented 70 rounds neural net. I used this neural net in the final product. <br /> ![enter image description here][11] <br /> The trained neural network was exported for use on the microsite. <br /> Export["/pnemonia.wlnet", combinedNet] <br /> #Determining Location of Pneumonia# <br /> 4 separate neural networks were created to determine the location of the pneumonia: leftVirus, rightVirus, leftBacteria, rightBacteria. The left lob images were evaluated in a separate neural network because an image of the left side of the chest when compared to the right side of the chest may appear to have pneumonia; the heart appears very white much like a pulmonary consolidation, a key evidence of pneumonia. Evaluating the lobes separately for each type of pneumonia helped ensure accuracy in location recognition. <br /> ##Importing Images <br /> For the pneumonia location detection part of this project a subset of data was created. The right and left lungs are often referred to as lobes. 100 images of left lob normal and right lobe normal each and 100 images of left lobe virus, right lobe virus, left lobe bacteria, right lobe bacteria pneumonia each were selected to train 4 separate neural nets to help determine location of the pneumonia. <br /> Below are the statements to import the unedited chest X-rays. halfNormalImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/NORMAL"]; leftVirusImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/VIRUS/Left_Real"]; rightVirusImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/VIRUS/Right_Real"]; leftBacImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/BACTERIA/Left_Real"]; rightBacImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/BACTERIA/Right_Real"]; <br /> ##Image Editing Functions# <br /> Below are the functions which were used to edit the images. "widthResize" and "cropFinder" both are used in conjunction to crop the image. The number of black pixels on both sides of an image are found at 5 separate places. The greatest amount of pixels for both side are used to crop the image. This function was useful because it almost always placed the spinal chord in the center, and as a result the images could be correctly separated. cropFinder[x_, img_] := Module[{mid, flatValues, cropLeft, cropRight}, mid = Ceiling[Last[ImageDimensions[img]]/(x)]; flatValues = Flatten@ImageData[ ImageTake[ColorConvert[img, "Grayscale"], {mid, mid}]]; cropLeft = LengthWhile[flatValues, # < 0.2 &]; cropRight = LengthWhile[Reverse[flatValues], # < 0.2 &]; List[cropLeft, cropRight] ] widthResize[img_] := Block[{ flatValues, cropLeft, cropRight, meanLeft, meanRight, list1, list2, list3, list4, list5, resizedImage}, list1 = cropFinder[10, img]; list2 = cropFinder[2, img]; list3 = cropFinder[4, img]; list4 = cropFinder[1.5, img]; list5 = cropFinder[1, img]; meanLeft = meanLeft /. x_ /; x > 300 :> 0; meanLeft = -(Max[{list1[[1]], list2[[1]], list3[[1]], list4[[1]], list5[[1]]}]); meanRight = meanRight /. x_ /; x > 300 :> 0; meanRight = -((Max[{list1[[2]], list2[[2]], list3[[2]], list4[[2]], list5[[2]]}])); resizedImage = ImagePad[img, {{meanLeft, meanRight}, {0, 0}}]; ImageResize[resizedImage, {512, 512}] ] <br /> ![enter image description here][12] <br /> ##Image Editing Functions cont.# <br /> The below functions were used to separate the images into halves. If a certain image was in the left lobe virus data set, only the left side was extracted. The respective operations were performed on all of the groups of data. (Note: The chest X-rays display the right side of the chest on the left and the left side of the chest on the right. When I crop the image for the left section it will extract the right side of the image.) <br /> halfFormat[x_, y_] := Block[{i} , Table[ImageCrop[widthResize[x[[i]]], {256, 512}, y], {i, Length[x]}]]; singleHalfFormat[x_, y_] := ImageCrop[widthResize[x], {256, 512}, y]; <br /> ![enter image description here][13] <br /> Below was the script that was used to edit the images <br /> leftNormalNorm = halfFormat[widthResize[halfNormalImg], Left]; rightNormalNorm = halfFormat[widthResize[halfNormalImg], Right]; leftVirusNorm = halfFormat[widthResize[leftVirusImg], Left]; rightVirusNorm = halfFormat[widthResize[rightVirusImg], Right]; leftBacNorm = halfFormat[widthResize[leftBacImg], Left]; rightBacNorm = halfFormat[widthResize[rightBacImg], Right]; <br /> ##Neural Network <br /> The ResNet 50 was used for transfer learning with a similar set up as the neural network created above. <br /> ![enter image description here][14] <br /> A "halfLearnNet" was created with a "DropoutLayer", "LinearLayer", "SoftmaxLayer", and an "Output" of two classes. The outputs were 0 and 1; 0 means that the chest X-ray given is a normal image(no pneumonia); 1 means that the lung had pneumonia. <br /> ![enter image description here][15] <br /> ##Preparing Images/Training the Neural Net# <br /> The below sequence of steps were used for the other neural nets (right lobe virus, left lobe bacteria, right lobe bacteria). The trained files were exported for use on the microsite. <br /> ![enter image description here][16] <br /> Below are error rates of the location detecting neural networks. <br /> - Virus Left : about 5% error <br /> - Virus Right : about 5% error <br /> - Bacteria Left: about 0% error <br /> - Bacteria Right: about 15% error <br /> #Creating a MicroSite# <br /> Due to the immense size of my trained neural nets, I needed to create a cloud objects for them. <br /> bacLeftNet = CloudImport[CloudObject["bacLeftNet.wlnet"]] bacRightNet = CloudImport[CloudObject["bacRightNet.wlnet"]] virusLeftNet = CloudImport[CloudObject["virusLeftNet.wlnet"]] virusRightNet = CloudImport[CloudObject["virusRightNet.wlnet"]] CloudImport[CloudObject["pneumonia.wlnet"]] <br /> I created a function which performed my desired tasks: Determine if the given chest X-ray is normal, bacterial, or virus. If the chest X-ray appeared to have pneumonia, the function outputted an the probability that the pneumonia was in the right lung and the left lung. In addition, the given image was returned so the user could view. it. <br /> ![enter image description here][17] <br /> Below was the code for my microsite. <br /> CloudDeploy[Delayed@FormPage[{"image" -> "Image"}, tesla[#image] &, AppearanceRules -> <| "Title" -> "Machine Learning Pneumonia Diagnosis", "Description" -> "<style>.wolfram-branding>.wolfram-branding-cloud:after{\ background-image: url(http://i.imgur.com/hFuh1YT.png)}</style>Upload \ a Chest X-Ray to recieve a diagnosis", "SubmitLabel" -> "Classify", "PageTheme" -> "Red"|>], "test3", Permissions -> "Public"] <br /> If an image is given to the microsite the following is outputted. <br /> ![enter image description here][18] #Real World Testing# <br /> Real world testing was performed to simulate a real use case. I took a picture of the screen, like someone would take a picture of an chest X-ray. <br /> ![enter image description here][19] <br /> The neural network correctly guessed that the image was an instance of viral pneumonia. <br /> ![enter image description here][20] <br /> #Conclusion and Acknowledgement <br /> I created a transfer learning neural networks which successful diagnosed pneumonia with an accuracy of about 83% and accurately detected the location of the pneumonia with an average accuracy of about 93%. Data editing and augmentation procedures and increasing the amount of training rounds helped increase accuracy of diagnoses. I would like to thank my mentor, Michael Kaminsky, for helping guide me. In addition, I would like to thank other mentors including Rick Harrigan, Andrea Griffin, Douglas Smith, and Greg Hurst for helping me. Finally, I would like to thank my parents for encouraging me to pursue my passions and keep learning. #Future Improvements# - Improving image formatting function to make the current program more compatible with poor quality chest X-Rays. - Using a similar Neural Network set up except with thermal images to efficiently diagnose pneumonia. - Further augmenting data to improve accuracy of the neural network. #Citations https://www.mayoclinic.org/diseases-conditions/pneumonia/symptoms-causes/syc-20354204 https://my.clevelandclinic.org/health/diseases/4471-pneumonia https://www.hopkinsmedicine.org/healthlibrary/conditions/respiratory_disorders/pneumonia_85,P01321 https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia https://www.med-ed.virginia.edu/courses/rad/cxr/pathology3chest.html https://machinelearningmastery.com/transfer-learning-for-deep-learning/ https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618/ref=as_li_ss_tl?ie=UTF8&qid=1505772822&sr=8-1&keywords=deep+learning&linkCode=sl1&tag=inspiredalgor-20&linkId=e16a8bbb28ecba154d225ca5f7efcbed http://www.cell.com/cell/fulltext/S0092-8674(18)30154-5 https://data.mendeley.com/datasets/rscbjbr9sj/2 [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-13at10.32.21PM.png&userId=1372752 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt1.jpeg&userId=1372752 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt2.jpeg&userId=1372752 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt4.jpeg&userId=1372752 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt5.jpeg&userId=1372752 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt6.jpeg&userId=1372752 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt7.jpeg&userId=1372752 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt8.jpeg&userId=1372752 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt9.jpeg&userId=1372752 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg2.jpeg&userId=1372752 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg3.jpeg&userId=1372752 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg4.jpeg&userId=1372752 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg5.jpeg&userId=1372752 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg6.jpeg&userId=1372752 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg7.jpeg&userId=1372752 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg8.jpeg&userId=1372752 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg9.jpeg&userId=1372752 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=lol1.jpeg&userId=1372752 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=lol2.jpeg&userId=1372752 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-13at11.11.17PM.png&userId=1372752 Rohit Panse 2018-07-14T03:48:35Z https://community.wolfram.com/groups/-/m/t/1137055 ## Introduction **Instance Segmentation** is one of many interesting computer vision tasks that combines **object detection** and **semantic segmentation**. It detects the object, and at the same time, generates a segmentation mask, which you can think as classifying each pixel - whether it belongs to an object or not. Therefore, solving object detection and semantic segmentation together is a good approach to solve instance segmentation. In the summer school, we borrowed the framework of Mask R-CNN, to combine object detection and semantic segmentation in one pipeline, and produced some promising results. Mask R-CNN is developed like this: R-CNN -> Fast R-CNN -> Faster R-CNN -> Mask RCNN. In each stage, researchers solved some bottleneck problems to get faster and better performance. R stands for the region based, so R-CNN a region based convolutional neural network. Mask R-CNN has two stages, the first stage is trying to produce valid bounding box, you can think it as "blobby" image regions because "blobby" regions are likely to contain objects. In the early stage, researchers feed these warped image regions into a convolutional network, and in the output, they put two head, one regression head to produce the bounding-box, another head like SVM to do classify. And peoples kept working on it and make the network more efficient using some tricks like switch the proposal layer and convolutional layer to avoid unnecessary computations. ![enter image description here][1] There is a Github repo [FastMaskRCNN][2], several machine learning enthusiasts are trying to reproduce this paper in Tensorflow. I already obtained >400k epoch weights, but haven't tested yet, I will keep working on it after summer school. This Mask R-CNN model a too big to finish within 2 weeks, especially since I am new to the TensorFlow framework. Here is the graph visualization from TensorBoard, which looks really complicated and I haven't figure out the pipeline yet. ![enter image description here][3] ## My Project Because building Mask R-CNN network in Mathematica side turned out to be too complicated to finish before the deadline. So in the last two days of summer school, I did something simpler, but it's inspired by Mask R-CNN's framework, which uses the bounding-box region and corresponding mask to train a network to produce a binary mask (pixel-to-pixel). And this process is like semantic segmentation. In order to get this "pixel-to-pixel" trianing dataset, I wrote a script to process 24k [COCO][4] train2014 dataset and to crop the bounding box region based on the annotation `json` file. Here is how information encoded for object instance annotations: annotation{ "id" : int, "image_id" : int, "category_id" : int, "segmentation" : RLE or [polygon], "area" : float, "bbox" : [x,y,width,height], "iscrowd" : 0 or 1, } The original annotation `json` data looks like this: {"area" -> 54653., "segmentation" -> {{312.29, 562.89, 402.25, 511.49, 400.96, 425.38, 398.39, 372.69, 388.11, 332.85, 318.71, 325.14, 295.58, 305.86, 269.88, 314.86, 258.31, 337.99, 217.19, 321.29, 182.49, 343.13, 141.37, 348.27, 132.37, 358.55, 159.36, 377.83, 116.95, 421.53, 167.07, 499.92, 232.61, 560.32, 300.72, 571.89}}, "iscrowd" -> 0, "bbox" -> {116.95, 305.86, 285.3, 266.03}, "image_id" -> 480023, "category_id" -> 58, "id" -> 86} The code is very simple, we just need to do the following things: - Use the bounding-box `{{x, y}, {dw, dh}` to trim the image into bounding-box region image. - Use the list of points (vertices of the polygon (encoding ground truth segmentation mask) ) and bounding-box image dimensions to produce the corresponding mask for the region image. convertAnnotationBBoxIntoImageCoord[bbox_List, h_]:= Module[ {x1, y1, dw, dh}, {x1, y1, dw, dh} = bbox; Transpose[{0, h} + {1, -1} Transpose[Partition[{x1, y1, x1 + dw, y1 + dh}, 2]]] ] getMask[pts_List, w_, h_]:= Module[ {vtx}, vtx = convertAnnotationMask[pts, h]; Binarize @ Rasterize[Graphics[{White, Polygon@@vtx}, PlotRange->{{0, w}, {0, h}}, Background->Black],"Image", ImageSize->{w, h}] ] extractBBoxAndMask[img_, bbox_, segPts_List]:= Module[ {bboxLocal, w, h, boxRegion, maskRegion}, {w, h} = ImageDimensions[img]; bboxLocal = convertAnnotationBBoxIntoImageCoord[bbox, h]; boxRegion = ImageTrim[img, bboxLocal]; maskRegion = ImageTrim[getMask[segPts, w, h], bboxLocal]; {boxRegion, maskRegion} ] imgsAndMasks = extractBBox[imgs[[#]], bboxes[[#]], maskCoord[[#]]] & /@ Range[10] ![enter image description here][5] For the network I used to train, I used ENet, which is a very fast and an efficient network. The Mask R-CNN paper used FCN, which is known as standard network to perform semantic segmentation. I also construct this network in Mathematica and I will try it later as well. The ENet architecture is as followings: ![enter image description here][6] ![enter image description here][7] The output is a 256 * 256 * 2 tensor produced by a softmax layer, so it encoded the mask as the Pr[ this pixel belongs to object]. Therefore, I also need to convert the mask image from {0,1} binary into {1,2} as the class label and save it as `.dat` as training labels. Because I was running out of the time, I just trained with this simple input and output. A better way Etienne suggested is to extract the output from the final convolutional layer of yolo as an input feature to feed in near the output, which I will definitely try soon. I trained my network only for 9 hours on a single Tesla K80 GPU, and already got very promising results. The way my `instanceSegmentation[image, net, detectionThreshold, overlapThreshold]` work is as followings: 1. Use YOLO network as detector to produce labels, bounding-boxes, and probabilities 2. Use bounding-boxes to crop the image object region and feed it to our trained network 3. Take the output tensor, convert it to binaryImages, resize it back to bounding box dimensions by using `ImagePad`. padRegionMask[trimmedMask_, bboxLocal_, w_, h_]:= Block[ {x1,y1,x2,y2}, {{x1,y1},{x2,y2}} = bboxLocal; ImagePad[trimmedMask, {{x1, w - x2}, {y1, h - y2}}] ] instanceSegmentation[img_, ennet_, detectionThreshold_, overlapThreshold_]:= Module[ {labels, bboxes, probs, masks, coloredMasks, yoloVis, yoloRes, rectangles, centers}, yoloRes = detection[img, detectionThreshold, overlapThreshold]; rectangles = Transpose[yoloRes][[2]]; {labels, bboxes, probs} = convertYoloResult[yoloRes]; centers = Mean/@ bboxes; masks = produceMask[ennet, img, bboxes]; coloredMasks = Flatten[{RandomColor[], Opacity[.45], #}&/@ masks]; yoloVis = Transpose @ MapThread[ { {Darker @ Green,Opacity[0],#2}, Style[ Inset[#1<>" \n("<>ToString@Round[#3, .01]<>")", #4, {Center, Center} ], FontSize -> Scaled[.03], FontColor -> White, GrayLevel[0,1], Background -> GrayLevel[1,0] ] }&, {labels, rectangles, probs, centers}]; HighlightImage[img, Join[{coloredMasks,yoloVis}], ImagePadding -> Scaled[.02]] ] Ok, here are some results, I only started training today Many cute dogs ![dogs][8] Me and my mentor ![Me with my mentor ][9] Me and my phone ![Me with my phone][10] Me and my coffee ![Me with my coffee][11] Me and my handbag ![Me with my handbag][12] Me and my classmates and his phone ![Me with my classmates with his phone][13] ## Some personal reflection I enjoyed the summer school overall. I have been hoping to explore the Tensorflow framework and watch Stanford CS231n class for few months, but I was always very occupied with school's classes, other projects, lab's assignments, coding interviews, etc. I finally find some peaceful time to sit down and learn things I had always to learn. And in the process, I am also very amazed by the neural network framework Wolfram people developed. This is a very powerful and user-friendly framework that inherits Wolfram Language's elegant syntax and interactive property. I still have some questions about this framework and plan to learn more about it. ## Future Direction 1. Use FCN to do mask semantic segmentation 2. After obtained Mask R-CNN trained network, deploy it on a server and build an interesting iOS application. 3. Collaborate with Medical school people and apply Mask R-CNN to some medical imaging problems. ## Reference 1. Mask R-CNN Paper: https://arxiv.org/abs/1703.06870 Code (under testing): https://github.com/CharlesShang/FastMaskRCNN 2. ENet: https://arxiv.org/abs/1606.02147 3. YOLO2: https://arxiv.org/abs/1506.02640 4. Fully Convolutional Network (FCN): https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf 5. Project repo: https://github.com/zhuwenzhen/InstanceSegmentation [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Framework.png&userId=524853 [2]: https://github.com/CharlesShang/FastMaskRCNN [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=png.png&userId=524853 [4]: http://mscoco.org/dataset/#download [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0515.14.08.png&userId=524853 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Enet.png&userId=524853 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0515.22.14.png&userId=524853 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.02.58.png&userId=524853 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.20.56.png&userId=524853 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.20.47.png&userId=524853 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.21.35.png&userId=524853 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.25.30.png&userId=524853 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.25.38.png&userId=524853 Wenzhen Zhu 2017-07-05T21:17:23Z https://community.wolfram.com/groups/-/m/t/908790 Introduction ============ In the 21st century with great advancement in technology, photo editing has become a form of art itself. Mosaic Image tool is a great way to create unique and expressive art. Image Mosaic is a simple application that will allow you to simply upload an image and create a mosaic out of a large database of pictures. The aim of my project was to implement image-processing based image mosaic tool within the Wolfram Language. The project consists of three parts: - Equal Size Image Mosaic - Unequal Size Image Mosaic - Interactive web pages ![enter image description here][1] Algorithm & Code ================ The project consists of two options. The first one is creating mosaic images which will be constructed from equal sized images. Whereas the second one considers the color distribution of the original image and based on the result divides it into different sized parts. Datasets -------- Datasets are the set of images that are being used for image colorization. You can set/provide your preferable dataset for creating beautiful images. flagList = ConformImages[ Image /@ Cases[DeleteDuplicates[ EntityValue[EntityList["Country"], "Flag"]], _Graphics]]; flagListMean = ImageMeasurements[#, "Mean"] & /@ flagList; Equal Size Mosaic ----------------- ![enter image description here][2] For creating equal size mosaic, first of we will use the function ImagePartition to divide the original image into equal sized parts. Afterwards, we will consider the mean value of each element and compare it with the mean values of the images of the database. Then the program creates a list which contains the images from the database which are the nearest to each part of the original image. Finally, I used the function ImageAssemble to assemble the images of the newly created list mentioned above. ![enter image description here][3] The function **imagePartition** takes two parameters and partitions an image into the list of partitionSize x partitionSize subimages. imagePartition[image_?ImageQ, partitionSize_?IntegerQ]:= Flatten[ImagePartition[image, partitionSize]]; **PiecewiseMatching** is finding and resizing the element of the dataset which is the nearest to the current image part considering their mean values. piecewiseMatching[imgPart_?ImageQ, dataSet_?ListQ, dataSetMean_?ListQ]:= Module[ {curImgMean =ImageMeasurements[imgPart, "Mean"], nearestMean, nearestMeanPos}, nearestMean =First[ Nearest[dataSetMean, curImgMean]]; nearestMeanPos = First[Flatten@Position[dataSetMean,nearestMean]]; ImageResize[dataSet[[nearestMeanPos]], ImageDimensions[imgPart]] ] **imageAssemble** assembles an image from the list of the already matched images which are the nearest to the original image parts. imageAssemble[imgPieces_?ListQ, inputImageDims_?ListQ, partitionSize_?IntegerQ]:= Module[ {nestedImagePieces, conformImg= ConformImages[imgPieces]}, nestedImagePieces = Partition[conformImg, IntegerPart[First[inputImageDims]/partitionSize]]; ImageAssemble[nestedImagePieces] ] **imageMosaic** uses all the functions mentioned above to create the mosaicImage. imageMosaic[image_?ImageQ, partSmallestSize_?IntegerQ, dataSet_?ListQ] := Module[ {orgImgDim = ImageDimensions[image], orgImgPieces = imagePartition[image, partSmallestSize], dataSetMean = ImageMeasurements[#,"Mean"]&/@dataSet, imgPieces, finalImg}, imgPieces = piecewiseMatching[#, dataSet, dataSetMean]&/@orgImgPieces; finalImg = imageAssemble[imgPieces, orgImgDim, partSmallestSize] ] ![enter image description here][4] Unequal Size Mosaic ------------------- ![enter image description here][5] For creating a mosaic image with unequal parts first we divide the image into already defined equal parts. At the same time, another function computes the exact coordinates of the generated pieces on the main image. Afterwards, the function creates an association from coordinates and corresponding image pieces. The next function compares the mean values of the images from the database to the mean values of the generated pieces. If the difference between those two number is higher than a constant the function divides that piece into other four equal sized parts in other case it transfers the association into new list. This function is repeated over and over again until in the main list is fully transformed. At the end using the function **ImageCompose** the program places each piece on the blank image considering its coordinates. The function **fourCords** computes the exact coordinates of newly generated pieces and **fourImgs** divides the given image into four equal parts. After all this, the **getAssoc** function creates an association from the coordinates and the corresponding images. **imageDivide** function breaks the image into equal parts as long as they are bigger than the given maximum size of the part. fourImgs[img_?ImageQ]:= Flatten[ImagePartition[img, ImageDimensions[img]/2]]; getAssoc[list_,image_]:=AssociationThread[fourCords[list] ,fourImgs[image]]; fourCords[cord_List]:=Module[ {res=Flatten@cord, midX, midY}, midX = res[[1]] + (res[[3]]-res[[1]])/2; midY = res[[2]] + (res[[4]]-res[[2]])/2; N@{ {{res[[1]],midY},{midX,res[[4]]}}, {{midX,midY},{res[[3]],res[[4]]}}, {{res[[1]],res[[2]]},{midX,midY}}, {{midX,res[[2]]},{res[[3]],midY}} } ]; imageDivide[image_?ImageQ, max_?IntegerQ] :=Module[ {imgAssoc}, imgAssoc = Association[{{1,1},ImageDimensions@image}->image]; While[First[ImageDimensions[First[Flatten[Values[imgAssoc]]]]]*Last[ImageDimensions[First[Flatten[Values[imgAssoc]]]]]>max, imgAssoc=Association@MapThread[getAssoc[#1,#2]&,{Keys[imgAssoc],Values[imgAssoc]}] ]; imgAssoc ]; The function **imageMatching** compares the generated pieces to the ones from the database and after finding a match replaces it with the image from the database. imageMatching[imgAssoc_?AssociationQ, smallImages_?ListQ, min_?IntegerQ, allowedDistance_] := Module[ {inList = imgAssoc, dist =(First@#)&/@smallImages, nearestMean,curImg,nearestMeanPos, curItem,curImgMean, finalAssoc = <||>}, While [ Length[inList]>0, curItem = First@Normal[inList]; curImg = Last@curItem; curImgMean = ImageMeasurements[curImg,"Mean"]; inList =DeleteCases[inList, curImg]; nearestMean = First@Nearest[dist,curImgMean ]; nearestMeanPos = First[Flatten@Position[dist,nearestMean]]; If[ Norm[curImgMean -nearestMean]<=allowedDistance || First[ImageDimensions[curImg]]* Last[ImageDimensions[curImg]]<=min, curItem = Rule[First[curItem],ImageResize[Last[smallImages[[nearestMeanPos]]], ImageDimensions[curImg]]]; AppendTo[finalAssoc,curItem];, inList = Join[inList,getAssoc[First[curItem],Last[curItem]]]; ] ]; finalAssoc ] **mosiacCompose** mainly uses the function **ImageCompose** to overlay each newly generated piece onto a blank image for creating the mosaic. mosaicCompose[associmage_?AssociationQ, blankImage_?ImageQ] := Module[ {coords,imgs, midCoords, mosaic,i}, imgs = Flatten[Values[associmage]]; coords = Keys[associmage]; midCoords = {#[[1,1]]+(#[[2,1]]-#[[1,1]])/2,#[[1,2]]+ (#[[2,2]] - #[[1,2]])/2}&/@coords; mosaic = blankImage; For[i=1,i<=Length@associmage,i++, mosaic=ImageCompose[mosaic,imgs[[i]],midCoords[[i]]] ]; mosaic ] **imageMosaic** combines all the functions mentioned above to create the mosaic image. imageMosaic[image_?ImageQ, max_?IntegerQ, dataSet_?ListQ, min_?IntegerQ, allowedDistance_]:=Module[ {imgPart= imageDivide[img,max], orgImgDim = ImageDimensions[image], assoc, blankImage, list}, list = Rule[ImageMeasurements[#,"Mean"],#]&/@dataSet; assoc =imageMatching[imgPart,list, min, allowedDistance]; blankImage = ConstantImage[0, orgImgDim]; mosaicCompose[assoc, blankImage] ] ![enter image description here][6] Cloud Deploy ============ Equal Size Mosaic Webpage ------------------------- To make a mosaic image of your image out of flags you can upload your picture to the web form, which can be found [here][7]. ![enter image description here][8] Unequal Size Mosaic Webpage --------------------------- To make an unequal mosaic image of your image out of flags you can upload your picture to the web form, which can be found [here][9] *In case the picture turns out the same as in the case of equal size try changing the specifications/input.* ![enter image description here][10] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=HQ.jpg&userId=900643 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=qq.jpg&userId=900643 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=gfd.jpg&userId=900643 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=asda.jpg&userId=900643 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=cha.PNG&userId=900643 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=czxc.jpg&userId=900643 [7]: https://www.wolframcloud.com/objects/1ba74267-8300-4935-a3d4-88082eff0b09 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Untitled-1.png&userId=900643 [9]: https://www.wolframcloud.com/objects/32b1be17-501d-46b3-9b08-fcb7dd231fcb [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Untitled-2.png&userId=900643 Yeva Gabrielyan 2016-08-19T14:58:13Z https://community.wolfram.com/groups/-/m/t/749947 The Reddit math forum [turned up][1] this [nice site][2] of sudokuish Garam puzzles yesterday. Fill in the blanks. ![A Garam puzzle][3] Someone was quick to post [their Mathematica reduction][4]: ![enter image description here][5] They had to enter the whole puzzle manually, a tedious task. The regular format and clean composition make these puzzles a tempting target for automation. Using colour frequency, I pick out the tiles of the puzzle with morphological tools. ![Said puzzles tiles][6] By checking tile adjacency, I compile lists of 'equations' reading down or across. Then the dark pixels in these tiles are extracted to collect the equation data. ![Some equation imagery][7] A classifier turns the images into equations. ![Some equations][8] Reduce solves all the equations for us, and churns out the answers left to right! ![Solutions][9] Notebook attached for all your Garam cheating needs. [1]: https://www.reddit.com/r/math/comments/3v9ik1/an_interesting_japanese_equation_puzzle_it_took/ [2]: https://www.garamgame.com/ [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=PuzzlePaste.png&userId=445606 [4]: http://i.snag.gy/8cmhp.jpg [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=8cmhp.jpg&userId=11733 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=PuzzleTiles.png&userId=445606 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Someequationimagery.PNG&userId=445606 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Someequations.PNG&userId=445606 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Solutions.PNG&userId=445606 David Gathercole 2015-12-04T09:04:41Z https://community.wolfram.com/groups/-/m/t/509162 By the number of related disciplines, you may appreciate that this is a multidisciplinary problem of significant interest. I will start with the bottom line and then provide brief background. 1. The Fundamental Problem: To Construct a 3-D Model of the 3 mm shell of the Human Cerebral Cortex so that microscopic neuroanatomic data specific to the six layers of 50 different cortical areas can be mapped onto it to show how the human cortex develops during the first six years of life. 2. The Background: We have the largest, quantitative database available on the microscopic, neuroanatomic features of the developing human cerebral cortex. In 1998, these data led to the overturning of the 100 year old dogma of no new neuron formation after birth, and ushered in the era of stem cell research. The data were described by the NY Times as the most important scientific finding of the last decade of the 20th century. Specifically, there are 8 age points of children who died of normal causes, at 0, 1, 3, 6, 15, 24, 48 and 72 months after birth. At each age point, for each layer of 50 cortical areas, there are measures of the numbers of neurons, their size, the numbers of inputs (large dendrites) and outputs (axons), the thickness of each cortical layer, the surface area of each cortical area. We have published that the signal-noise ratio of these data is extremely high, and that there is a common (undiscovered yet) pattern that characterizes the development of each cortical area. 3. The Challenge: To build a 3-D model that can take the values of these microscopic, neuroanatomic features at each age point, layer, and cortical locus, so that these changes can be visualized and analyzed with more complex mathematical methods to discover the underlying rules governing the development of the human cerebral cortex. These rules are likely to govern the structure of development of the cortex of other mammals. 4. An Additional Challenge: We also have camera lucida drawings of the cortical columns of each of these 50 cortical areas for each of the 8 age points. These camera lucida drawings are tracings of representative neurons in each layer of each cortical area. By quantifying these 2-D images and making some assumptions about their 3-D representation, one could incorporate these data into the above model to learn even more. Such research and development should lead to programs and algorithms of extraordinary power that will have use in many different disciplines. I can see how different disciplines would approach this problem in different ways, which is why I posted so many fields of interest. If anyone would like to know more, or has suggestions for developing this project, I have the database, am a statistician and cognitive scientist by training and experience, and am one of the clinical authorities on Alzheimer's disease and other dementing disorders, which is essentially the reverse of development. I would appreciate input from anyone with a creative streak and a taste for a challenging problem. Thanks very much Rod Shankle, MS MD FACP PS: I have attached a PNAS article giving a nice brief summary of the data. William Shankle 2015-06-04T05:57:54Z https://community.wolfram.com/groups/-/m/t/1018424 Suggestion algorithms trained on passed personal preferences are extensively used by various automated assistants. Their purpose could be, for example, marketing of the best possible product choices based on previous purchases, suggestion of new places to visit, semantically organizing photo albums, and many other similar tasks. One of the cornerstones of such algorithms is semantic image distance. The project goal will be building such a measure with help of Wolfram Language neural network tools and dictionary labels with hierarchical semantic structure. High level operation of the application is the following. 1. Recognize produced images and get the probability of each recognized result. 2. Get the leaf vertices of the knowledge graph which corresponds to recognized image results. 3. Pick the top 10 most probable result vertices of each image and calculate the distance of each vertex of the first image with vertices of the second image. 4. Multiply the distance result of the step 3 with their corresponding probabilities. 5. Sum all the results from step 4 to get overall distance between images. For the knowledge graph a subset of WorldNet graph database has been used. The following code is responsible to fetch and prune the WorldNet graph database. createWordnetGraph[parents_]:= Module[{edges}, edges = (First@# -> Last@#)& /@parents; edges = DeleteCases[edges, Null]; Graph[edges] ]; generateGraph[]:=Module[ {wordnetParents,wordnetgraph,networkGraph,ids,currentSynset,extendedGraph}, wordnetParents=Import["http://www.image-net.org/archive/wordnet.is_a.txt","Data"]; wordnetgraph=createWordnetGraph[wordnetParents] (*This graph has been further pruned using the list of synsets known to ImageIdentify. The code to do that is internal and cannot be published here. *) ]; $networkGraph=generateGraph[]; $undirectedNetworkGraph=Graph[VertexList[$networkGraph],EdgeList[$networkGraph]/.DirectedEdge->UndirectedEdge]; At the bottom you can see the diagram of graph that is being used to find out the distances between images. SetProperty[$undirectedNetworkGraph, {EdgeStyle->Directive[White,Opacity[.8]], Background->Black, VertexStyle->Directive[Red,Opacity[.5]], VertexSize->.001}] ![enter image description here][1] The following code is responsible to calculate the distance between two vertices of the graph. First it finds the common ancestor of two vertices, then it sums the distance between first vertex and common ancestor to the second vertex and common ancestor to get the distance between two vertices. calculateDistance[vertex1_, vertex2_]:=Module[{id1,id2,formatedVertex1,formatedVertex2,in1,in2,commanVertices,firstCommanVertex,commonToVertex1Distance,commonToVertex2Distance,distance}, in1=VertexInComponent[$networkGraph,vertex1]; in2=VertexInComponent[$networkGraph,vertex2]; commanVertices=Cases[in1,x_/;MemberQ[in2,x]]; If[Length[commanVertices]>0,( firstCommanVertex=First[MaximalBy[commanVertices,Length@VertexInComponent[$networkGraph,#]&]]; commonToVertex1Distance=GraphDistance[$undirectedNetworkGraph,vertex1, firstCommanVertex]; commonToVertex2Distance=GraphDistance[$undirectedNetworkGraph,vertex2, firstCommanVertex]; distance=commonToVertex1Distance+commonToVertex2Distance; distance ), Infinity] ]; The picDistance function is the main function of the application. It returns the distance between two images. The smaller the return value the more similar the images are. It uses the above calculateDistance function to calculate the distances between vertices. Then it multiplies the distances with the probabilities. picDistance[p1_,p2_]:=Module[{result1,result2,result1Keys,result2Keys,result1Values,result2Values,distance}, (*result1 and result2 contain associations of vertices and their correspondent probabilities for a given image. Internal tools has been used to obtain theses values.*) result1=Sort[result1,Greater]; result2=Sort[result2,Greater]; result1=result1[[1;;10]]; result2=result2[[1;;10]]; result1Keys=Keys[result1]; result2Keys=Keys[result2]; result1Values=Values[result1]; result2Values=Values[result2]; distance=0; itterFunc[result1Item_,result1ItemIndex_]:=MapIndexed[ Set[distance,distance+calculateDistance[result1Item, #1]*result1Values[[result1ItemIndex]]*result2Values[[First[#2]]]] &,result2Keys]; MapIndexed[itterFunc[#1,First[#2]]&,result1Keys]; If[distance==0,Infinity,distance] ]; The following code is a simple test case for picDisttance function. Notice the distance between pictures of the dog and the cat is much higher than the pictures of two cats. ![enter image description here][2] Here is simple demonstration of application. It compares picture of a wolf with 5 random images. ![enter image description here][3] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=afer45w6yjthr.png&userId=11733 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-02-21at10.10.15AM.png&userId=11733 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-02-21at10.11.56AM.png&userId=11733 Ara Yeressian 2017-02-21T16:15:42Z https://community.wolfram.com/groups/-/m/t/2960096 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=lidar_hero_image.png&userId=2955757 [2]: https://www.wolframcloud.com/obj/6a3e49a9-281b-4732-9598-0bbf87e7419b Richard Medina 2023-07-13T01:56:13Z https://community.wolfram.com/groups/-/m/t/1729892 Introduction ======= The aim of this project is to create an algorithm for generating semantically and chromatically similar looking Emojis [avatars] from faces of human beings. This has been achieved by a combination of deep learning and heuristic analysis to produce appropriate results. The data sets used in this project are available for public use and have been referenced below. This report has been divided into two major sections, namely - human analysis and emoji segmentation. Finally, a mapping function is presented to generate the final results. The key principle involved in achieving the goal is cross-domain semantic preservation for effective results. Human Domain ======= Face/Hair Segmentation mask generation ------- One of the key features of a human face to keep in mind is the shape of the hair. For effectively extracting the hair out of a given picture, an edited version of the Dilated ResNet-105 has been trained on a dataset containing face/hair masks of faces from the CelebA dataset. The changes made in the network are for incorporating the training of appropriate dimensions of input and target vectors. net = NetModel["Dilated ResNet-105 Trained on Cityscapes Data"]; netReplaced = NetReplacePart[net, "input_0" -> NetEncoder[{"Image", {178, 218}}]]; netPart = NetTake[netReplaced, {All, "broadcast_add0"}]; newNet = NetGraph[{netPart, DeconvolutionLayer[3, {24, 28}, "Stride" -> {6, 9}, "PaddingSize" -> {{4, 4}, {4, 4}}, "Input" -> {19, 28, 23}], TransposeLayer[{1 <-> 3, 1 <-> 2}], SoftmaxLayer[]}, {1 -> 2, 2 -> 3, 3 -> 4}] This network is trained to classify every pixel of any input image into one of three classes, each of those classes corresponding to one colour of the segmentation mask [face, hair or background]. The loss function utilised is the "**CrossEntropyLossLayer**" function. It has been trained on the 3500 pictures using cross validation techniques over 10 epochs with a batch size of 4 images, due to the relatively small number of training samples available for the task. ![Input/Output for the trained network][1] Colour Preservation for translation to Emoji ------- For colour extraction of the various facial features for a given input image, the approach adopted is face-point based analysis predicted using the "2D Face Alignment" network from the Wolfram Neural Network Repository. With some simple rules established with respect to the morphological components of the face, we are easily able to extract the skin, hair and iris colours of an arbitrary input. PickSkinColorPerson[img_] := Module[{image = img, landmarks, splits, vals, mean}, landmarks = netevaluation[image]; splits = landmarks[[#]] & /@ groupings; vals = PixelValue[image, #] & /@ ScalingTransform[ImageDimensions[image]]@splits[[4]]; mean = Mean[vals[[1 ;; 4]]]; RGBColor[mean]] PickHairColorPerson[img_] := Module[{image = img, net, i, m}, net = Import["/home/kashikar/trained.mx"]; i = ImageResize[image, {178, 218}]; image = net[i]; m = ColorNegate[ Binarize[ ColorDistance[ImageResize[Image[Round[image]], {178, 218}], Red]]]; DominantColors[i, Masking -> m][[1]]] ![Color preservation][2] Emoji Domain ======= ## Structural Extraction and Analysis ## The dataset we use for the making avatars is the Cartoon Set compilation of images wherein a certain number of hairstyles, jawline types, and eye variations have been procedurally combined to create 100k avatars. It should be noted here that the nose and mouth of every emoji are in them same coordinates, with variance in jawlines and appropriate hairstyles. ![Emojis][3] Considering we have hair masks for faces, a heuristic can be setup based on masking specific regions of any arbitrary emoji, taking advantage of the common positions in the dataset, to extract hair, and corresponding jawline/ear combinations for the input. ExtractHairEmoji[img_] := Module[{image = img, image1, poly, skin, cols, haircolor}, image1 = RemoveAlphaChannel[image, White]; poly = Polygon[{{132.72727272727275`, 410.90909090909093`}, { 372.7272727272727, 407.27272727272725`}, {363.6363636363636, 290.90909090909093`}, {133.63636363636363`, 291.8181818181818}}]; skin = RGBColor[ PixelValue[image, {{247, 197}}][[1]]]; cols = DominantColors[image1, Masking -> poly]; haircolor = SelectFirst[cols, ColorDistance[#, White] > 0.001 && ColorDistance[#, skin] > 0.001 &]; FillingTransform[ SelectComponents[ Binarize[ImageAdjust[ColorDistance[image, haircolor]], {0, 0.1}], "Count", -1]]] ![Emoji Hairmask][4] Similarly, we can also extract jawlines and ear combinations, which could be potentially used for better fitting of face to emoji contours. It must be noted that this works on region based colour distance mapping, and hence doesn't necessarily return the required outputs. Some helper functions for the colour extraction of face features of emojis have also been created. ![Typical compilation sample][5] Data Processing ------- The next steps involved are to create chunks of associations labelling emoji segmentation masks with their names, and more importantly, generating a final list with unique hairstyles that conform to the heuristics involved in the generating process. This has been followed by a manual intervention over the remaining 423 images to remove edge cases such as beards, spectacles and other similar outlier combinations. The data processing steps and outputs can be viewed in the full project notebook provided in the links below. Mapping ======= Now that we have extracted all the required data for mapping, we utilise Wolfram Language's **ImageCrop** function to create a function to calculate the percentage overlap mapped over the curated unique emoji list and return the filename of the emoji with the best fitting skeleton. SelectFitter[Image_] := Module[{image = Image, nf, hairmask, emojifile, pics, l1, num}, hairmask = GetHairMaskPerson[image]; pics = ImageResize[#, {178, 218}] & /@ hairlist[[All, 2, 3]]; l1 = Divide[ Total[Flatten[ ImageData[ Abs[ImageCrop[#] - ImageResize[ImageCrop[Binarize[ImageAlign[#, hairmask]]], ImageCrop[#] // ImageDimensions]]]]], 2*Times @@ ImageDimensions@ImageCrop@#] & /@ pics; num = Position[l1, Min[l1]][[1]]; hairlist[[num, 1]]] ![Best Fit Emoji][6] And finally, we colorize this skeleton emoji appropriately! Emoji[Image_] := Module[{img = Image, eyecolperson, haircolperson, skincolperson, emojistd, skincolemoji, eyecolemoji, emojichanged, hairmask}, eyecolperson = PickIrisColorPerson[img]; haircolperson = PickHairColorPerson[img]; skincolperson = PickSkinColorPerson[img]; emojistd = Import[SelectFitter[img][[1]]]; skincolemoji = SkinColorEmoji[emojistd]; eyecolemoji = DominantColors[emojistd, Masking -> Polygon[{{280.0224466891134, 251.96408529741862`}, { 306.3973063973064, 253.64758698092032`}, {306.9584736251403, 228.39506172839504`}, {281.1447811447812, 226.71156004489336`}}]][[1]]; emojichanged = ColorReplace[ emojistd, {skincolemoji -> skincolperson, eyecolemoji -> eyecolperson}, {0.1, 0.1}]; hairmask = ExtractHairEmoji[emojistd]; emojichanged = ImageResize[ ImageApply[# - # + {haircolperson[[1]], haircolperson[[2]], haircolperson[[3]], 1} &, emojichanged, Masking -> hairmask], 300]] ![Emojify][7] Some more results... ![Emojify][8] ![Emojify][9] ![Emojify][10] A visual representation of the fitting process is as follows: ![Fitting procedure][11] Future Improvements ======= - Integration of some functions to map hairstyles and jawlines together, varying the hairstyle according to the jawline chosen. - Spectacle detection and improvement of the hair/face segmentation dataset for inclusion of a wider variety of hair. - Improving upon the **SelectFitter** function for a better metric of contour-shape similarity. - Curating a dataset with pre-decided anchor points of the graphics to manipulate the required semantic features, and clearer colouring. Relevant Links ======= - [Full project notebook][12] - [Cartoon Set Dataset][13] - [CelebA Face-Hair Segmentation Dataset][14] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=hairmaskpic.bmp&userId=1729371 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=colors-people.bmp&userId=1729371 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=emojis.bmp&userId=1729371 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=gethairmask.bmp&userId=1729371 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=jawlinemoji.bmp&userId=1729371 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=selectfitterpic.bmp&userId=1729371 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=emojify.bmp&userId=1729371 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=emojify2.bmp&userId=1729371 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=2.bmp&userId=1729371 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=3.bmp&userId=1729371 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=final.bmp&userId=1729371 [12]: http://ttps://github.com/kashikarparth/WSS-Template/blob/master/Final%20Project/Final%20Submission/FinalProjectNBTemplate.nb [13]: https://google.github.io/cartoonset/ [14]: http://www.cs.ubbcluj.ro/~dadi/face-hair-segm-database.html Parth Kashikar 2019-07-10T19:31:26Z https://community.wolfram.com/groups/-/m/t/1379517 ![enter image description here][1] #Rooftop Recognition for Solar Energy Potential The aim of this project is to detect the rooftop of buildings to determine the available area at different locations and to identify the most suitable ones for solar energy application such as solar PV using Neural Networks and satellite imagery. Github link for files and notebooks: https://github.com/enricocg/Project The Dataset =========== The Inria Aerial Image Labeling addresses a core topic in remote sensing: the automatic pixelwise labeling of aerial imagery. Dataset features: - Coverage of 810 km (405 km for training and 405 km for testing) - Aerial orthorectified color imagery with a spatial resolution of 0.3 m - Ground truth data for two semantic classes : building and not building (publicly disclosed only for the training subset) [https : // project.inria.fr/aerialimagelabeling/][2] Select the images for the input and those for the results: trainFilesInput = Select[FileNames[ "image/*.tif"], ! StringMatchQ[#, ___ ~~ "._" ~~ ___] &]; trainFilesResult = StringReplace[#, "image/" -> "mask/"] & /@ trainFilesInput; Import an image to test: imgInput = Import[trainFilesInput[[1]]]; imgResult = Import[trainFilesResult[[1]]]; Partition of the images in 100 from a 5000x5000 to images 500x500 : splicesInput = Join @@ ImagePartition[imgInput, 500]; splicesResults = Join @@ ImagePartition[imgResult, 500]; Assemble Images --------------- Assemble images in sets of ten to verify: assambleImage = ImageAssemble[splicesInput[[41 ;; 50]]] ![Image assembly for row 4][3] Assemble mask images in sets of ten to verify: assambleMask = ImageAssemble[ Image /@ Round[ImageData /@ splicesResults[[41 ;; 50]]]] ![Mask assembly for row 4][4] Compose ------- Compose the images to verify image matching: ImageCompose[assambleImage, {assambleMask, 0.5}] ![Image composition of row 4 with mask][5] rand = RandomInteger[{1, 100}]; ImageCompose[splicesInput[[rand]], {splicesResults[[rand]], 0.5}] ![Image composition of a random image][6] Association ----------- mxTrain = Thread[splicesInput -> splicesResults]; ImageCompose[Keys[mxTrain[[rand]]], {Values[mxTrain[[rand]]], 0.4}] ![Image association][7] Export the MX file Export["File.mx", mxTrain] Export ------ **mxFileCreator** The first approach to organize the data was to make MX files, one per image, each file contain the 100 images with their respective mask. In order to do that a function mxFileCreator was build. Function that creates a MX file per each 5000x5000 image: mxFileCreator[trainFilesInput_,trainFilesResult_,i_]:=Block[ {Flag,imgInput,imgResult,splicesInput,splicesResults,mxTrain}, Flag=TextString[i]; imgInput=Import[trainFilesInput]; imgResult=Import[trainFilesResult]; splicesInput=Join@@ImagePartition[imgInput,500]; splicesResults=Round[ImageData/@(Join@@ImagePartition[imgResult,500])]; mxTrain=Thread[splicesInput-> splicesResults]; Export["MXFiles/File"<>Flag<>".mx",mxTrain] ] Test the function mxFileCreator[trainFilesInput[[1]], trainFilesResult[[1]], 1] "MXFiles/File1.mx" file = Import["MXFiles/File1.mx"]; file[[RandomInteger[{1, 100}]]] ImageCompose[Keys[file[[rand]]], {Image[Values[file[[rand]]]], 0.5}] ![mxFileCreator test][8] To Map all the images and convert them into association in a MX files : MapIndexed[mxFileCreator[#[[1]], #[[2]], Echo[#2[[1]]]] &, Transpose[{trainFilesInput, trainFilesResult}]] The net ======= The net selected for this project was at Wolfram Neural Net Repository for Semantic Segmentation. Released in 2016 by the University of Adelaide, Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO Data was modify to identify two classes instead of 21. Take the net model for Semantic Segmentation from the repositories: netModel = NetModel["Ademxapp Model A1 Trained on PASCAL VOC2012 and MS-COCO \ Data"] ![Net model for Semantic Segmentation from the repositories:][9] Net surgery ----------- **Modify the Input for the size of the images** netModel500 = NetReplacePart[netModel, "Input" -> NetEncoder[{"Image", {500, 500}, "MeanImage" -> {0.485, 0.456, 0.406}}] ] ![Replace the encoder to accept images 500x500][10] **Resize the layer** Drop the last three layer to modify the net: firstPartNet = NetDrop[netModel500, -3] **Convolution layer** Add a convolution layer to have two outputs: convLayer=NetChain[{firstPartNet, ConvolutionLayer[2, {3, 3}, "Stride" -> {1, 1}, "PaddingSize" -> {12, 12}, "Dilation" -> {12, 12}], ResizeLayer[{500, 500}]}]; Take the last two layers from the one with the new encoder: lastPartNet = NetReplacePart[NetTake[netModel500, -2], "Input" -> Automatic] **SoftmaxLayer** Append the last layers and Softmax Layer with a net decoder of classes: finalNet = NetAppend[convLayer, {lastPartNet[[1]], SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", {0, 1}, "InputDepth" -> 3}]] **Initialize the net** Initialize the final net to check for errors: iniFinalNet = NetInitialize[finalNet] ![Final net][11] Net to train ------------ **Loss Function** Connect the final net to a loss function: LossNet = NetGraph[<|"eval" -> finalNet, "loss" -> CrossEntropyLossLayer["Index"]|>, {"eval" -> "loss"} ] Initialize the LossNet: iniLossNet = NetInitialize[LossNet] Export the complete net for training: Export["iniLossNet.wlnet", iniLossNet] Generator function ================== Reduce Dataset Size ------------------- From the MX files created in section. Set the file path for the MX files: path = "MXFiles/"; mxFiles = FileNames[path <> "*.mx"]; RandomSample[mxFiles, 1][[1]] **Binary Files** Binary files are a way to reduce of the size while importing the train set. Once the files are binarize the file size reduces and ones read in can be deserialize. At the same time TIF images were converted to JPG to remove unnecessary information while reducing size. Binary write: binaryWrite[file_,expr_]:=With[{bytes=BinaryWrite[file,BinarySerialize[expr]]}, Close[file]; bytes] Function to convert TIF images to JPG and matrix to Binary: importAndReExport[path_,folder_]:=Block[{imported,imgs,masks,hashes}, imported=Import[path]; imgs=imported[[All,1]]; masks=imported[[All,2]]; hashes=Hash/@imgs; Print[MapThread[Export[folder<>"/"<>ToString[#1]<>".jpg",#2]&,{hashes,imgs}];//AbsoluteTiming]; Print[MapThread[binaryWrite[folder<>"/"<>ToString[#1]<>".bin",#2]&,{hashes,masks}];//AbsoluteTiming]; Clear[imported,imgs,masks,hashes] ] Take each MX file and create the correspondent JPG and BIN file : Table[importAndReExport[mxFiles[[i]], "binFiles"], {i, 1, Length[mxFiles], 1}] **Binary Read** Read the binary file and Deserialize the file: BinaryDeserialize[ReadByteArray["/Users/enricocastro/Documents/GitHub/Project/binFiles/\74004748475675200.bin"]] // Dimensions Training Out of Core ==================== Partitional Function -------------------- **Data from MX files** Take the name of the JPG and BIN files: trainingDataSet = {FileNames["binFiles/*.jpg"], FileNames["binFiles/*.bin"]}; Set the data to train: imageDataSet = Table[File[trainingDataSet[[1, i]]], {i, 10}]; maskDataSet = Table[ReadByteArray[trainingDataSet[[2, i]]], {i, 10}]; Associate the image with his respective mask and take a random sample: data = RandomSample@Thread[imageDataSet -> maskDataSet]; data[[2]] Generator --------- In order to train the net with a big amount of data a generator function was created. The generator function load a single batch of data from an external source to train each time. NetTrain[net, f, \[Ellipsis]] calls f at each training batch iteration, thus only keeping a single batch of training data in memory. The function can depend on the batch size, which can be set or set automatic by the computer, the absolute batch that is the number of batches load during the training and the round. partGenerator associate and load a batch of size BatchSize into the net training. In order to do so it takes partitions of the whole dataset in batches of size BatchSize and load the element of this partition taking the module of the AbsoluteBatch. The function also adds one to the mask matrix to get the right values for the net: partGenerator=Function[Block[{batch,dataSet,batchData}, If[!ValueQ[partitionedData],partitionedData=Partition[Range[Length[data]],#BatchSize,#BatchSize,1]]; Print[Association@Thread[Keys[#]->Values[#]]]; batch=Mod[#AbsoluteBatch,Floor[Length@data/#BatchSize]]; batchData=data[[partitionedData[[Echo@(batch+1)]]]]; Thread[Keys[batchData]->1+BinaryDeserialize/@Values[batchData]] (*<|"Input"\[Rule]Keys[batchData],"Target"\[Rule]BinaryDeserialize/@Values[batchData]|>*) ] ]; **Test of partGenerator** partGeneratorData = partGenerator[<|"BatchSize" -> 2, "Round" -> 0, "AbsoluteBatch" -> 1|>] ![<|BatchSize->2,Round->0,AbsoluteBatch->1|>][12] Train test ---------- Before training for a lot of data is always good to try first with a small set. Clear partitionedData for different batch sizes: ClearAll[partitionedData]; iniLossNet = Import["iniLossNet.wlnet"] Set the checkpoint directory: checkpointDir = "checkpoint" Net training with partGenerator: iniLossTrainNet = NetTrain[iniLossNet, {partGenerator, "RoundLength" -> 3}, All, MaxTrainingRounds -> 3, TrainingProgressCheckpointing -> {"Directory", checkpointDir}] Trained Net =========== Once trained one can look at the relevant information of the training process. Import the trained net as an object: **Encoder & Decoder** Set the encoders and decoders for further tests. Set the encoder for image size 500x500 and a MeanImage: enc = NetEncoder[{"Image", {500, 500}, "MeanImage" -> {0.485, 0.456, 0.406}}]; Set the decoder for the classe which 0 means no rooftop and 1 means rooftop: dec = NetDecoder[{"Class", {0, 1}, "InputDepth" -> 3}]; **Training information** ![finalTrainedNet["LossEvolutionPlot"]][13] ![finalTrainedNet["ErrorRateEvolutionPlot"]][14] ![Properties][15] Extract the trained net: finalTrainedNetEval = NetExtract[finalTrainedNet["TrainedNet"], "eval"] Set the encoder: finalTrainedNetEvalEnc = NetReplacePart[finalTrainedNetEval, "Input" -> enc ] Set the decoder: finalTrainedNetEvalDec = NetReplacePart[finalTrainedNetEvalEnc, "Output" -> dec ] Export the net to test: Export["FinalTrainedNetEval.mx", finalTrainedNetEvalDec] Net Evaluation ============== netEvaluation = Import["/Volumes/ECG/ProjectWSS/AerialImageDataset/\ FinalTrainedNetEval.mx"] Test Files ---------- Take a random file from the training set: randomFile = RandomSample[FileNames["image/*.tif"], 1]; TestImage = Import[randomFile[[1]]]; TestMask = Import[StringReplace[randomFile[[1]], "image/" -> "mask/"]]; Take 100 partition of the image to feed into the net: imageToTest = ImagePartition[TestImage, {500, 500}]; maskToTest = ImagePartition[TestMask, {500, 500}]; rand1 = RandomInteger[{1, 10}]; rand2 = RandomInteger[{1, 10}]; ImageCompose[ imageToTest[[rand1, rand2]], {Image[netEvaluation[imageToTest[[rand1, rand2]]]], 0.5}] ![Image test][16] ImageCompose[ maskToTest[[rand1, rand2]], {Image[netEvaluation[imageToTest[[rand1, rand2]]]], 0.5}] ![Mask comparision][17] Compare the percentage of building in the area with the net and the mask: {{"Net", "Mask"}, {ImageMeasurements[ Image[netEvaluation[imageToTest[[rand1, rand2]]]], "Mean"], ImageMeasurements[maskToTest[[rand1, rand2]], "Mean"]}} // Dataset ![Percentage comparision][18] GeoImage -------- testImage[latitud_, longitud_, net_] := Module[{geoImage, geoImageResize}, geoImage = GeoImage[GeoPosition[{latitud, longitud}], GeoRange -> Quantity[75, "Meters"], GeoProjection -> "Mercator", ImageSize -> Small]; geoImageResize = ImageResize[geoImage, {500, 500}]; {ImageCompose[geoImageResize, {Image[net[geoImageResize]], 0.3}] , Image[net[geoImageResize]]}] **IER-UNAM** IER = testImage[18.839940, -99.235635, netEvaluation] ![IER-UNAM, Temixco, Morelos][19] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ImageCollage.jpeg&userId=1363133 [2]: https://project.inria.fr/aerialimagelabeling/ [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=A1.jpeg&userId=1363133 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=A2.jpeg&userId=1363133 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=C1.jpeg&userId=1363133 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=C2.jpeg&userId=1363133 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Association.jpeg&userId=1363133 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=mxFileTest.jpeg&userId=1363133 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=netModel.jpeg&userId=1363133 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=netModel500.jpeg&userId=1363133 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=finalNet.jpeg&userId=1363133 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=partGenerator.jpeg&userId=1363133 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=FinalLoss.jpeg&userId=1363133 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ErrorRate.jpeg&userId=1363133 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Properties.jpeg&userId=1363133 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Test1.jpeg&userId=1363133 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MaskTest.jpeg&userId=1363133 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Percentage.jpeg&userId=1363133 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=IER.jpeg&userId=1363133 Enrico Castro 2018-07-11T19:56:30Z https://community.wolfram.com/groups/-/m/t/1379392 #Summary Detailed digital information about water is of great important not only for urban design and development, but also for [geopolitical strategic plans][1]. A large portion of world population is now living in [water stress regions][2]. There are alarming patterns between the water crisis and political/civil unrest in parts of the world (e.g., Yemen, [Syria][3], even [Iran][4]). Therefore, having digital model of water sources is essential. My main goal here is to extract the hydrographic basin (also called, the drainage basin) out of digital elevation models (DEM). Detailed watershed segmentation of a geographical region is obtained using built-in morphological functions of Wolfram language that I applied on the DEM of the region. The watershed lines dividing neighboring basins, together with their elevation values, are extracted. We treated the centroid of sub-basins as vertexes of a graph where connections between neighboring vertexes depend on the land surface characteristics and boundary lines. We assumed that a valid connection between two vertexes (say, two sub-basins) can be made only if the two vertexes have minimum elevation at the boundary compared to all neighboring vertexes. Our results show a great match with the the watershed segmentation obtained manually by appropriate choice of markers (say, using a seasoned geographer knowledge). # DEM of Lake Urmia (Orumiyeh, Persian: ??????) The region that I choose for my analysis is [lake Urmia][5] on the north west of Iran. It is a UNESCO biosphere reserve which is now dangerously shrinking due to [water mismanagement and also drought][6]. We set the region a box between two points with altitudes and longitudes {35.5,44} and {39,48}. We extract the DEM using GeoElevationData function. region = {{35.5, 44}, {39, 48}}; data = N[QuantityMagnitude@ GeoElevationData[region, UnitSystem -> "Metric"]]; reliefmap = ReliefPlot[data, DataReversed -> True, PlotLegends -> BarLegend[Automatic, LegendLabel -> "elevation(m)"]] ![enter image description here][7] # Watershed segmentation We covert the DEM data into an image. mimg = ImageAdjust[Image[data]]; From now on, we shall treat our DEM as an image with colors corresponds to elevations. We find watershed segmentation (i.e., a detailed sub-basin structure) using Immersion method, mainly because the labeling of segments correspond to the relative value for a sub-basin's minimum elevation. wsc = WatershedComponents[mimg, Method -> "Immersion"]; bnds = ColorNegate@Image[wsc, "Bit"]; ![enter image description here][8] Next, we label the boundary between two adjacent sub-basins by the indexes of corresponding sub-basins. To do so, we use black-and-white image of watersheds where the boundaries (say, lines) have value one (white lines). Then, we assign two indexes to the line separating two sub-basins. These indexes are the indexes of adjacent sub-basins separated by the line. We shall do index assignment of lines in two steps: 1. Shift the sub-basin with higher index into the lines (using Erosion). bindex = Erosion[Replace[wsc, 0 -> Max[wsc] + 1, {2}], 1]; 2. Shift the sub-basin with lower index into the lines (using Dilation) tindex = Dilation[wsc, 1]; The first step has one subtlety: to shift the higher index into the lines, first we need to set the boundary value higher than any value in the image; otherwise the Erosion function will extend the boundaries only. doubleIndexArray = Replace[Transpose[{bindex, tindex}, {3, 1, 2}], {n_, n_} -> {0, 0}, {2}]; doubleIndexPairs = Prepend[DeleteCases[ DeleteDuplicates[Flatten[doubleIndexArray, 1]], {0, 0}], {0, 0}] After labeling the boundaries by two indexes, we find the minimum elevation along the boundaries separating two sub-basins. Now we have the information of adjacent sub-basins and the minimum elevation along the boundaries. bndSegs = Replace[doubleIndexArray, Dispatch@MapIndexed[#1 -> First[#2] - 1 &, doubleIndexPairs], {2}]; MinATborders = Thread[Rest[doubleIndexPairs] -> ComponentMeasurements[{mimg, bndSegs}, "Min"][[All, 2]]] # Graph made out of centroids of sub-basins From now on, we shall treat a sub-basin as a vertex. For the sake of representation, the location of the vertex is set as the centroids of sub-basins. The most important question is how the vertexes are connected together and what are the corresponding clusters made out of connected vertexes. In other word, the clusters represent larger basins that form a unique hydro-graphical ecosystem with a unique water graph. g0=Graph[Apply[UndirectedEdge,Rest@doubleIndexPairs,{1}]]; g1=WeaklyConnectedGraphComponents[g0][[1]]; basinConnect=Sort[Map[# -> AdjacencyList[g1,#]&,VertexList[g1]]] sortBasin[x_]:=Keys[x]->SortBy[Values[x],Association[MinATborders][Sort[{#,Keys[x]}]]&]; sortedBasins=sortBasin[#]&/@basinConnect ##Algorithm to make connections between sub-basins (how/where to connect vertexes and form edges) 1. Built a set for each sub-basin whose elements are the adjacent sub-basins. 2. Sort them by the minimum value of the boundary separating them. 3. Make a connection between two adjacent sub-basin by choosing the first element of the sorted list in step#2 4. Check for possible cycles in the graph (i.e., loops). If any, replace the original connection by the making the connection with the second element in the list of sorted sub-basins obtained in step#2. 5. Repeat step#5 until there is no more change in the graph Here is the graph that we obtain following above steps: ![enter image description here][9] # Comparison of our result with a basin structure obtained by pre-determined markers Let us assume that we are a seasoned geographer with a good knowledge of large basins (e.g., where the rivers are running and what branches of a river are independent and where they merge) and how they are connected on a map. Then, one shall obtain the following basins for our selected region: ![enter image description here][10] Now, let us overlay our graph on the top of map and compare different results ![enter image description here][11] # Concluding remarks Detailed watershed segmentation of a geographical region is obtained using built-in morphological functions of Wolfram language that we applied on the raw Digital Elevation Model (DEM) of the region. The watershed lines dividing neighboring basins, together with their elevation values, are extracted from DEM. We treated the centroid of sub-basins as vertexes of a graph where connections between neighboring vertexes depend on the land surface characteristics and boundary lines. We assumed that a valid connection between two vertexes (say, two sub-basins) can be made only if the two vertexes have minimum elevation at the boundary compared to all neighboring vertexes. Our results show a great match with the the watershed segmentation obtained manually by appropriate choice of markers (say, using a seasoned geographer knowledge). #Future works With the detailed dynamic relationships of hydrographic basins in force, we will be able to develop models describing the water flow (or water level) among basins using the precipitation data. Next, the water usage for irrigation and urban uses can be added into the model. Using these information, one will be able to develop an elaborate model for water management of a large hydrographic basin. Our approach can be improved further by treating a few loops (cycles) that appear in the graph. In this regard, one can remove the connection between two vertexes where the initial one has higher elevation and replace it with the second vertex in the list of neighboring basins sorted by the minimum boundary elevation. [1]: https://geopoliticalfutures.com/water-geopolitical-imperatives/ [2]: https://www.theguardian.com/cities/2016/jul/29/where-world-most-water-stressed-cities-drought [3]: http://www.pnas.org/content/112/11/3241 [4]: https://www.nytimes.com/2018/01/18/climate/water-iran.html [5]: https://www.theguardian.com/world/gallery/2016/sep/02/iran-lake-urmia-in-pictures [6]: https://link.springer.com/article/10.1007/s11269-011-9909-8 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1665ReliefMap.png&userId=1351917 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=watershed.png&userId=1351917 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=graph.png&userId=1351917 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=geo-basin.png&userId=1351917 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=overlay.png&userId=1351917 Mads Bahrami 2018-07-11T19:47:32Z https://community.wolfram.com/groups/-/m/t/1137218 The goal of OCR (Optical Character Recognition) is to recognize characters in images. Here is how I used convolutional neural network to create an OCR working on images of single characters that supports 8 major writing systems: Arabic, Chinese, Cyrillic, Devanagari, Greek, Japanese, Korean, and Latin. ## Neural Network ## I developed my neural network based on LeNet by adding batch normalization layers, dropout layers, and more convolutional layers. Here is the architecture of my network: OUCR = NetChain[{ ConvolutionLayer[32, {5, 5}, "PaddingSize" -> {2, 2}, "Stride" -> 2], BatchNormalizationLayer["Input" -> {32, 32, 32}], ElementwiseLayer[Ramp], PoolingLayer[{2, 2}, {2, 2}], ConvolutionLayer[64, {3, 3}, "PaddingSize" -> {1, 1}], BatchNormalizationLayer["Input" -> {64, 16, 16}], ElementwiseLayer[Ramp], PoolingLayer[{2, 2}, {2, 2}], ConvolutionLayer[128, {3, 3}, "PaddingSize" -> {1, 1}], BatchNormalizationLayer["Input" -> {128, 8, 8}], ElementwiseLayer[Ramp], PoolingLayer[{2, 2}, {2, 2}], ConvolutionLayer[256, {3, 3}, "PaddingSize" -> {1, 1}], BatchNormalizationLayer["Input" -> {256, 4, 4}], ElementwiseLayer[Ramp], PoolingLayer[{2, 2}, {2, 2}], FlattenLayer[], DropoutLayer[0.5], LinearLayer[], SoftmaxLayer[] }, "Output" -> NetDecoder[{"Class", unicode}], "Input" -> NetEncoder[{"Image", {64, 64}, "Grayscale", "MeanImage" -> 0.85}] ] The first convolutional layer has kernels of size 5 and stride 2 to reduce the influence of noise. The fully connected layers are very simple due to the large volume (over 32000) of classes. ## Training Set ## To train my neural network, I generated over a million images of characters with different fonts and rotations using Mathematica. At first I used Rasterize[], which turned out to be so slow that it would take tens of hours to generate all the images. So I improved my algorithm by using Image[], Graphics[], and Text[] instead, which took only a few minutes. Here are the two functions I designed to generate images: unrotated[list_, size_, scale_, font_, horizontal_, vertical_] := Module[{len, col, row}, len = Length[list]; col = Floor[Sqrt[len]]; row = Ceiling[len / col]; Thread[(Join @@ ImagePartition[ Image[ Graphics[ MapIndexed[ Text[ Style[#1, FontFamily -> font, FontSize -> Scaled[scale / col]], Reverse[#2]] &, Reverse[ Partition[FromCharacterCode /@ list, UpTo[col]] ],{2}], PlotRange -> {{horizontal, col + horizontal}, {vertical, row + vertical}}, ImageSize -> {size * col, size * row}], ColorSpace -> "Grayscale"], size])[[;; len]] -> FromCharacterCode /@ list]] rotated[list_, size_, scale_, font_, angle_, horizontal_, vertical_] := Module[{len, col, row}, len = Length[list]; col = Floor[Sqrt[len]]; row = Ceiling[len / col]; Thread[(Join @@ ImagePartition[ Image[ Graphics[ MapIndexed[ Rotate[ Text[ Style[#1, FontFamily -> font, FontSize -> Scaled[scale / col]], Reverse@#2], RandomVariate[ NormalDistribution[0, angle]] Degree] &, Reverse[ Partition[FromCharacterCode /@ list, UpTo[col]] ], {2}], PlotRange -> {{horizontal, col + horizontal}, {vertical, row + vertical}}, ImageSize -> {size * col, size * row}], ColorSpace -> "Grayscale"], size])[[;; len]] -> FromCharacterCode /@ list]] These functions take a list of Unicodes and generate images of unrotated and rotated characters of the corresponding Unicodes. There are several tunable parameters. "size" controls the size of the images (size * size), where I used 64 for my training set. "scale" controls the scale of characters in images, where I found 0.7~0.8 suitable for most writing systems and fonts. For "font", I chose several fonts for each language to make the characters as diverse as possible. "horizontal" and "vertical" are compensations, where 0.5 works for most cases. rotated[] also takes "angle" and generates rotated images whose rotational angles follow a normal distribution with standard deviation of "angle" degrees, where I used 4 degrees in most cases. For each font of each writing system, I generated 1 unrotated and 3 rotated images for the training set and 1 rotated image for the validation set. There was no obvious overfitting or underfitting during the training process, and after 70 rounds of training, the validation loss was down to 5*10^-3. ## Tests ## Here is the bar chart of accuracies recognizing characters of each writing system, tested on random samples with random fonts and random rotations: ![enter image description here][1] The accuracies recognizing Cyrillic, Greek, and Latin characters are rather low because some of their characters are very similar. The accuracy recognizing Latin characters is especially low because I tested on a random sample of all fonts, and Latin characters can look very different in different fonts, while many fonts don't support other writing systems. Here is the bar chart of accuracies recognizing a random sample of 200 Chinese characters, tested on characters generated by the computer (different fonts), handwritten by Yan, Zhenqing (709-785, one of the best calligraphers in Chinese history, images from http://www.shufazidian.com/), and handwritten by myself (written on paper, scanned, and processed using Mathematica): ![enter image description here][2] Two short examples of how it works (first handwritten by Yan, second handwritten by myself): ![enter image description here][3] I also tested my network on many variations of the images, including blurring, adding noise, distortion, zooming in and out, horizontal and vertical movement, and rotation: ![enter image description here][4] ![enter image description here][5] ![enter image description here][6] ![enter image description here][7] ![enter image description here][8] ![enter image description here][9] ![enter image description here][10] ## Acknowledgements ## I would like to thank my mentor Vladimir Grankovsky for helping me along the whole project and Matteo Salvarezza and Timothée Verdier for helping me set up AWS GPU computing services. ## References ## 1. Image source for Yan's handwritten Chinese characters: http://www.shufazidian.com/ 2. My GitHub repository: https://github.com/MatthewChen7211/WSS2017 [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=6830WS.png&userId=1123149 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1449HW.png&userId=1123149 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-07-05at5.21.25PM.png&userId=1123149 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Blur.png&userId=1123149 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Noise.png&userId=1123149 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Distortion.png&userId=1123149 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Zoom.png&userId=1123149 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Horizontal.png&userId=1123149 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Vertical.png&userId=1123149 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Rotate.png&userId=1123149 Matthew Chen 2017-07-05T21:51:05Z https://community.wolfram.com/groups/-/m/t/3217332 ![Test Accuracy][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=CoverImage.png&userId=3216554 [2]: https://www.wolframcloud.com/obj/7e50ca1b-23b7-43e9-a8dc-0b01f627756e Junseo Lee 2024-07-11T22:58:30Z https://community.wolfram.com/groups/-/m/t/1784061 # Testing the concept: To test and calibrate the method, I used an example (*not from a moon*): Olympus Mons on Mars, and highlighted areas of the image that stand out as bright white areas, enhanced by code and subsequently inverting the color: olympus = GeoGraphics[Entity["SolarSystemFeature", "OlympusMonsMars"], GeoRange -> Quantity[200, "Miles"]] ![pic1][1] ColorNegate[ ChanVeseBinarize[ ColorConvert[ImageAdjust[olympus, {0, 0.8, 0.8}, {0.5, 1}, {0, 1}], "Grayscale"], Gray]] ![pic2][2] After dividing the image (above) into many small regions (5810 smaller images), I characterize each piece with its values (ImageData) using the following line: cd = Parallelize[ Table[Table[ N[Mean@Flatten@ImageData[((Take[pat, {vv}][[1]])[[uu]])]], {uu, 1, 83}], {vv, 1, 70}]]; Then I made *Mathematica* choose regions with average data values with a specific degree of pixels to generate red dots that will be used to mark the regions (it was noted that the method has limitations caused mainly by the size of the smaller regions, requiring a lot of memory and processing, but positively had a relative accuracy): ur = ImageAssemble[ Parallelize[ Table[Table[ If[((cd[[v]])[[u]]) < 0.85, ImageCompose[((pat[[v]])[[ u]]), {Graphics[{PointSize[Large], Red, Point[{0, 0}]}]}], ImageCompose[((pat[[v]])[[ u]]), {Graphics[{PointSize[Tiny], Opacity[0], Point[{0, 0}]}]}]], {u, 1, 83}], {v, 1, 70}]]] ![pic3][3] ColorReplace[ur, Black] ![pic4][4] Now testing with the original Olympus Mons image (with 5810 divisions and marking bright areas for: brightness > 0.67), the code for partition the image and selecting data values to mark light areas: ![pic5][5] cd3 = Parallelize[ Table[Table[ N[Mean@Flatten@ImageData[((Take[pat3, {vv}][[1]])[[uu]])]], {uu, 1, 83}], {vv, 1, 70}]]; In the case of the actual image, the average pixel value must be above a specific value for selecting the required areas, the reverse of the concept test. Here is the test done with the raw image: ur5 = ImageAssemble[ Parallelize[ Table[Table[ If[((cd3[[v]])[[u]]) > 0.67, ImageCompose[((pat3[[v]])[[ u]]), {Graphics[{PointSize[Large], Red, Point[{0, 0}]}]}], ImageCompose[((pat3[[v]])[[ u]]), {Graphics[{PointSize[Tiny], Opacity[0], Point[{0, 0}]}]}]], {u, 1, 83}], {v, 1, 70}]]] ![pic6][6] # Testing surface maps of moons: Now testing the surface map of some of the solar system's moons, so we can see the previous code performance and later some variations of the same for better visualization. Below are the other examples of surface maps tested by the code, with the limit of values of the bright and the number of smaller computed areas of each image chosen for each example. These are the basic lines of code that will be used on the surfaces: (1) Calculates the values of each piece of the image, same code to all examples (note: set the image on the code!): x = ImagePartition["image", {Scaled[1/70]}]; y = Parallelize[ Table[Table[ N[Mean@Flatten@ImageData[((Take[x, {vv}][[1]])[[uu]])]], {uu, 1, 70}], {vv, 1, 70}]]; (2) Generates the points and reassembles the full image. Has some variations as seen in the examples later in the post (note: choose the brightness limit value on the code!): z = ImageAssemble[ Parallelize[ Table[Table[ If[((y[[v]])[[u]]) > "brightvalue", ImageCompose[((x[[v]])[[ u]]), {Graphics[{PointSize[Large], Red, Point[{0, 0}]}]}], ImageCompose[((x[[v]])[[ u]]), {Graphics[{PointSize[Tiny], Opacity[0], Point[{0, 0}]}]}]], {u, 1, 70}], {v, 1, 70}]]] Some examples: - **Jupiter´s Moon, Ganymede**: GeoGraphics[GeoModel -> Entity["PlanetaryMoon", "Ganymede"], GeoRange -> Quantity[1500, "Miles"]] ![pic7][7] Using the codes gives the following result for Ganymede (marking for brightness > 0.54): ![pic8][8] - **Jupiter´s Moon, Europa** (a bright moon): GeoGraphics[GeoModel -> Entity["PlanetaryMoon", "Europa"], GeoRange -> Quantity[1000, "Miles"]] ![pic9][9] Using the codes gives the following result for Europa, which has a brighter surface (marking for brightness > 0.71): ![pic10][10] - **Jupiter´s Moon, Callisto** (a dark moon): GeoGraphics[GeoModel -> Entity["PlanetaryMoon", "Callisto"], GeoRange -> Quantity[1500, "Miles"]] ![pic11][11] Using the codes, we get the following result for Callisto, which has a relatively dark surface (marking for brightness > 0.27): ![pic12][12] # Exploring the image with dynamics: - **Saturn´s Moon, Dione**: GeoGraphics[GeoModel -> Entity["PlanetaryMoon", "Dione"], GeoRange -> Quantity[400, "Miles"]] ![pic13][13] Using the first (1) code and making the second (2) code dynamic (below), I got the following result for Dione, a surface with well-distributed bright spots (marking for brightness > 0.55): {Evaluate[ urm = ImageAssemble[ Parallelize[ Table[Table[ If[((cdm[[v]])[[u]]) > 0.55, ImageCompose[((patm[[v]])[[ u]]), {Graphics[{PointSize[Large], Red, Point[{0, 0}]}]}], ImageCompose[((patm[[v]])[[ u]]), {Graphics[{PointSize[Tiny], Opacity[0], Point[{0, 0}]}]}]], {u, 1, 70}], {v, 1, 70}]]]], DynamicImage[urm]} ![pic14][14] # Marking light and dark areas simultaneously: - **Earth´s Moon**: GeoImage[GeoRange -> {{-25, -5}, {45, 65}}, GeoModel -> "Moon"] ![pic15][15] Using the first (1) code and modifying the second (2) code as below, we can mark the darkest and lightest areas on the map simultaneously. Getting the following result for Earth's Moon (marking for brightness: > 0.68 {Red}, < 0.61 {Orange}): urm2 = ImageAssemble[ Parallelize[ Table[Table[ If[((cdm[[v]])[[u]]) > 0.68, ImageCompose[((patm[[v]])[[ u]]), {Graphics[{PointSize[Large], Red, Point[{0, 0}]}]}], If[((cdm[[v]])[[u]]) < 0.61, ImageCompose[((patm[[v]])[[ u]]), {Graphics[{PointSize[Large], Orange, Point[{0, 0}]}]}], ImageCompose[((patm[[v]])[[ u]]), {Graphics[{PointSize[Tiny], Opacity[0], Point[{0, 0}]}]}]]], {u, 1, 76}], {v, 1, 80}]]] ![pic16][16] # Viewing different intensities simultaneously: - **Jupiter´s Moon, Io** (a colorful moon): GeoGraphics[GeoModel -> Entity["PlanetaryMoon", "Io"], GeoRange -> Quantity[1000, "Miles"]] ![pic17][17] Modifying the second (2) code so that we have points with different intensities throughout the image and make them stand out with a convenient Hue function: urm2 = ImageAssemble[ Parallelize[ Table[Table[ ImageCompose[((patm[[v]])[[ u]]), {Graphics[{PointSize[Large], Hue[(150 - ((cdm[[v]])[[u]])*149)/50], Point[{0, 0}]}]}], {u, 1, 70}], {v, 1, 70}]]] ![pic18][18] And finally, modifying again the second code (2), making the points show variations of bright with another Hue function, while dynamically exploring and labeling the intensities: {Evaluate[ urm = ImageAssemble[ Parallelize[ Table[Table[ ImageCompose[((patm[[v]])[[ u]]), {Graphics[{PointSize[Large], Hue[((cdm[[v]])[[u]])^4], Point[{0, 0}]}]}], {u, 1, 70}], {v, 1, 70}]]]], DynamicImage[urm], TableForm[ Table[{N[(h/30), 2], Graphics[{PointSize[Large], Hue[(h/30)^4], Point[{0, 0}]}, ImageSize -> 10]}, {h, 9, 29}], TableHeadings -> {None, {"bright", "point"}}]} ![pic19][19] Thanks. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic1.png&userId=1316061 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic2.png&userId=1316061 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic3.png&userId=1316061 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic4.png&userId=1316061 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic5.png&userId=1316061 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic6.png&userId=1316061 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic7.png&userId=1316061 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic8.png&userId=1316061 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic9.png&userId=1316061 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic10.png&userId=1316061 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic11.png&userId=1316061 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic12.png&userId=1316061 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic13.png&userId=1316061 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic14.png&userId=1316061 [15]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic15.png&userId=1316061 [16]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic16.png&userId=1316061 [17]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic17.png&userId=1316061 [18]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic18.png&userId=1316061 [19]: https://community.wolfram.com//c/portal/getImageAttachment?filename=pic19.png&userId=1316061 Claudio Chaib 2019-09-05T03:33:44Z https://community.wolfram.com/groups/-/m/t/1731520 #Introduction Emojis are ideograms used in electronic texts and web pages that were first introduced by Japanese artist Shigetaka Kurita around the late 1900s. Since emojis convey facial expressions and common objects like earth, weather, and food relatively well, they became increasingly popular. Face emojis are everywhere these days because they come in handy when we are trying to communicate with people more comfortably. One can decorate their Instagram, Snapchat or other social media posts with emoji stickers. The Basic idea of my project is to find what face emojis are used in an image. The project aims to train a classifier to learn various types of face emojis and identify them in an image. For example, a person might add a face emoji on his or her image in order to decorate it. When the image is processed, the program should be able to find the face emojis he or she used in the image. After locating them, the program should return what the face emoji is, such as, ‘Crying face’ or ‘Sleepy face.’ Classifying the emojis is accomplished via the Classify function where the training data is an association of ‘emoji-names’ and emoji-images from Apple, Google, and Twitter. Locating the emojis is accomplished via SelectComponents and a masking workflow using ColorDistance and Filling Transform. #Running the Face Emoji Classifier In order to identify the emojis in the sub-images, I had to make a solid custom face emoji classifier. Since I had issues with web-scraping face emoji images, I had to manually. I used 3 different images from Apple, Google,and Twitter, for each type of face emoji. **Small Part of the Dataset, 'faceEmojiNames'** ![part of the dataset:faceEmojiNames][1] **Small Part of the Dataset, 'faceEmojiLists'** ![part of the dataset:faceEmojiLists][2] Using that dataset, I trained the **classifier** on different face emoji images with Logistic Regression and the "ImageFeatures" FeatureExtractor. ``` faceEmojiAT = AssociationThread[faceEmojiNames -> faceEmojiLists] ``` ``` emojiClassification = Classify[faceEmojiAT, FeatureExtractor -> "ImageFeatures"] ``` ![emojiClassification][3] ![example results][4] #Finding Face Emojis in an Image ![general process][5] Given that most of the face emojis have a shape of a circle and a dominant color of yellow, I ran **ColorDistance** with yellow on the example image which returned a gray-scale image where darker shades are closer to the target color, yellow. **Example Image** ![example image][6] ``` stepOne = ColorDistance[exampleImage, Yellow] ``` ![output for stepOne][7] I, then, binarized the gray scale image from ColorDistance using the function **Binarize**. It created a binary image with black and white where black is below the set threshold and white is above. ``` stepTwo = Binarize[stepOne, {0, 0.3} ``` ![output of stepTwo][8] After taking the step of Binarize, I used **FillingTransform**, simplifying the features of the image by filling the entire circle with white. This gave me a completely black image with several white circles detected during the process. ``` stepThree = FillingTransform[stepTwo] ``` ![output for stepThree][9] Next, I used **SelectComponents** to find features with circularity greater than .9 in the example image and applied **ComponentMeasurements** which returned the position of those circles. ``` stepFour = SelectComponents[stepThree, #Circularity > .9 &] ``` ![output for stepFour][10] ``` stepFive = ComponentMeasurements[stepFour, {"Centroid", "EquivalentDiskRadius"}, All, "ComponentPropertyAssociation"] ``` ![output for stepFive][11] Later, **Disk** function became useful. It created a disk out of centroid and equivalent disk radius properties, helping me to find the exact coordinates of the circles’ positions. ``` stepSix = Disk[#Centroid, #EquivalentDiskRadius] & /@ stepFive ``` ![output for stepSix][12] In order to return the corners of the subimages that contain circles from these coordinate values, I wrote a function called **emojiC**. Using emojiC, I picked up the coordinate values of the corners of the sub-images and used ImageTrim to trim the input image using the coordinates. ``` emojiC[{{x_, y_}, r_}] := {{x - r, y - r}, {x - r, y + r}, {x + r, y - r}, {x + r, y + r}} stepSeven = emojiC[Extract[Part[Values[stepSix], #], {{1}, {2}}]] & /@ Range[1, Length[Values[stepSix]]] ``` ![output for stepSeven][13] ``` stepEight = ImageTrim[exampleImage, #] & /@ stepSeven ``` ![output for stepEight][14] The issue of this process was that there were some sub-images of a circle that don’t contain face emojis. Given that in most cases the sub-image containing the emoji will have the largest dimensions, I went through the area of each sub-images and drop those that are smaller than 1% of the largest sub-image using **Select**. ``` areasOfPixels[{x_, y_}] := (x*y) stepNine = Select[stepEight, (areasOfPixels[ImageDimensions[#]]) > 0.01 (Last[ Sort[Map[areasOfPixels, Map[ImageDimensions, stepEight]]]]) &] ``` ![output for stepNine][15] Lastly, I applied custom **face emoji classifier** trained on different face emoji images with a method of Image Feature Extractor & Logistic Regression, and ran on the subimages containing the face emojis. Those face emojis with different colors such as red, green, blue, and purple, will also be easily detected by using their dominant colors. ``` stepTen = emojiClassification[stepNine] ``` ![output for stepTen, the final result][16] #Extension I took these face emojis with yellowish color as an example, but those face emojis with different colors such as red, green, blue, and purple, will also be easily detected by using their own dominant colors. **Example of Face Emojis with Different Colors** ![example of face emojis with different colors][17] Since I detected the face emojis on the images using the circular shape of those, I experienced difficulty in creating masks for those face emojis that are not circular. In order to expand on my work, I have to find out a way to detect those face emojis with external features, such as horns, glasses, and hearts. I believe this concept would be achieved by refining or finding the properties and values for the function SelectComponents instead of circularity. [Project Materials & Computational Essay][18] Mentor: [Faizon Zaman][19] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at11.00.46AM.png&userId=1579063 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.48.37AM.png&userId=1579063 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.53.29AM.png&userId=1579063 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.54.32AM.png&userId=1579063 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-10at3.07.14PM.png&userId=1579063 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.21.11AM.png&userId=1579063 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.36.11AM.png&userId=1579063 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.31.32AM.png&userId=1579063 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.31.43AM.png&userId=1579063 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.31.54AM.png&userId=1579063 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.32.05AM.png&userId=1579063 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.32.10AM.png&userId=1579063 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.32.15AM.png&userId=1579063 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.32.21AM.png&userId=1579063 [15]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.32.29AM.png&userId=1579063 [16]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.33.07AM.png&userId=1579063 [17]: https://community.wolfram.com//c/portal/getImageAttachment?filename=%E1%84%8B%E1%85%B5%E1%84%80%E1%85%A5.png&userId=1579063 [18]: https://github.com/SeoyoonHan/WSS-Template [19]: https://www.linkedin.com/in/faizonzaman Seoyoon (Jade) Han 2019-07-11T16:07:03Z