https://community.wolfram.com RSS Feed for Wolfram Community showing any discussions tagged with Image Processing with no replies sorted by most viewed. https://community.wolfram.com/groups/-/m/t/1383429 ![*Add Cool Image][1] #Microsite# https://one.wolframcloud.com/objects/rohit.panse8/pneumoniadiagnosis #Introduction# Pneumonia is an infection that causes inflammation in one or both of the lungs; it is induced by a variety of organisms: bacteria, viruses, and fungus. Bacterial pneumonia, the more common type, is caused by bacteria that multiplies in the lungs; Viral Pneumonia is caused by an array of viruses, the most common one being Influenza. Although the severity of pneumonia can vary, young children, seniors, and people with a weakened immune system are the most vulnerable.The CDC reports that of the close to 540,000 cases of pneumonia each year, 50,000 people die. Early and proper diagnosis can tremendously decrease mortality rate. Conventional diagnosis methods like examining patient medical history and symptoms have their shortcomings. Dutch researchers, who published their findings in the European Respiratory Journal, found that of 140 patients who had their pneumonia diagnosed by x-ray, doctors initially thought only 41 of them had the severe lung infection. Chest X-rays have long been reliable diagnostic tools for pneumonia. Since pneumonia, is an airspace disease, patients' air spaces are filled with bacteria, viruses, pus, and other microorganisms. A diagnosis is determined by infiltrates, or white spots, present in a chest X-ray. My project uses a convolutional neural network to diagnose the type of pneumonia that a patient has and. Thank you to Daniel Kermany, Daniel Zhang, and Michael Goldbaum for creating and labeling the dataset. Thanks to Paul Timothy Mooney for making the dataset available on Kaggle. <br /> <br /> Below is an example of an infiltrate present in a chest X-ray. ![enter image description here][2] <br /> Below is an example of how chest X-rays of the three classifications generally look. Chest X-rays of pneumonia patients are characterized by loss of diaphramatic shadows, and evidence of pulmonary consolidation(an area of the lung that has filled with liquid instead of air). ![enter image description here][3] #Diagnosing pneumonia# ##Neural Network I used transfer learning to create my neural network. Transfer learning is the process of using a pre-trained model as a starting point as a model for a second task. > "Transfer learning and domain adaptation refer to the situation where > what has been learned in one setting \[Ellipsis] is exploited to > improve generalization in another setting"-(Page 526, Deep Learning, > 2016) I used ResNet 50, an architecture released in 2015 by Microsoft Research Asia; this network was originally trained on 1.2 million images with 1000 classes. The immense computing power, time resources, and knowledge dedicated to perfect this network made it a great neural network to use for transfer learning. <br /> <br /> Below is an example of the ResNet 50 in action. It correctly predicts that the image is a golden retriever. In addition, I set the program to output the top 10 probabilities of what it thought the image was. ![enter image description here][4] There are two notable layers in this network: linear layer and softmax layer. First, the Linear Layer maps the features processed by the other layers to the classifications given. Classifications(classes) are the different objects the neural network is trained to recognize. In the standard ResNet's case the classes are the 1000 different objects that it is trained to recognize. In the case of my project, the classes are a normal, bacterial pneumonia, and viral pneumonia chest X-rays. Second, the Softmax layer determines the probabilities of what the image is. Above the probabilities of the image are displayed; the class with the highest probabilities is what the Neural Net outputs. <br /> <br /> Below, I remove the Linear and the Softmax layer. These layers are currently configured to recognize 1000 classes; I only need them to recognize 3 classes of images(normal, bacterial pneumonia, viral pneumonia). I will add these two layers later on. ![enter image description here][5] <br /> Below, I replaced the "Input" layer to recieve an input of an image of dimensions 512 by 512. I will configure my training images to be of this size. ![enter image description here][6] <br /> I added 3 layers to a new neural net. The above resizedNet will extract features from the dataset. This new neural net that I created will be used to interpret the features and output what it thinks the input image was. The 3 layers that will do this are the "DropoutLayer", "LinearLayer", and "SoftmaxLayer". The "DropoutLayer" is used to reduce overfitting; this is the tendency of a neural network to become extremely specialized to a limited set of data points. Overfitting is not desirable because it makes the neural network very ineffective on data points outside of the training set. The Linear layer maps the features from the above layers to the 3 classes, and the softmax layer determines the probabilities of the features. Additionally, the output, determining what class the chest X-ray is, is specified. 0 means a normal chest X-ray, 1 means virus pneumonia, 2 means bacterial pneumonia. <br /> ![enter image description here][7] <br /> ##Data Importing and Formatting I imported the files from their respective folders. The virus and bacterial pneumonia files were in the same formula, so I used a simple separating sort tool organize only images of a certain class to their corresponding variables. <br /> normalImg = Import /@ FileNames["*.jpeg", NotebookDirectory[] <> "train\\NORMAL"]; virusImg = Import /@ FileNames["*_virus_*.jpeg", NotebookDirectory[] <> "train\\PNEUMONIA"]; bacImg = Import /@ FileNames["*_bacteria_*.jpeg", NotebookDirectory[] <> "train\\PNEUMONIA"]; <br /> I conformed the images to have dimensions of 512 by 512. I did this because I specified the Input layer to take images of only those dimensions. <br /> normNorm = ConformImages[Import /@ normalFiles, {512, 512}]; virusNorm = ConformImages[Import /@ virusFiles, {512, 512}]; bacNorm = ConformImages[Import /@ bacFiles, {512, 512}]; <br /> Below is how conformed images look. <br /> Before: <br /> ![enter image description here][8] <br /> After: <br/> ![enter image description here][9] <br /> Later, I adjusted the way I conformed my images. I augmented my data set to improve my neural net. By altering the data set, my neural net would become adapted to a greater variety of images. <br /> normNorm = ConformImages[ Table[If[RandomReal[] > .8, If[RandomReal[] > .5, rotateImage[i], shearImage[i]], i], {i, Import /@ normalFiles}], {512, 512}]; virusNorm = ConformImages[ Table[If[RandomReal[] > .8, If[RandomReal[] > .5, rotateImage[i], shearImage[i]], i], {i, Import /@ virusFiles}], {512, 512}]; bacNorm = ConformImages[ Table[If[RandomReal[] > .8, If[RandomReal[] > .5, rotateImage[i], shearImage[i]], i], {i, Import /@ bacFiles}], {512, 512}]; <br /> I made 40% of the images undergo a rotation via rotateImage, and I made another 40% of the images undergo a perspective transformation via shearImage. <br /> rotateImage[i_] := ImageRotate[i, RandomReal[{-Pi/27, Pi/27}]]; shearImage[i_] := ImagePerspectiveTransformation[ i, {{1, RandomReal[{-.25, .25}], 0}, {0, 1, 0}, {0, 0, 1}}]; <br /> Below are the images before a rotate and perspective change respectively. ![enter image description here][10] <br /> ##Preparing Images Above, two separate neural networks were created. One, the resizedNet, extracts features. The other, the learnNet, would interpret the features and determine the output. By creating two separate neural networks we can preprocess the data instead of having to process the data repeatedly as we would have to in a prematurely combined network. Below, I process the images to extract features using the resizedNet, a neural network that uses the ResNet 50 framework. <br /> normalFeats = resizedNet[normNorm, TargetDevice -> "GPU"]; virusFeats = resizedNet[virusNorm, TargetDevice -> "GPU"]; bacFeats = resizedNet[bacNorm, TargetDevice -> "GPU"]; <br /> Below, the feature extracted images undergo two processes. First, the images are mapped to certain numerical codes. The normal chest X-rays are mapped to the code 0, the virus chest X-rays are mapped to the code 1, and the bacteria chest X-rays are mapped to the code 2. The images are mapped to codes, so the neural network knows the identity of the images when training the network. In addition, the codes are outputted when the trained network interprets an image to be a certain class. For example if the trained network is given a normal chest X-ray, then it should output the code 0. Second, the processed images are joined into one massive list. <br /> feats = RandomSample[ Join[Map[# -> 0 &, resizedNet[normNorm, TargetDevice -> "GPU"]], Map[# -> 1 &, resizedNet[virusNorm, TargetDevice -> "GPU"]], Map[# -> 2 &, resizedNet[bacNorm, TargetDevice -> "GPU"]]]]; <br /> The massive list of processed images are partitioned into a training set and a test set. The training set will train the network and determine the proper weights needed to recognize the images properly. The test set will be used to check the true progress of the neural network. Since, the neural network is never trained on the test set, it would serve as an objective indicator of progress when training. <br /> trainFeat = feats[[1 ;; 5000]]; testFeat = feats[[5000 ;; 5200]]; ##Training the Neural Net# <br /> The below script was used to train the learnNet with the training data set, trainFeat, and the testing data set. <br /> trainedNet = NetTrain[learnNet, trainFeat, ValidationSet -> testFeat, MaxTrainingRounds -> 35, TargetDevice -> "GPU"] <br /> I trained the neural network 4 separate times each with different settings: 1. Regular data set 35 rounds of training 2. Regular data set 70 rounds of training 3. Augmented data set 35 rounds of training 4. Augmented data set 70 rounds of training <br /> The accuracy results of the neural network are below 1. Regular 35 rounds: about 24.9% error 2. Regular 70 rounds: about 19.9% error 3. Augmented 35 rounds: about 20.9% error 4. Augmented 70 rounds: about 17.4% error <br /> Below is an image loss function and error rate of the Augmented 70 rounds neural net. I used this neural net in the final product. <br /> ![enter image description here][11] <br /> The trained neural network was exported for use on the microsite. <br /> Export["/pnemonia.wlnet", combinedNet] <br /> #Determining Location of Pneumonia# <br /> 4 separate neural networks were created to determine the location of the pneumonia: leftVirus, rightVirus, leftBacteria, rightBacteria. The left lob images were evaluated in a separate neural network because an image of the left side of the chest when compared to the right side of the chest may appear to have pneumonia; the heart appears very white much like a pulmonary consolidation, a key evidence of pneumonia. Evaluating the lobes separately for each type of pneumonia helped ensure accuracy in location recognition. <br /> ##Importing Images <br /> For the pneumonia location detection part of this project a subset of data was created. The right and left lungs are often referred to as lobes. 100 images of left lob normal and right lobe normal each and 100 images of left lobe virus, right lobe virus, left lobe bacteria, right lobe bacteria pneumonia each were selected to train 4 separate neural nets to help determine location of the pneumonia. <br /> Below are the statements to import the unedited chest X-rays. halfNormalImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/NORMAL"]; leftVirusImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/VIRUS/Left_Real"]; rightVirusImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/VIRUS/Right_Real"]; leftBacImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/BACTERIA/Left_Real"]; rightBacImg = Import /@ FileNames[Except["."] ~~ "*.jpeg", NotebookDirectory[] <> "HalfData/BACTERIA/Right_Real"]; <br /> ##Image Editing Functions# <br /> Below are the functions which were used to edit the images. "widthResize" and "cropFinder" both are used in conjunction to crop the image. The number of black pixels on both sides of an image are found at 5 separate places. The greatest amount of pixels for both side are used to crop the image. This function was useful because it almost always placed the spinal chord in the center, and as a result the images could be correctly separated. cropFinder[x_, img_] := Module[{mid, flatValues, cropLeft, cropRight}, mid = Ceiling[Last[ImageDimensions[img]]/(x)]; flatValues = Flatten@ImageData[ ImageTake[ColorConvert[img, "Grayscale"], {mid, mid}]]; cropLeft = LengthWhile[flatValues, # < 0.2 &]; cropRight = LengthWhile[Reverse[flatValues], # < 0.2 &]; List[cropLeft, cropRight] ] widthResize[img_] := Block[{ flatValues, cropLeft, cropRight, meanLeft, meanRight, list1, list2, list3, list4, list5, resizedImage}, list1 = cropFinder[10, img]; list2 = cropFinder[2, img]; list3 = cropFinder[4, img]; list4 = cropFinder[1.5, img]; list5 = cropFinder[1, img]; meanLeft = meanLeft /. x_ /; x > 300 :> 0; meanLeft = -(Max[{list1[[1]], list2[[1]], list3[[1]], list4[[1]], list5[[1]]}]); meanRight = meanRight /. x_ /; x > 300 :> 0; meanRight = -((Max[{list1[[2]], list2[[2]], list3[[2]], list4[[2]], list5[[2]]}])); resizedImage = ImagePad[img, {{meanLeft, meanRight}, {0, 0}}]; ImageResize[resizedImage, {512, 512}] ] <br /> ![enter image description here][12] <br /> ##Image Editing Functions cont.# <br /> The below functions were used to separate the images into halves. If a certain image was in the left lobe virus data set, only the left side was extracted. The respective operations were performed on all of the groups of data. (Note: The chest X-rays display the right side of the chest on the left and the left side of the chest on the right. When I crop the image for the left section it will extract the right side of the image.) <br /> halfFormat[x_, y_] := Block[{i} , Table[ImageCrop[widthResize[x[[i]]], {256, 512}, y], {i, Length[x]}]]; singleHalfFormat[x_, y_] := ImageCrop[widthResize[x], {256, 512}, y]; <br /> ![enter image description here][13] <br /> Below was the script that was used to edit the images <br /> leftNormalNorm = halfFormat[widthResize[halfNormalImg], Left]; rightNormalNorm = halfFormat[widthResize[halfNormalImg], Right]; leftVirusNorm = halfFormat[widthResize[leftVirusImg], Left]; rightVirusNorm = halfFormat[widthResize[rightVirusImg], Right]; leftBacNorm = halfFormat[widthResize[leftBacImg], Left]; rightBacNorm = halfFormat[widthResize[rightBacImg], Right]; <br /> ##Neural Network <br /> The ResNet 50 was used for transfer learning with a similar set up as the neural network created above. <br /> ![enter image description here][14] <br /> A "halfLearnNet" was created with a "DropoutLayer", "LinearLayer", "SoftmaxLayer", and an "Output" of two classes. The outputs were 0 and 1; 0 means that the chest X-ray given is a normal image(no pneumonia); 1 means that the lung had pneumonia. <br /> ![enter image description here][15] <br /> ##Preparing Images/Training the Neural Net# <br /> The below sequence of steps were used for the other neural nets (right lobe virus, left lobe bacteria, right lobe bacteria). The trained files were exported for use on the microsite. <br /> ![enter image description here][16] <br /> Below are error rates of the location detecting neural networks. <br /> - Virus Left : about 5% error <br /> - Virus Right : about 5% error <br /> - Bacteria Left: about 0% error <br /> - Bacteria Right: about 15% error <br /> #Creating a MicroSite# <br /> Due to the immense size of my trained neural nets, I needed to create a cloud objects for them. <br /> bacLeftNet = CloudImport[CloudObject["bacLeftNet.wlnet"]] bacRightNet = CloudImport[CloudObject["bacRightNet.wlnet"]] virusLeftNet = CloudImport[CloudObject["virusLeftNet.wlnet"]] virusRightNet = CloudImport[CloudObject["virusRightNet.wlnet"]] CloudImport[CloudObject["pneumonia.wlnet"]] <br /> I created a function which performed my desired tasks: Determine if the given chest X-ray is normal, bacterial, or virus. If the chest X-ray appeared to have pneumonia, the function outputted an the probability that the pneumonia was in the right lung and the left lung. In addition, the given image was returned so the user could view. it. <br /> ![enter image description here][17] <br /> Below was the code for my microsite. <br /> CloudDeploy[Delayed@FormPage[{"image" -> "Image"}, tesla[#image] &, AppearanceRules -> <| "Title" -> "Machine Learning Pneumonia Diagnosis", "Description" -> "<style>.wolfram-branding>.wolfram-branding-cloud:after{\ background-image: url(http://i.imgur.com/hFuh1YT.png)}</style>Upload \ a Chest X-Ray to recieve a diagnosis", "SubmitLabel" -> "Classify", "PageTheme" -> "Red"|>], "test3", Permissions -> "Public"] <br /> If an image is given to the microsite the following is outputted. <br /> ![enter image description here][18] #Real World Testing# <br /> Real world testing was performed to simulate a real use case. I took a picture of the screen, like someone would take a picture of an chest X-ray. <br /> ![enter image description here][19] <br /> The neural network correctly guessed that the image was an instance of viral pneumonia. <br /> ![enter image description here][20] <br /> #Conclusion and Acknowledgement <br /> I created a transfer learning neural networks which successful diagnosed pneumonia with an accuracy of about 83% and accurately detected the location of the pneumonia with an average accuracy of about 93%. Data editing and augmentation procedures and increasing the amount of training rounds helped increase accuracy of diagnoses. I would like to thank my mentor, Michael Kaminsky, for helping guide me. In addition, I would like to thank other mentors including Rick Harrigan, Andrea Griffin, Douglas Smith, and Greg Hurst for helping me. Finally, I would like to thank my parents for encouraging me to pursue my passions and keep learning. #Future Improvements# - Improving image formatting function to make the current program more compatible with poor quality chest X-Rays. - Using a similar Neural Network set up except with thermal images to efficiently diagnose pneumonia. - Further augmenting data to improve accuracy of the neural network. #Citations https://www.mayoclinic.org/diseases-conditions/pneumonia/symptoms-causes/syc-20354204 https://my.clevelandclinic.org/health/diseases/4471-pneumonia https://www.hopkinsmedicine.org/healthlibrary/conditions/respiratory_disorders/pneumonia_85,P01321 https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia https://www.med-ed.virginia.edu/courses/rad/cxr/pathology3chest.html https://machinelearningmastery.com/transfer-learning-for-deep-learning/ https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618/ref=as_li_ss_tl?ie=UTF8&qid=1505772822&sr=8-1&keywords=deep+learning&linkCode=sl1&tag=inspiredalgor-20&linkId=e16a8bbb28ecba154d225ca5f7efcbed http://www.cell.com/cell/fulltext/S0092-8674(18)30154-5 https://data.mendeley.com/datasets/rscbjbr9sj/2 [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-13at10.32.21PM.png&userId=1372752 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt1.jpeg&userId=1372752 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt2.jpeg&userId=1372752 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt4.jpeg&userId=1372752 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt5.jpeg&userId=1372752 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt6.jpeg&userId=1372752 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt7.jpeg&userId=1372752 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt8.jpeg&userId=1372752 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Unt9.jpeg&userId=1372752 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg2.jpeg&userId=1372752 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg3.jpeg&userId=1372752 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg4.jpeg&userId=1372752 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg5.jpeg&userId=1372752 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg6.jpeg&userId=1372752 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg7.jpeg&userId=1372752 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg8.jpeg&userId=1372752 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=arg9.jpeg&userId=1372752 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=lol1.jpeg&userId=1372752 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=lol2.jpeg&userId=1372752 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-13at11.11.17PM.png&userId=1372752 Rohit Panse 2018-07-14T03:48:35Z https://community.wolfram.com/groups/-/m/t/240312 Dear All, I am using NonlinearModelFit to fit data with a function with 4 parameters. With Marquardt Levenberg Algorithm the errors are considered independent, but in reality errors are dependent each other. Is there a build in function which  I can use that when I type nonlimearmodelfit["ParameterErrors"] I can obtain an evaluation of the errors that takes into consideration that they are dependent each other? Many thanks, Best regards, Maria Maria Giovanna Dainotti 2014-04-21T13:05:46Z https://community.wolfram.com/groups/-/m/t/415505 Take a look and consider applying to this year's summer school for doing projects using Wolfram Language https://www.wolframscience.com/summerschool/ Most students are in their early 20s but the range has been 14-76. The Wolfram Science Summer School is about doing projects related to Stephen Wolfram's "A New Kind of Science" http://www.wolframscience.com/nksonline/toc.html and these projects can be from math, physics, computer science, economics, and other sciences, but also have been from art, music, image processing, architecture, data science, etc.. This year there will be a new program for doing technology projects (not just software development, image processing, and data, but also anything related to actual products). More details coming soon. For reference, here is Abby's community post from last year http://community.wolfram.com/groups/-/m/t/196111?p_p_auth=ExbZ2bNA Todd Rowland 2015-01-02T16:39:39Z https://community.wolfram.com/groups/-/m/t/1135996 A Road Map for Rivers ================= The goal of this project was to identify rivers on maps and satellite images, and to calculate each river's graph structure. We produced two functions: one will return a highlighting of all the water in a satellite image given a location entity, and the other will return the graph structure of the water in the related street map image of a given location entity. Both functions start by taking the related street map of a location entity and stripping the image of every color other than one light shade of blue. Let's use Seoul, South Korea as an example: ![enter image description here][1] ![enter image description here][2] waterhighlight[entity_] := Module[{streetmap, color, r, bluemap, mask, satmap}, streetmap = RemoveAlphaChannel@ FirstCase[ GeoGraphics[entity, GeoBackground -> "StreetMap"], _Image, $Failed, Infinity]; satmap = ImageResize[ RemoveAlphaChannel@ FirstCase[ GeoGraphics[entity, GeoBackground -> "Satellite"], _Image, $Failed, Infinity], ImageDimensions[streetmap]]; color = Apply[List, ColorConvert[Interpreter["Color"]["RGB 158 197 226"], "RGB"]]; r = If[ImageColorSpace[streetmap] != "RGB", ColorConvert[streetmap, "RGB"], streetmap]; bluemap = SetAlphaChannel[r, Binarize[r, (Norm[# - color] < .15) &]]; mask = Closing[#, 7] &@FillingTransform@Binarize@bluemap; HighlightImage[ satmap, {EdgeForm[{Red, AbsoluteThickness[1.5]}], FaceForm[{Red, Opacity[.2]}], mask}] ]; The first function takes the blue image and uses it as a mask to highlight the related satellite image. ![enter image description here][3] ![enter image description here][4] watergraph[entity_] := Module[{streetmap, color, r, bluemap, mask}, streetmap = RemoveAlphaChannel@ FirstCase[ GeoGraphics[entity, GeoBackground -> "StreetMap"], _Image, $Failed, Infinity]; color = Apply[List, ColorConvert[Interpreter["Color"]["RGB 158 197 226"], "RGB"]]; r = If[ImageColorSpace[streetmap] != "RGB", ColorConvert[streetmap, "RGB"], streetmap]; bluemap = SetAlphaChannel[r, Binarize[r, (Norm[# - color] < .15) &]]; mask = Closing[#, 7] &@FillingTransform@Binarize@bluemap; MorphologicalGraph@SkeletonTransform[mask] ]; The second function smooths out the blue image, takes it's skeleton, and uses that to calculate the graph structure. The end product really is a graph object, so one could play with it using Wolfram's graph functionality to get some interesting results. ![enter image description here][5] ![enter image description here][6] Since one of the original goals of the project was to identify whether or not an image contained a river, we did make an attempt. We tried to first cut up the satellite images, binarize them, and then sort the tiny images based on the percentage of black and white pixels. We did this because in a large number of the initial images we saw, the water was usually much darker than the surrounding area. However as we looked at more test cases, this method of classifying images became useless. Many pictures require different binarize thresholds, some rivers are actually lighter than their surroundings, they may even be indistinguishable from the background, cloud coverage either blocks the water or can even create dark spots on its own via shadows, rivers may appear too thin to be recognized as such, and so on. As the list of issues grew, we decided to use the street maps due to their reliability. The street maps may have gaps in the water due to bridges, but this can be largely solved with basic image processing. In the future, we would like to implement a random river generation function and find a way to accurately automate the classification of satellite images as: "river" or "not river" to make a training set. From there, you could probably train a neural network to identify rivers from satellite images. [Github][7] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=10658streetmap.png&userId=1081910 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=bluemap.png&userId=1081910 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=satellite.png&userId=1081910 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Highlighted.png&userId=1081910 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Skeleton.png&userId=1081910 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=9925Graphstructure.png&userId=1081910 [7]: https://github.com/Mathfig Z D 2017-07-05T19:22:36Z https://community.wolfram.com/groups/-/m/t/1142221 ## Introduction ## ---------- Hi, I`m Anish Mahendran, and the goal of my project was to create an automated system to identify buildings from Satellite Imagery. This task was accomplished primarily using machine learning through convolution neural networks. The final result was a program which takes an input satellite image, and highlights all of the buildings within the image by segmenting the input image, and classifying each segmented region as either "contains a building" or "does not contain a building," using the neural network. The specific operation of the neural network and the segmentation algorithm are described below. ## Creating and Training the Convolution Neural Network ## ---------- The first step was to create a Neural Network which could identify a single image as a building or not a building. The structure of this Neural Network is illustrated below: lenet = NetChain[{ ConvolutionLayer[30, 5], Ramp, PoolingLayer[2, 2], ConvolutionLayer[60, 5], Ramp, PoolingLayer[2, 2], FlattenLayer[], 700, Ramp, 2, SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", {"Building", "NotBuilding"}}], "Input" -> NetEncoder[{"Image", {100, 100}, "RGB"}] ] Next, it was necessary to provide this Neural Network with a vast set of training data in order to effectively identify a wide variety of building types, and to differentiate between buildings and visually similar structures such as roads and parking lots. Unfortunately I could not find any repository of satellite images which were classified in this form, so I had to generate this training data myself. In order to do so, the following procedure was used: First, the google maps api was used in order to get satellite images and street view images of different locations. GetSatImage[x_, y_, zoom_, range_] := Image[GeoGraphics[GeoPosition[{x, y}], GeoServer -> \ {"http://mt0.google.com/vt/lyrs=s&x=`2`&y=`3`&z=`1`"}, GeoRange -> Quantity[range, "Meters"], GeoZoomLevel -> zoom]]; GetStreetImage[x_, y_, zoom_, range_] := Image[GeoGraphics[GeoPosition[{x, y}], GeoBackground -> GeoStyling["StreetMapNoLabels"], GeoRange -> Quantity[range, "Meters"], GeoZoomLevel -> zoom]]; ![enter image description here][1] ![enter image description here][2] As illustrated in the above example, a building is represented by gray pixels within street view. By binarizing the street view Image and placing it atop the original satellite image, the following image is created. HighlightImageParts[x_, y_, zoom_, range_] := Module[{SatImage, StreetImage, B, configured, partioned, values, data, Replaced, color}, SatImage = GetSatImage[x, y, zoom, range]; StreetImage = GetStreetImage[x, y, zoom, range]; color = RGBColor[{0.8705882352941177`, 0.8627450980392157`, 0.8313725490196079`, 1.`}]; Replaced = ColorReplace[StreetImage, color -> Black, 0.01]; B = Binarize[Replaced, 0]; configured = DeleteSmallComponents[ColorNegate[B], 10]; HighlightImage[SatImage, configured]]; ![enter image description here][3] The street view Images were processed so that only the buildings (represented by gray in the street view image) remained, and the image was subsequently binarized and color-negated so that the buildings were represented by white, and all other components of the image were represented by black. This image was then segmented into 100 pixel by 100 pixel parts, and the mean color of each segment was calculated. If this mean exceeded 0.2, that would indicate an abundance of white pixels,so it must have contained a building. The following function takes in arguments of latitude, longitude, zoom level, and range, and returns a list of image segments followed by their building index (a value of above 0.2 represents the presence of a building). AutoGenData[x_, y_, zoom_, range_, partSize_] := Module[{SatImage, StreetImage, B, configured, partioned, values, data, Replaced, color}, SatImage = GetSatImage[x, y, zoom, range]; StreetImage = GetStreetImage[x, y, zoom, range]; color = RGBColor[{0.8705882352941177`, 0.8627450980392157`, 0.8313725490196079`, 1.`}] Replaced = ColorReplace[StreetImage, color -> Black, 0.01]; B = Binarize[Replaced, 0]; configured = DeleteSmallComponents[ColorNegate[B], 10]; partioned = ImagePartition[configured, partSize]; values = Map[1 - ImageMeasurements[#, "Mean"] &, partioned, {2}]; data = Thread[Flatten[ImagePartition[SatImage, partSize]] -> Flatten[values]]]; This approach was used in order to generate approximately 8,000 classified images as the training data. ## Image Segmentation (Naive Approach) ## The above steps were used in order to generate a Neural Network which was quite effective at differentiating between images that are not buildings, and images that are. ![enter image description here][4] Thus, in order to identify all of the buildings within a satellite image, the input image was cut into squares using the Image Partition function, and each part was individually classified. NeuralNetImageCut[img_Image, partSize_Integer, function_] := ImageAssemble[ Map[If[function[#] == "Building", Blend[{#, Red}], #] &, ImagePartition[img, partSize], {2}]]; ![enter image description here][5] However, as illustrated in the above examples, this method is not entirely effective. The issue becomes more prevalent when buildings aren`t arranged in linear order (see below) ![enter image description here][6] The primary cause of these inaccuracies is not entirely due to the ineffectiveness of the neural net. Instead, it is primarily a result of the ineffectiveness of the segmentation algorithm. Arbitrarily cutting the image in this manner results in many buildings being clipped (for instance, a given segment may just have one corner of a building). The Neural Network functions incredibly well when provided with a "convenient" crop containing the entire building, but when the image is cut in this manner, it is no longer possible to ensure that the Neural Network is provided with such conveniently cropped images. Thus, the following alternative algorithm was used: ## Image Segmentation (Interpolation) ## In order to expose the neural network to more orientations, and thus provide the algorithm with more opportunities to correctly identify the building, instead of simply cutting the image into segments, the algorithm would "pan" across the image and stop whenever it noticed something that the neural network would classify as a building. In other words, instead of cutting the image into disjoint parts, the images would have some overlap. This partition was generated using the ImagePartition function along with the secondary argument which specifies an "offset." The Neural Network was then applied to each resulting image, and the neural network`s certainty in whether or not the input was a building was recorded. The results were then plotted using an array plot. TempMap[img_Image, function_, partSize_Integer, offset_Integer] := ArrayPlot@ Rescale@Clip[Rescale[Map[function[#, "Probabilities"]["Building"] &, ImagePartition[ img, {partSize, partSize}, {offset, offset}], {2}]], {0, .5}]; ![enter image description here][7] This "temperature map" of probabilities was processed using the interpolation function in order to make the results smooth and continuous. The resulting, interpolated function was then graphed using a density plot: TempMapDensity[img_Image, function_, partSize_Integer, offset_Integer] := Module[{data, f, cfInt, cf, xr, yr, tempMap}, data = TempMapData[img, function, partSize, offset]; f = ListInterpolation[data]; cfInt = Interpolation@ Table[{p, Append[List @@ ColorData["TemperatureMap"][p], p]}, {p, 0, 1, 0.05}]; cf = RGBColor@*cfInt; {xr, yr} = Dimensions[data]; tempMap = Rasterize[ DensityPlot[f[xr + 1 - x, y], {y, 1, yr}, {x, 1, xr}, ColorFunction -> (cf[#] &), PlotPoints -> 50, AspectRatio -> Divide @@ Dimensions[data], Frame -> False, Background -> None, PlotRangePadding -> None, ImagePadding -> None], Background -> None]; ImageCompose[img, ImageResize[tempMap, ImageDimensions[img]]]]; ## Sample Results ## ![enter image description here][8] ![enter image description here][9] ![enter image description here][10] ## Conclusions and Extensions ## ---------- In conclusion, this program was quite effective at identifying buildings from satellite images. However, there are multiple ways that this program could be improved. First, the size of the test data set could be increased dramatically by automating the process of generating test data (one could possibly make a web crawler which scans a large data set of satellite images in order to generate vast amounts of test data). Furthermore, the current program requires the user to specify the partition size and the offset value manually based on the zoom level of the image. It would be interesting to see this process automated so that the program automatically identifies the zoom level of the image. Future iterations of this program should also try to identify the edges of the buildings themselves once they have been localized to a given, so that the user could perform calculations such as identifying their relative sizes. This could possibly be accomplished through an image processing approach, identifying the gradients within the image, or through a machine learning approach, in which the neural network is trained to identify the vertices of the building. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Capture1.PNG&userId=1140736 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Capture.PNG&userId=1140736 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2366Capture.PNG&userId=1140736 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=6831Capture.PNG&userId=1140736 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=10308Capture.PNG&userId=1140736 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=7765Capture.PNG&userId=1140736 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=3855Capture.PNG&userId=1140736 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=5064Capture.PNG&userId=1140736 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=3666Capture.PNG&userId=1140736 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=10365Capture.PNG&userId=1140736 Anish Mahendran 2017-07-08T18:07:59Z https://community.wolfram.com/groups/-/m/t/1606199 **[Open code in Cloud][1] | Download code to Desktop via Attachments Below** -------------------------------------- ![enter image description here][2] ![enter image description here][3] [@Marco Thiel][at0] pointed me to an interesting project called [The Venice Time Machine][4] (see also a review [video][5] and a [TED talk][6]). The project aims to digitize and algorithmically process a wealth of data packed in ancient manuscripts. Some steps include the following: > **SCANNING**. Paper documents are turned into high-resolution digital images with the help of scanning machines. > **TRANSCRIPTION**. The graphical complexity and diversity of hand-written documents make transcription a daunting task. For the Venice Time Machine, scientists are currently developing novel algorithms that can transform images into probable words. The images are automatically broken down into sub-images that potentially represent words. Each sub-image is compared to other sub-images, and classified according to the shape of word it features. Each time a new word is transcribed, it allows millions of other word transcripts to be recognized in the database. > **TEXT PROCESSING**. The strings of probable words are then turned into possible sentences by a text processor. This step is accomplished by using, among other tools, algorithms inspired by protein structure analysis that can identify recurring patterns. ![enter image description here][7] Because words in the manuscripts are mutually compared to identify similar features and ultimately same words - it reminded me of unsupervised learning. I wanted to play a bit with tech built in in Wolfram Language to see some toy examples at work. (Note, a supervised learning version of this was developed by Marco in a cool project [Classifying Japanese characters from the Edo period][8]). To avoid hunting and cleaning real datasets, lets "simulate handwriting" reflecting on randomness of the character shapes. I also like it because it touches upon some art sides of things, relevant to generation of beautiful random fonts. And actually due to ancient writers' (for example monks) skills and attitudes often handwritten characters will differ less than the typed ones we distort here artificially (see image above). It's a different story of course if we to compare between handwritings of different authors, which is a challenge and what the Venice project envisions ultimately. This becomes even harder if the letters of a handwritten word are seamlessly connected. I deal with single characters, not words. Let's define a function that makes an image from a letter: raster[s_]:=ImagePad[ImageCrop[Rasterize[s,ImageSize->150]],50,White] I padded it to give enough room for random distortions we will apply. Next function looks convoluted, but it is just a transformation formula that sends sin and cos waves over the letters to distort the shapes (think of your reflection in the water of a pond when you drop a stone there). transf[rn_,amp_,fre_]:= #+{ amp Sin[rn[[1]]+fre(#[[1]]+#[[2]])]+amp Cos[rn[[2]]+fre(#[[1]]-#[[2]])], amp Cos[rn[[1]]-fre(#[[1]]-#[[2]])]-amp Sin[rn[[2]]-fre(#[[1]]+#[[2]])]}& The next function actually applies a random transformation to a letter image: randTransf[amp_,fre_][i_]:= With[{rn=RandomInteger[99,2]},ImageTransformation[i,transf[rn,amp,fre]]] Trying this in action for the letters of the word WOLFRAM and "Felipa" font: set:=ImagePad[#,-50,Red]&/@randTransf[.02,20]/@raster/@ (Style[#,FontFamily->"Felipa"]&/@Characters["WOLFRAM"]) set//ConformImages//ImageAssemble ![enter image description here][9] Trying this twice and finding the difference reveals that different random samples differ quite strongly: ImageSubtract[set//ConformImages//ImageAssemble,set//ConformImages//ImageAssemble ![enter image description here][10] Lets generate 6 different random versions (each per a CPU core of my machine for parallel processing): data=Flatten[ParallelTable[set,6]]; Thumbnail[#,50,Padding->None]&/@data ![enter image description here][11] While letters look the same, if you pay attention to the details, the smaller features are different, - that reminds me of ink and hands effects on real paper. Now, to the machine learning. Here is Wolfram Language "out of the box" function, FeatureSpacePlot, that uses machine learning to deduce most relevant features of objects and find similar ones demonstrating the similarity in 2D space. Here the closer objects are more similar. It picks the similarity, correctly gathering letters into 7 different groups - each group for each letter in WOLFRAM: FeatureSpacePlot[data,PlotTheme->"Frame"] ![enter image description here][12] Let's try now another effect. I will increase the frequency of distortion making letters "fuzzy" (I cannot ignore the artistry of the visual :-) ). Repeated generations will give different small-scale features: set:=ImagePad[#,-50,Red]&/@randTransf[.05,100]/@raster/@(Characters["WOLFRAM"]) set ![enter image description here][13] While general shape is the same internal details are very different and noisy. Again generating 6 different sets and applying FeatureSpacePlot we are able to group properly: data=Flatten[ParallelTable[set,6]]; Thumbnail[#,50,Padding->None]&/@data FeatureSpacePlot[data, PlotTheme -> "Frame", PerformanceGoal -> "Quality"] ![enter image description here][14] Of course, with more distortion we would need more sophisticated processing to find clusters of similarity. But this was fun to play with. It would be interesting to come up with something that could cluster same letters even between the 2 fonts shown at the top image in this post. [at0]: https://community.wolfram.com/web/mthiel [1]: https://www.wolframcloud.com/objects/wolfram-community/Random-Fonts-&-Similarities-in-Deformed-Shapes-by-Vitaliy-Kaurov [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=34qtregadsf.png&userId=11733 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=weraw4qtergafdv.png&userId=11733 [4]: http://timemachine.eu/ [5]: https://vtm.epfl.ch/page-110486-en-html/ [6]: https://www.youtube.com/watch?v=b6nfoE9ly90 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-02-05at4.59.26AM.jpg&userId=11733 [8]: https://community.wolfram.com/groups/-/m/t/1221098 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=34qtregadsf.png&userId=11733 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ert43q5etrsbdvzs.png&userId=11733 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=sfaawereg.png&userId=11733 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ertgf34.png&userId=11733 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=weraw4qtergafdv.png&userId=11733 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=sdf3q4tergafsdhng.png&userId=11733 Vitaliy Kaurov 2019-02-05T12:24:26Z https://community.wolfram.com/groups/-/m/t/1137055 ## Introduction **Instance Segmentation** is one of many interesting computer vision tasks that combines **object detection** and **semantic segmentation**. It detects the object, and at the same time, generates a segmentation mask, which you can think as classifying each pixel - whether it belongs to an object or not. Therefore, solving object detection and semantic segmentation together is a good approach to solve instance segmentation. In the summer school, we borrowed the framework of Mask R-CNN, to combine object detection and semantic segmentation in one pipeline, and produced some promising results. Mask R-CNN is developed like this: R-CNN -> Fast R-CNN -> Faster R-CNN -> Mask RCNN. In each stage, researchers solved some bottleneck problems to get faster and better performance. R stands for the region based, so R-CNN a region based convolutional neural network. Mask R-CNN has two stages, the first stage is trying to produce valid bounding box, you can think it as "blobby" image regions because "blobby" regions are likely to contain objects. In the early stage, researchers feed these warped image regions into a convolutional network, and in the output, they put two head, one regression head to produce the bounding-box, another head like SVM to do classify. And peoples kept working on it and make the network more efficient using some tricks like switch the proposal layer and convolutional layer to avoid unnecessary computations. ![enter image description here][1] There is a Github repo [FastMaskRCNN][2], several machine learning enthusiasts are trying to reproduce this paper in Tensorflow. I already obtained >400k epoch weights, but haven't tested yet, I will keep working on it after summer school. This Mask R-CNN model a too big to finish within 2 weeks, especially since I am new to the TensorFlow framework. Here is the graph visualization from TensorBoard, which looks really complicated and I haven't figure out the pipeline yet. ![enter image description here][3] ## My Project Because building Mask R-CNN network in Mathematica side turned out to be too complicated to finish before the deadline. So in the last two days of summer school, I did something simpler, but it's inspired by Mask R-CNN's framework, which uses the bounding-box region and corresponding mask to train a network to produce a binary mask (pixel-to-pixel). And this process is like semantic segmentation. In order to get this "pixel-to-pixel" trianing dataset, I wrote a script to process 24k [COCO][4] train2014 dataset and to crop the bounding box region based on the annotation `json` file. Here is how information encoded for object instance annotations: annotation{ "id" : int, "image_id" : int, "category_id" : int, "segmentation" : RLE or [polygon], "area" : float, "bbox" : [x,y,width,height], "iscrowd" : 0 or 1, } The original annotation `json` data looks like this: {"area" -> 54653., "segmentation" -> {{312.29, 562.89, 402.25, 511.49, 400.96, 425.38, 398.39, 372.69, 388.11, 332.85, 318.71, 325.14, 295.58, 305.86, 269.88, 314.86, 258.31, 337.99, 217.19, 321.29, 182.49, 343.13, 141.37, 348.27, 132.37, 358.55, 159.36, 377.83, 116.95, 421.53, 167.07, 499.92, 232.61, 560.32, 300.72, 571.89}}, "iscrowd" -> 0, "bbox" -> {116.95, 305.86, 285.3, 266.03}, "image_id" -> 480023, "category_id" -> 58, "id" -> 86} The code is very simple, we just need to do the following things: - Use the bounding-box `{{x, y}, {dw, dh}` to trim the image into bounding-box region image. - Use the list of points (vertices of the polygon (encoding ground truth segmentation mask) ) and bounding-box image dimensions to produce the corresponding mask for the region image. convertAnnotationBBoxIntoImageCoord[bbox_List, h_]:= Module[ {x1, y1, dw, dh}, {x1, y1, dw, dh} = bbox; Transpose[{0, h} + {1, -1} Transpose[Partition[{x1, y1, x1 + dw, y1 + dh}, 2]]] ] getMask[pts_List, w_, h_]:= Module[ {vtx}, vtx = convertAnnotationMask[pts, h]; Binarize @ Rasterize[Graphics[{White, Polygon@@vtx}, PlotRange->{{0, w}, {0, h}}, Background->Black],"Image", ImageSize->{w, h}] ] extractBBoxAndMask[img_, bbox_, segPts_List]:= Module[ {bboxLocal, w, h, boxRegion, maskRegion}, {w, h} = ImageDimensions[img]; bboxLocal = convertAnnotationBBoxIntoImageCoord[bbox, h]; boxRegion = ImageTrim[img, bboxLocal]; maskRegion = ImageTrim[getMask[segPts, w, h], bboxLocal]; {boxRegion, maskRegion} ] imgsAndMasks = extractBBox[imgs[[#]], bboxes[[#]], maskCoord[[#]]] & /@ Range[10] ![enter image description here][5] For the network I used to train, I used ENet, which is a very fast and an efficient network. The Mask R-CNN paper used FCN, which is known as standard network to perform semantic segmentation. I also construct this network in Mathematica and I will try it later as well. The ENet architecture is as followings: ![enter image description here][6] ![enter image description here][7] The output is a 256 * 256 * 2 tensor produced by a softmax layer, so it encoded the mask as the Pr[ this pixel belongs to object]. Therefore, I also need to convert the mask image from {0,1} binary into {1,2} as the class label and save it as `.dat` as training labels. Because I was running out of the time, I just trained with this simple input and output. A better way Etienne suggested is to extract the output from the final convolutional layer of yolo as an input feature to feed in near the output, which I will definitely try soon. I trained my network only for 9 hours on a single Tesla K80 GPU, and already got very promising results. The way my `instanceSegmentation[image, net, detectionThreshold, overlapThreshold]` work is as followings: 1. Use YOLO network as detector to produce labels, bounding-boxes, and probabilities 2. Use bounding-boxes to crop the image object region and feed it to our trained network 3. Take the output tensor, convert it to binaryImages, resize it back to bounding box dimensions by using `ImagePad`. padRegionMask[trimmedMask_, bboxLocal_, w_, h_]:= Block[ {x1,y1,x2,y2}, {{x1,y1},{x2,y2}} = bboxLocal; ImagePad[trimmedMask, {{x1, w - x2}, {y1, h - y2}}] ] instanceSegmentation[img_, ennet_, detectionThreshold_, overlapThreshold_]:= Module[ {labels, bboxes, probs, masks, coloredMasks, yoloVis, yoloRes, rectangles, centers}, yoloRes = detection[img, detectionThreshold, overlapThreshold]; rectangles = Transpose[yoloRes][[2]]; {labels, bboxes, probs} = convertYoloResult[yoloRes]; centers = Mean/@ bboxes; masks = produceMask[ennet, img, bboxes]; coloredMasks = Flatten[{RandomColor[], Opacity[.45], #}&/@ masks]; yoloVis = Transpose @ MapThread[ { {Darker @ Green,Opacity[0],#2}, Style[ Inset[#1<>" \n("<>ToString@Round[#3, .01]<>")", #4, {Center, Center} ], FontSize -> Scaled[.03], FontColor -> White, GrayLevel[0,1], Background -> GrayLevel[1,0] ] }&, {labels, rectangles, probs, centers}]; HighlightImage[img, Join[{coloredMasks,yoloVis}], ImagePadding -> Scaled[.02]] ] Ok, here are some results, I only started training today Many cute dogs ![dogs][8] Me and my mentor ![Me with my mentor ][9] Me and my phone ![Me with my phone][10] Me and my coffee ![Me with my coffee][11] Me and my handbag ![Me with my handbag][12] Me and my classmates and his phone ![Me with my classmates with his phone][13] ## Some personal reflection I enjoyed the summer school overall. I have been hoping to explore the Tensorflow framework and watch Stanford CS231n class for few months, but I was always very occupied with school's classes, other projects, lab's assignments, coding interviews, etc. I finally find some peaceful time to sit down and learn things I had always to learn. And in the process, I am also very amazed by the neural network framework Wolfram people developed. This is a very powerful and user-friendly framework that inherits Wolfram Language's elegant syntax and interactive property. I still have some questions about this framework and plan to learn more about it. ## Future Direction 1. Use FCN to do mask semantic segmentation 2. After obtained Mask R-CNN trained network, deploy it on a server and build an interesting iOS application. 3. Collaborate with Medical school people and apply Mask R-CNN to some medical imaging problems. ## Reference 1. Mask R-CNN Paper: https://arxiv.org/abs/1703.06870 Code (under testing): https://github.com/CharlesShang/FastMaskRCNN 2. ENet: https://arxiv.org/abs/1606.02147 3. YOLO2: https://arxiv.org/abs/1506.02640 4. Fully Convolutional Network (FCN): https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf 5. Project repo: https://github.com/zhuwenzhen/InstanceSegmentation [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Framework.png&userId=524853 [2]: https://github.com/CharlesShang/FastMaskRCNN [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=png.png&userId=524853 [4]: http://mscoco.org/dataset/#download [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0515.14.08.png&userId=524853 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Enet.png&userId=524853 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0515.22.14.png&userId=524853 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.02.58.png&userId=524853 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.20.56.png&userId=524853 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.20.47.png&userId=524853 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.21.35.png&userId=524853 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.25.30.png&userId=524853 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2017-07-0516.25.38.png&userId=524853 Wenzhen Zhu 2017-07-05T21:17:23Z https://community.wolfram.com/groups/-/m/t/511375 **Position**: Mathematica Software Engineer at [Sciton Inc.][1] **Location**: Palo Alto, California Job Description --------------- Sciton, Inc. is looking for an exceptional Mathematica Software Engineer with an interest in the medical device (aesthetic laser) technology. The right candidate will use their Mathematica knowledge and experience to work with 3D graphics, perform image processing, and develop user interfaces. We are a diverse group of technical engineers who really enjoy working together to create the best products in the laser industry. We’ve got a good thing here and are looking for the right combination of skills and technical experience. We are a mature startup company with the right attitude of working hard (including long hours) and getting recognized and rewarded for your creativity. Specific Duties & Responsibilities ---------------------------------- Development of multiple applications using Wolfram Mathematica software package, use of graphics, communication with hardware, image processing. Qualifications -------------- - Bachelor’s degree (Electrical Engineering, Computer Science, Mathematics, Physics or related fields) required. - Programming experience - 3 plus years of work with Mathematica. - User interface design experience - Worked in a Medical Device industry - Self starter and/or able to work as a team when needed - Strong oral and written communication skills as well as project management skills - Must be willing to be flexible in job duties and take on added responsibility when necessary ---------- To apply for one of the following positions, please see our [careers page][2] or send us your résumé as an electronic attachment (PDF format or Word preferred) to jobs@sciton.com with the job title in the subject line. Sciton is an equal opportunity employer. [1]: http://www.sciton.com/ [2]: http://www.sciton.com/company/careers.html Robert Morquecho 2015-06-09T22:41:05Z https://community.wolfram.com/groups/-/m/t/516227 *Vitaliy Kaurov suggested that I cross post this [StackExchange original][1] here.* I am trying out Mathematica as a prototyping tool. As an initial exercise I have put together a brute force ray tracer, with a view to using Mathematica's built in probability distribution functions to test out some fancy shaders. (My starting point was the F# ray tracer available http://code.msdn.microsoft.com/ParExtSamples.) As this is a first attempt at using Mathematica, other than as a fancy calculator, I would welcome some **guidance or critique on whether this coding style is going to be effective**. (You will detect from the code that I am accustomed to an object-oriented paradigm.) My own observations, in no particular order, are: 1. It's an enjoyable way to work, as a lot of the busy type definitions and additional syntax required by other languages is not needed here; the code is quite compact and the intention fairly readable (IMHO). 2. I have built the ray tracer from scratch, without using the ***built-in graphics primitives*** as that would defeat the learning purpose of the exercise and I would probably not be able to roll my own shaders. As a simple example, addition is not defined for RGBColor, for example. 3. Probably as a consequence, the tracer is very slow, even if I use all the 4 kernels that are available to me. I dare say that the existing code could be speeded up considerably by removing all the type / pattern matching, but the code would be much harder to follow, I think, and certainly much harder to debug. My approach seems to mean that I cannot take advantage of Compile optimization, so far as I can see. If I wanted a fast ray tracer I would write it in C++, but I wonder whether there are any **easy optimizations that I have missed** (and that don't involve mangling what I have to the extent that the prototyping benefits of Mathematica -- easy refactoring -- would be lost). However, I am surprised that even with 4 kernels, the cpu of this 4-core (8-core, if you count hyperthreading) never maxes out. For example, is it worth changing some of the `:=` to `=`? Things would be even slower if I aliased the results by averaging 4 adjacent traces, for example. **Real time ray tracing** seems out of reach. 4. Ray tracing results in a lot of "corner cases" that are dealt with naturally with **IEEE maths**. Unfortunately Mathematica does produce +/-Infinity for +/-1/0, for example, so some of the code should really be extended to treat those cases properly. 5. It would be great if Mathematica had more built-in ***vector algebra*** so that I could write the equations defining the objects and the rays involved and getting Mathematica to calculate the ray intersection points that are at the heart of the ray tracer. As things stand, `Reduce` and `Solve` have not helped me to find better intersection algorithms, producing either nothing at all or something large an unintelligible, depending on how I posed the problem. Anyway, here is what you get after 110s from `raytrace[400, 400, basescene, 6]`: ![Here is what you get after 110s from `raytrace[400, 400, basescene, 6]`][2] ... and here is the code: (* Colour helpers *) black = {0., 0., 0.}; darkgrey = {.25, .25, .25}; grey = {.5, .5, .5}; white = {1., 1., 1.}; background = black; defaultcolor = black; brightness[{r_, g_, b_}] = Mean[{r, g, b}]; scale[k_, c: {r_, g_, b_}] = k * c; zero = {0.,0.,0.}; (* Mainly for reference; pattern matching normally used instead *) ray /: start[ray[s_, d_]] = s; ray /: dir[ray[s_, d_]] = d; camera /: pos[camera[p_, l_]] = p; camera /: lookat[camera[p_, l_]] = l; camera /: forward[camera[p_, l_]] = Normalize[l - p]; camera /: down[camera[p_, l_]] = {0., -1., 0.}; camera /: right[c : camera[p_, l_]] := 1.5 * Normalize[Cross[forward[c], down[c]]]; camera /: up[c : camera[p_, l_]] := 1.5 * Normalize[Cross[forward[c], right[c]]]; light /: pos[light[p_, c_]] = p; light /: color[light[p_, c_]] = c; scene /: things[scene[t_, l_, c_]] = t; scene /: lights[scene[t_, l_, c_]] = l; scene /: camera[scene[t_, l_, c_]] = c; surface /: diffuse[surface[d_, s_, re_, ro_]] = d; surface /: specular[surface[d_, s_, re_, ro_]] = s; surface /: reflect[surface[d_, s_, re_, ro_]] = re; surface /: roughness[surface[d_, s_, re_, ro_]] = ro; intersection /: thing[intersection[t_, r_, d_]] = t; intersection /: ray[intersection[t_, r_, d_]] = r; intersection /: dist[intersection[t_, r_, d_]] = d; miss = intersection[nothing, ray[zero, zero], Infinity]; sceneobject /: surface[sceneobject[s_, i_, n_]] = s; sceneobject /: intersect[sceneobject[s_, i_, n_]] = i; sceneobject /: normal[sceneobject[s_, i_, n_]] = n; sphere /: center[sphere[c_, r_, s_]] = c; sphere /: radius[sphere[c_, r_, s_]] = r; sphere /: surface[sphere[c_, r_, s_]] = s; normal[sphere[center_, _, _], pos_] = Normalize[pos - center]; plane /: normal[plane[n_, o_, s_]] = n; plane /: offset[plane[n_, o_, s_]] = o; plane /: surface[plane[n_, o_, s_]] = s; normal[plane[n_, _, _], _] = n; (* Axis-aligned bounding box *) (* TODO: not yet used; integrate into tracer *) box /: lowerb[box[l_, u_]] := l; box /: upperb[box[l_, u_]] := u; extendby[box[l_, u_], pt_] := box[MapThread[Min, {l, pt}], MapThread[Max, {u, pt}]]; size[box[l_, u_]] := u - l; majoraxis[b : box[l_, u_]] := Ordering[size[b], -1]; (* TODO: This does not work for cases where dir has 0 compnent as Mathematic returns ComplexInfinity, not +/-Infinity for +/-1/0 *) intersectboxQ[b : box[l_, u_], r : ray[start_, dir_]] := Module[ {tl = (l - start) / dir, tu = (u - start) / dir, tmin, tmax}, (* Swap u[i] and l[i] if dir[i] < 0 to avoid erroneous result because 0 == -0 *) tmin = Max[MapThread[Min, {tu, tl}]]; tmax = Min[MapThread[Max, {tu, tl}]]; Not[tmax < 0 && tmin > tmax]; (* Use Not to cover some Infinity comparisons *) (* Which[ tmax < 0, False, (* Intersection at t = tmax, but it's behind us *) tmin > tmax, False, (* No intersection *) True, True (* Interesection at t = tmin *) ]*) ]; intersect[s : sphere[center_, radius_, _], r : ray[start_, dir_], i : intersection[_, _, currentdist_]] := Module[ {eo = center-start, v, dist, disc}, v = eo.dir; dist = If[ v < 0., 0., disc = radius * radius - (eo.eo - v * v); If[ disc < 0., 0., v - Sqrt[disc] ] ]; If[ dist == 0. || dist > currentdist, i, intersection[s, r, dist] ] ]; intersect[p : plane[norm_, offset_, _], r : ray[start_, dir_], i : intersection[_, _, currentdist_]] := Module[ {denom = norm . dir, candidatedist}, If[ denom >= 0., i, candidatedist = (norm . start + offset) / (-denom); If[ candidatedist > currentdist, i, intersection[p, r, candidatedist] ] ] ]; testray[ray_, scene_] := dist[Fold[intersect[#2, ray, #1]&, miss, things[scene]]]; traceray[ray_, scene_, depth_, maxdepth_] := shade[Fold[intersect[#2, ray, #1]&, miss, things[scene]], scene, depth, maxdepth]; shade[miss, _, _, _] := background; shade[intersection[thing_, ray[start_, dir_], dist_], scene_, depth_, maxdepth_] := Module[ {pos = dist * dir + start, n, reflectdir, naturalcolor, reflectedcolor}, n = normal[thing, pos]; reflectdir = dir - 2. * n . dir * n; naturalcolor = defaultcolor + getnaturalcolor[thing, pos, n, reflectdir, scene]; reflectedcolor = If[ depth >= maxdepth, grey, getreflectioncolor[thing, pos + (0.001*reflectdir), n, reflectdir, scene, depth, maxdepth] ]; naturalcolor + reflectedcolor ]; getreflectioncolor[thing_, pos_, n_, rd_, scene_, depth_, maxdepth_] := reflect[surface[thing]][pos] * traceray[ray[pos, rd], scene, depth + 1, maxdepth]; getnaturalcolor[thing_, pos_, n_, rd_, scene_] := Module[ {addlight, normraydir = Normalize[rd], howrough = roughness[surface[thing]]}, SetAttributes[addlight, Listable]; addlight[light[p_, c_]] := Module[ {ldis = p - pos, livec, neatisect, isinshadow, illum, lcolor, spec, scolor}, livec = Normalize[ldis]; neatisect = testray[ray[pos, livec], scene]; isinshadow = neatisect <= Norm[ldis]; If[ isinshadow, defaultcolor, illum = livec . n; lcolor = If[ illum > 0., illum * c, defaultcolor ]; spec = livec . normraydir; scolor = If[ spec > 0., (spec ^ howrough) * c, defaultcolor ]; diffuse[surface[thing]][pos] * lcolor + specular[surface[thing]][pos] * scolor ] ]; defaultcolor + Total[addlight[lights[scene]]] ]; raytrace[screenwidth_ : 64, screenheight_ : 64, scene_ : basescene, maxdepth_ : 1] := Module[ {getpoint}, getpoint[x_, y_, camera_] := With[ {recenterx = (x - screenwidth / 2.) / (2. * screenwidth), recentery = -(y - screenheight / 2.) / (2. * screenheight)}, Normalize[forward[camera] + recenterx * right[camera] + recentery * up[camera]] ]; Image[ParallelArray[traceray[ray[pos[camera[scene]], getpoint[#2-1, #1-1, camera[scene]]], scene, 0, maxdepth]&, {screenheight, screenwidth}]] // AbsoluteTiming ]; (* Harness *) (* surface diffuse, specular reflect, roughness *) uniformsurface[diffuse_, specular_, reflect_, roughness_] = surface[diffuse&, specular&, reflect&, roughness]; shiny = uniformsurface[white, grey, .7, 250.]; matteshiny = uniformsurface[white, darkgrey, .7, 250.]; checkerboard = surface[ If[ OddQ[Floor[#[[3]]] + Floor[#[[1]]]], white, black ]&, white &, If[ OddQ[Floor[#[[3]]] + Floor[#[[1]]]], .1, .7 ]&, 150.]; basescene = scene[{ sphere[{0., 1., -.25}, 1., shiny], sphere[{-.5, 1.3, 1.5}, 0.5, matteshiny], plane[{0., 1., 0.}, 0., checkerboard]}, {light[{-2., 2.5, 0.}, {.5,.45,.41}], light[{2.,4.5,2.},{.99,.95,.8}]}, camera[{2.75, 2.0, 3.75}, {-.6, .5, 0.}]]; ---------- To illustrate where I am going with this, here is an example of how one could generate a mesh for a more complex object (a polysphere): polyspherepoints[rad_Real, divs_Integer] := With[ {u = -(Pi/2.), v = -Pi, du = Pi/divs, dv = (2.*Pi)/divs}, rad * Flatten[Table[{Cos[du*i + u]*Cos[dv*j + v], Sin[du*i + u], Cos[du*i + u]*Sin[dv*j + v]}, {j, 0, divs}, {i, 0, divs}], 1] ]; (* Put the polygon vertices in the right order *) ordervertices[{{a_, b_}, {c_, d_}}] := {a, b, d, c}; orderverticestotriangeles[{{a_, b_}, {c_, d_}}] := {{a, b, d}, {a, c, d}} (* Generate a list of (polyspherepoint) vertice numbers, partition them cyclically, and then into quads, and associate them with Polygons *) polyspheremeshtriangles[rad_Real, divs_Integer] := Normal @ GraphicsComplex[polyspherepoints[rad, divs], Map[Polygon, Map[orderverticestotriangeles, Partition[Partition[Range[(divs+1)^2], divs+1], {2, 2}, 1, 1], {2}], 1]]; polyspheremeshtriangles[rad_Real, divs_Integer] := Normal @ GraphicsComplex[polyspherepoints[rad, divs], Map[Polygon, Map[orderverticestotriangeles, Partition[Partition[Range[(divs+1)^2], divs+1], {2, 2}, 1, 1], {2}], 1]]; (It would have been satisfying to use some of the geometric transform functions built into Mathematica to generate the vertices, but life was too short.) And here is what `Graphics3D @ polyspheremeshtriangles[1., 8]` generates: ![PolySphere, of trianges][3] [1]: http://mathematica.stackexchange.com/questions/48675/is-this-the-most-effective-and-efficient-programming-style-for-prototyping-a-ra [2]: http://i.stack.imgur.com/AZE08.png [3]: http://i.stack.imgur.com/RU6dD.png jr p 2015-06-20T11:53:40Z https://community.wolfram.com/groups/-/m/t/193779 We can use the pin notebook to capture the position and time in space of the pendulum.[mcode]pics = {}; Pause[10]; Do[AppendTo[pics, {AbsoluteTime[], CurrentImage[]}]; Pause[0.1], {i, 120}];[/mcode][img]/c/portal/getImageAttachment?filename=PendulumFilm.gif&userId=78214[/img] All we need is the section of interest of the frames for the experiment, so using the ImageTrim functionality we can just keep the part of the frame that captures the movement of the golf ball.Using image processing functions, we can reduce the image to the components that will be key to obtaining the centroid of the ball. [mcode]pics2 = ImageTrim[pics[[#, 2]], {{50., 0.}, {305., 85.}}] & /@ Range[Length@pics]; Manipulate[DeleteSmallComponents@ChanVeseBinarize[Dilation[Blur[pics2[[i]], 5.5], 2],"TargetColor" -> Yellow], {i, 1, Length@pics, 1}][/mcode][img=width: 321px; height: 222px;]/c/portal/getImageAttachment?filename=Pendulum2Film.gif&userId=78214[/img] [mcode]Manipulate[ Show[pics2[[i]], Graphics[{Red, PointSize[Large], Point[Mean@ComponentMeasurements[DeleteSmallComponents@ChanVeseBinarize[Dilation[Blur[pics2[[i]], 5.5], 2],"TargetColor" -> Yellow], "Centroid"][[All, 2]]]}]], {i, 1,Length@pics, 1}][/mcode][img=width: 321px; height: 222px;]/c/portal/getImageAttachment?filename=Pendulum3Film.gif&userId=78214[/img] times will be used to hold the amounts of seconds that have passed sinced we captured the first frame, positions will be holds the centroid location in pixels. [mcode]firstFrame = pics[[1, 1]]; times = pics[[All, 1]] - firstFrame; positions = Mean@ComponentMeasurements[DeleteSmallComponents@ChanVeseBinarize[pics2[[#]], "TargetColor" -> Yellow], "Centroid"][[All, 2]] & /@ Range[Length@pics]; [/mcode]Let's graph the horizontal position of the ball based on the lapsed time. [mcode]ListPlot[Transpose[{times, positions[[All, 1]]}], Joined -> True, AxesLabel -> {"Seconds", "Horizontal Distance\n(pxls)"}][/mcode][img=width: 360px; height: 222px;]/c/portal/getImageAttachment?filename=xAxisPosition.png&userId=78214[/img] Clearly a periodic function. There are 5 peaks between 0 and 6.3 seconds (2 Pi). The period of the function is around 5. Let find an equation that fits the calculation.[mcode]sol = FindFit[Transpose[{times, positions[[All, 1]]}], a Cos[b x + c] + d, {{a, 100}, {b, 5}, c, d}, x, MaxIterations -> 1000] (*{a -> 99.7899, b -> 5.19945, c -> -0.239585, d -> 127.879}*) Show[ListPlot[Transpose[{times, positions[[All, 1]]}], AxesLabel -> {"Seconds", "Horizontal Distance\n(pxls)"}, PlotStyle -> Red, Joined -> False], Plot[a Cos[b x + c] + d /. sol, {x, 0, 17}]] [/mcode][img=width: 360px; height: 222px;]/c/portal/getImageAttachment?filename=xAxisPositionFit.png&userId=78214[/img] The equation that fits the bill and the period of the function are:[mcode](a Cos[b x + c] + d) /. sol (*127.879 + 99.7899 Cos[0.239585 - 5.19945 x]*) T = 2 \[Pi]/b /. sol (*1.20843*)[/mcode] Wikipedia has all the details on the modelling of the simple pendulum experiment.[url=http://en.wikipedia.org/wiki/Simple_pendulum]http://en.wikipedia.org/wiki/Simple_pendulum[/url] which shows that the equation that relates the period with the length of the pendulum is given by. [img=width: 334px; height: 51px;]http://upload.wikimedia.org/math/5/a/a/5aa04824df4e09c1ae352502bdee9c92.png[/img] [mcode]Solve[Period == 2 \[Pi] Sqrt[L/g], g] (*{{g -> (4 L \[Pi]^2)/Period^2}}*)[/mcode] Our measurement of the length was as follows. [img=width: 640px; height: 480px;]/c/portal/getImageAttachment?filename=1836photo.JPG&userId=78214[/img] If we plug the numbers in then we get. [mcode]g = 4 \[Pi]^2 0.364/T^2 (*9.84047*)[/mcode] Checking WolframAlpha [img=width: 552px; height: 259px;]/c/portal/getImageAttachment?filename=gInGa.png&userId=78214[/img] which is 0.5% error. Diego Zviovich 2014-02-01T17:23:49Z https://community.wolfram.com/groups/-/m/t/749947 The Reddit math forum [turned up][1] this [nice site][2] of sudokuish Garam puzzles yesterday. Fill in the blanks. ![A Garam puzzle][3] Someone was quick to post [their Mathematica reduction][4]: ![enter image description here][5] They had to enter the whole puzzle manually, a tedious task. The regular format and clean composition make these puzzles a tempting target for automation. Using colour frequency, I pick out the tiles of the puzzle with morphological tools. ![Said puzzles tiles][6] By checking tile adjacency, I compile lists of 'equations' reading down or across. Then the dark pixels in these tiles are extracted to collect the equation data. ![Some equation imagery][7] A classifier turns the images into equations. ![Some equations][8] Reduce solves all the equations for us, and churns out the answers left to right! ![Solutions][9] Notebook attached for all your Garam cheating needs. [1]: https://www.reddit.com/r/math/comments/3v9ik1/an_interesting_japanese_equation_puzzle_it_took/ [2]: https://www.garamgame.com/ [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=PuzzlePaste.png&userId=445606 [4]: http://i.snag.gy/8cmhp.jpg [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=8cmhp.jpg&userId=11733 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=PuzzleTiles.png&userId=445606 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Someequationimagery.PNG&userId=445606 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Someequations.PNG&userId=445606 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Solutions.PNG&userId=445606 David Gathercole 2015-12-04T09:04:41Z https://community.wolfram.com/groups/-/m/t/509162 By the number of related disciplines, you may appreciate that this is a multidisciplinary problem of significant interest. I will start with the bottom line and then provide brief background. 1. The Fundamental Problem: To Construct a 3-D Model of the 3 mm shell of the Human Cerebral Cortex so that microscopic neuroanatomic data specific to the six layers of 50 different cortical areas can be mapped onto it to show how the human cortex develops during the first six years of life. 2. The Background: We have the largest, quantitative database available on the microscopic, neuroanatomic features of the developing human cerebral cortex. In 1998, these data led to the overturning of the 100 year old dogma of no new neuron formation after birth, and ushered in the era of stem cell research. The data were described by the NY Times as the most important scientific finding of the last decade of the 20th century. Specifically, there are 8 age points of children who died of normal causes, at 0, 1, 3, 6, 15, 24, 48 and 72 months after birth. At each age point, for each layer of 50 cortical areas, there are measures of the numbers of neurons, their size, the numbers of inputs (large dendrites) and outputs (axons), the thickness of each cortical layer, the surface area of each cortical area. We have published that the signal-noise ratio of these data is extremely high, and that there is a common (undiscovered yet) pattern that characterizes the development of each cortical area. 3. The Challenge: To build a 3-D model that can take the values of these microscopic, neuroanatomic features at each age point, layer, and cortical locus, so that these changes can be visualized and analyzed with more complex mathematical methods to discover the underlying rules governing the development of the human cerebral cortex. These rules are likely to govern the structure of development of the cortex of other mammals. 4. An Additional Challenge: We also have camera lucida drawings of the cortical columns of each of these 50 cortical areas for each of the 8 age points. These camera lucida drawings are tracings of representative neurons in each layer of each cortical area. By quantifying these 2-D images and making some assumptions about their 3-D representation, one could incorporate these data into the above model to learn even more. Such research and development should lead to programs and algorithms of extraordinary power that will have use in many different disciplines. I can see how different disciplines would approach this problem in different ways, which is why I posted so many fields of interest. If anyone would like to know more, or has suggestions for developing this project, I have the database, am a statistician and cognitive scientist by training and experience, and am one of the clinical authorities on Alzheimer's disease and other dementing disorders, which is essentially the reverse of development. I would appreciate input from anyone with a creative streak and a taste for a challenging problem. Thanks very much Rod Shankle, MS MD FACP PS: I have attached a PNAS article giving a nice brief summary of the data. William Shankle 2015-06-04T05:57:54Z https://community.wolfram.com/groups/-/m/t/1730668 ## Introduction ## The advent of the field of artificial intelligence initiated various industries, with a prominent one being digit evaluation. The most common AI assisted image processing project to date is the handwritten digit analysis utilizing the MNIST data set, which contains various data samples on handwritten digits, which are organized into uniform sizes. This project is an extension to this, by evaluating handwritten characters (for example, from the EMNIST dataset), and progresses to recognizing whole words and evaluating "possible" words that can be derived from writing. ## Collecting Data ## There are various datasets available online, however we will be using the EMNIST dataset (Extended MNIST) which contains both characters and digits. The EMNIST dataset includes: - Datasets organized into: Byclass, Bymerge, Balanced, Letters, and Digits - Both training and test data according to those organizations - A mapping to convert from class to ASCII decimal codes. For our training purposes, we will be using the balanced training dataset, which is meant to address misclassification errors that occur in the byclass and bymerge datasets. There are a total of 47 classes in the EMNIST Balanced set. Link: https://www.kaggle.com/crawford/emnist After that, we import the downloaded files. balancedTrainData = Import["/Users/danielshin/Desktop/wf_proj/data/emnist/train/emnist-\ balanced-train.csv", "Data"]; balancedMap = Import["/Users/danielshin/Desktop/wf_proj/data/emnist/mapping/emnist-\ balanced-mapping.txt"; balancedTestData = Import["/Users/danielshin/Desktop/wf_proj/data/emnist/test/emnist-\ balanced-test.csv", "Data"]; ## Structure of the EMNIST Dataset ## Unlike the MNIST dataset, which is already included in the wolfram data repository, it is necessary for us to process the data provided by the EMNIST dataset before we can perform operations on it. - The EMNIST dataset is a list of sublists of length 785. - This is because every list is an image classified by its class. Each image is of size 28x28 (28^2 = 784), and one number is allocated to indicate the class (letter or number) of the image. As the dataset includes 13 thousand image samples, we will be taking some examples from the set and run operations on it: test = RandomSample[balancedTrainData, 5]; convertedTest = Image /@ Transpose /@ (Divide[#, 255] & /@ Partition[#[[All]][[2 ;;]], 28] & /@ test) withIndexTest = Table[{test[[n, 1]], convertedTest[[n]]}, {n, 1, Length[test]}] This returns: ![output of variable withIndexTest][1] Similarly, both the training and test dataset can be converted to the appropriate format with this code. However, we can also see that the classes are quite ambiguous. The classes indicated by the EMNIST dataset does not correspond to any character code format. To deal with this, EMNIST provides a mapping .txt file, which converts the classes to ASCII decimal character code format. words = TextWords[balancedMap]; cvtCode2Char[code_Integer] := FromCharacterCode[ ToExpression[words[[Position[words, ToString[code]][[1, 1]] + 1]]]] The **cvtCode2Char** function helps to convert a class to the actual character. Also, we need a special method of indexing to feed the data into the neural network. The neural network accepts a list of associations, which can be converted to with the following code: cvtIndex2Train[index_List] := Table[index[[n, 2]] -> index[[n, 1]], {n, 1, Length[index]}] ** Note that this function can be used both for the training and testing dataset. ## Creating a CNN to Analyze and Interpret Characters ## A convolutional neural network, also known as CNN, is a deep neural network most commonly utilized to perform image processing. CNN structures are similar to the human eye, breaking down and extracting features that are crude or complex. A typical CNN is a combination of convolutional layers, activation functions, and pooling layers. The final neural network created for this project is shown in the code below: convolutionalNeuralNet = NetChain[{ ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2], ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2], FlattenLayer[], DropoutLayer[], LinearLayer[500], Ramp, LinearLayer[47], SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", Range[0, 46]}], "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}] ] There were many iterations of the neural network based on optimizer types and the addition/subtraction of specific layers. First of all, a dropout layer, included to assist in the vanishing gradient problem, was added (and deleted) to check whether it might have an influence on performance. ![With vs. Without DropoutLayer][2] Furthermore, research into convolutional neural nets revealed two optimizers that are currently industry-standard, which were the ADAM and SGD optimizers. A comparison between these two optimizers are shown below: ![Adam vs. SGD][3] With this, it was revealed that for our purposes, an SGD optimizer used with a DropoutLayer[] was the best combination in terms of neural network accuracy. Code used to train network: balancedTrainingData = cvtIndex2Train[indexed]; convolutionalNeuralNet = NetInitialize[convolutionalNeuralNet]; trained = NetTrain[convolutionalNeuralNet, balancedTrainingData, Method -> "SGD"] ## Evaluating Probabilities of Outputs ## Obviously, it is difficult for a neural network to return 100% probability for a single class for every image input. For each classification, there is a probability value associated with it. However, many probability values (especially those that are small) aren't that useful. returnTopPossible[input_Image, int_Integer] := Table[{cvtCode2Char[#[[n, 1]]], #[[n, 2]]}, {n, 1, int}] &[ Flatten /@ List @@@ Normal@ReverseSort[net[input, "Probabilities"]][[;; int]]] The returnTopPossible function can help to return the top few probabilities based on a given input. ![Sample "returnTopPossible" function output][4] Now, it is important to note that probability values range from 0~1, so 0.92 will indicate 92%. The reason why we limit the probability outputs to a few values is because there is only a minimal chance that the neural net processed an image correctly with only an output probability of, lets say, 1/10000. ## Testing on Images Not from EMNIST ## While EMNIST provides diverse data, still the format is similar, so we decided to test the neural network on other external images obtained from the internet. testImage2 = ColorConvert[ ImageResize[ Import["/Users/danielshin/Desktop/test.png"], {28, 28}], "Grayscale"] // ColorNegate; Thread[testImage2 -> returnTopPossible[testImage2, 5]] The output of the testing processed was quite satisfying: ![Detect Q][5] As shown, the neural network outputted a confidence score of the 96% for the image to be the capital letter "Q". The reason we converted the image to grayscale, and then performed a color negate was because the EMNIST dataset is comprised of images that are white in a black background. Therefore, the results of the neural network is vastly inconsistent when we input an image with a white background. ## Finding Characters in a Word ## The neural network we made previously is confined to detecting characters that are confined specifically to an image of size 28x28. But what if there are multiple letters in an image, and the image is not in the appropriate dimensions to be reduced? In this section, we will be creating methods to detect multiple characters out of a single image. For instance, if we have the image down below, our current neural network can't detect it properly. ![New][6] First of all, we extract each letter from the image using functions such as Morphological Components. We created a function called selectChar which outputs a list of images that are single characters extracted from an image of a word. selectChar[image_, negate_: True] := Module[ {binarized, length}, binarized = If[negate, ColorNegate[Binarize[image]], Binarize[image]]; length = Length[Union[Flatten[MorphologicalComponents[binarized]]]] - 1; Table[SelectComponents[binarized, #Label == n &], {n, 1, length}] ] selectChar[word] ** Note that the "negate" parameter is defaulted to True, as we assume that the image has a white background by default. When the function is applied to the image shown above, it outputs: ![New Divided][7] However, it is evident that these words are not organized properly, so if we run the neural network on words that are the way that is the network will return "Nwe" not "New". Therefore, we need a method to organize the images from left to right. In our case, we utilized pixel positions and got the leftmost pixel coordinates to order each image. orderFromLeft[charList_] := Table[#[[i, 1]], {i, 1, Length[charList]}] &[ SortBy[Table[{charList[[n]], SortBy[PixelValuePositions[charList[[n]], 1], First][[1, 1]]}, {n, 1, Length[charList]}], Last]] However, what about letters like "i", or "E" in this case? Letters like that have separations between the top and bottom part, but utilizing MorphologicalComponents will separate them and classify them into different components, as shown in the image below: ![MOVE][8] A function called checkOverlap was created to determine whether to combine components together to form a whole image: checkOverlap[image1_, image2_] := Module[ {overlap, imgW}, imgW = Min[ Max[First /@ PixelValuePositions[image1, 1]] - Min[First /@ PixelValuePositions[image1, 1]], Max[First /@ PixelValuePositions[image2, 1]] - Min[First /@ PixelValuePositions[image2, 1]]]; overlap = Min[Max[First /@ PixelValuePositions[image1, 1]], Max[First /@ PixelValuePositions[image2, 1]]] - Max[Min[First /@ PixelValuePositions[image1, 1]], Min[First /@ PixelValuePositions[image2, 1]]]; If[imgW*0.3 < overlap, True, False]] Still, we can't just feed these images to the neural network. The network chain was defined to take in an image that is of size 28x28. Therefore, we need to find a way to trim and reshape the images into a 28x28 box. reshape[image_] := Module[ {h, w, trimmed, targetH = 26, targetW = 26, fw, fh, scale, resizedImage, rectangle}, trimmed = ImageCrop[image]; {w, h} = ImageDimensions[trimmed]; fw = w / targetW; fh = h / targetH; scale = Max[fw, fh]; resizedImage = ImageResize[trimmed, {Max[w/scale, 1], Max[h/scale, 1]}]; rectangle = Image[Table[Table[0, 28], 28]]; ImageCompose[rectangle, resizedImage, {Center}] ] Finally, there were instances where images contain noise. These random particles were also extracted from morphological components and were resized. To filter these noise out, we decided to compare the black pixel to white pixel ratio. Assuming that each noise is a circle, the ratio would be 0.25 Pi : 1, which is about 615 pixels per 784. filter[images_] := Select[Table[ If[Length[PixelValuePositions[images[[n]], 1]] < 615, images[[n]]], {n, 1, Length[images]}], ImageQ] Still, we realized that by only using image processing, it was hard to get accurate results. Therefore, we decided to employ other methods such as character associations as well. Character association is basically a map of characters and their associations. For instance, "q" is almost always followed by a "u", etc. ![Graph][9] This is the character association map that was created. ## Overarching Prediction Algorithm ## This was the function utilized to create the final algorithm (that encompasses both the image and association probabilities) trail[input_, weights_, edges_, negate_: True] := Module[ {wordList, probabilityList}, wordList = Tuples[imageProbability[moveWord, negate]]; probabilityList = Table[outputProbability[ Select[wordList[[n]] // Flatten, NumberQ], associationProbability[ Select[wordList[[n]] // Flatten, StringQ], edgeWeights, withoutDuplicates ]], {n, Length[wordList]}]; Select[wordList[[ Position[probabilityList, Max[probabilityList]][[1, 1]]]] // Flatten, StringQ] // ToLowerCase // StringJoin ] - imageProbability is a function which outputs classification probabilities of each image. - associationProbability is a function which outputs the probabilities of word associations based on the character association map. We combined these two probabilities to create a final output. ## Conclusion & Future Works ## During this project, algorithms were designed to recognize words in an image. However, results weren't exactly as satisfying as expected. The accuracy of the trained neural network was quite high, but there were some other issues as well. Using character-based relationships, the program successfully impedes consonants from following consonants, such as "Q" following "M", however the replaced vowel wasn't always quite accurate. Other attempts were made, such as adding weights as well. Only after creating weights to decrease the influence of the character association map, was the combination of the two probabilistic values yielded confident results. There are various methods on simulating OCR. For future works, creating neural networks to input whole words and sentences will be able to create a more feasible and easy-to-use product. Furthermore, utilizing similar concepts from this project, using word-mapping (Markov chains), it might be possible to predict wrong characters or words in a sentence, and using image processing neural networks to compensate and replace certain characters. ### Links ### - https://www.kaggle.com/crawford/emnist - [https://commons.wikimedia.org/wiki/File:Math_i_square.png][10] - https://www.codeproject.com/Articles/160868/A-C-Project-in-Optical-Character-Recognition-OCR-U - http://www.fki.inf.unibe.ch/databases/iam-handwriting-database https://www.dreamstime.com/handwritten-word-new-white-background-can-be-used-different-concepts-handwritten-word-new-white-background-image104188885 [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.00.15AM.png&userId=1720539 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.14.20AM.png&userId=1720539 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.16.12AM.png&userId=1720539 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.25.58AM.png&userId=1720539 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.32.48AM.png&userId=1720539 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=test1.png&userId=1720539 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.43.47AM.png&userId=1720539 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at10.50.23AM.png&userId=1720539 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2019-07-11at11.13.20AM.png&userId=1720539 [10]: https://commons.wikimedia.org/wiki/File:Math_i_square.png Daniel Shin 2019-07-11T15:19:26Z https://community.wolfram.com/groups/-/m/t/1137429 ![Main picture][1] # Introduction The aim of this project is to train a Deep Reinforcement Learning agent to bring a smile to your face. The first part of the project was to train a Deep Convoluted Neural Network to predict one of five (happy, neutral, sad, angry, surprised) facial emotions. The input to this classifier is taken directly from the user's camera. Then, using this prediction as an input, a reinforcement learning agent was trained to make you smile. To do that a customizable emoticon was used. ## Why Laetitia? Laetitia is the Roman Goddess of joy, gaiety, and celebration, and is especially linked with holidays and festivals. # Phase one: facial emotion detector The first phase of the project was to implement the facial emotion detector. ## Dataset The very first part of this phase was dataset selection. I based my study on the Fe.r2013 database. It is free and available online on [Kaggle platform][2]. The reason standing behind my choice were small size of training example (pictures 48x48 pixels), huge number of labeled training O(20k) and test O(7k) examples. Each of database element belongs to one of following classes happy, neutral, sad, angry and surprised. Unfortunately, the database is contaminated by noised images or animated faces. It needed lot of afford to manually remove all the useless examples. ![training examples][3] ## Model training At the beginning of my study I trained classical Machine Learning models, such as Supported Vector Machine, Random Forest, k Nearest Neighborhood. These models were used as a performance baseline for further studies based on Deep Convoluted Neural Networks. My analysis shows that models that contain two hidden convoluted layers are the best classifiers. A selected model achieved a performance of 91% in happiness recognition, measured as area under Receiver Operating Characteristic curve. As mentioned before, the best facial emotion detector is a Deep Convolution Neural Network. Above cell presents architecture (upper right plot) of selected network and training progress related plots. ![CNN training][4] ## Convoluted Neural Network performance Classifier performance measurement was performed on O(7k) testing images. At first I present a confusion matrix which is a specific table layout that allows visualization of the performance of an algorithm. Each column of the matrix represents the instances in a predicted class while each row represents the instances in an actual class. The name stems from the fact that it makes it easy to see if the system is confusing two classes (i.e. commonly mislabelling one as another). ![cm][5] Another widely used classifier performance measuring metrics is the area under the Receiver Operating Characteristic curve, which can be interpreted as equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one. The obtained results for particular classes: ![rocs][6] ## Classifier evaluation The crucial project constraint is classifier evaluation time. The classifier need to be able to return response in real time. Here you find sample. The delays are caused by gif segmentation. ![sample][7] ## Transfer learning approach I also tried to use idea of transfer learning. Roughly speaking, we take pre-trained model such as [VGG16](http://www.robots.ox.ac.uk/~vgg/research/very_deep/) or [Inception](https://arxiv.org/abs/1512.00567) and then replace two last layers. emotionTypes = {"happy", "neutral", "sad", "angry", "surprise"} deepCNN = NetModel["Wolfram ImageIdentify Net for WL 11.1"] truncatedNetwork = Take[deepCNN, {1, -3}]; deepCNNInceptionV3 = NetInitialize @ NetChain[{truncatedNetwork, LinearLayer[Length[emotionTypes]], SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", emotionTypes}] ] Unfortunately, due to technical limitation of my notebook (no GPU unit) I was not able to finish training that kind of model. It probably will perform better than shallow version. This idea will be implemented in the future release of the project. # Phase Two: Reinforcement Learning In this phase, based on chosen model prediction, the policy to find the funniest emoticon was obtained. To do it the Monte Carlo based searching method was applied. At first I would like to present tunable emoticon that was used to make you smile. The animation was taken from [Wolfram Demonstration Project site](http://demonstrations.wolfram.com/SmileyChanger/). ![emoticon][8] This element will be replaced into auto generated pictures or jokes in the future release. ## Monte Carlo optimization Monte Carlo methods (or Monte Carlo experiments) are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. Their essential idea is using randomness to solve problems that might be deterministic in principle. From the point of view this work I am using numerical simulation to find the "funniest" emoticon configuration. Please take a look at the simulation sudo code. nFrames = 10; currentHappines = 0.; previousHappines = 0.; happinessesList = {{r, t , s, a , b, c , ey , er, currentHappines}}; ResetParameters[]; Dynamic[ parameter = SelectParameter[]; previousParamValue = KeepPreviousParameterValue[parameter]; update = CalculateParameterUpdate[parameter, step]; UpdateParmeter[parameter, update]; frame = 0; currentHappines = 0.; Column[{ animation, While[frame < nFrames, image = CurrentImage[]; greyFace = ExtractFaceFromImage[image]; currentHappines += PredictHappinessProbability[greyFace]; frame++; ]; currentHappines /= nFrames; image, greyFace, currentHappines, If[currentHappines > previousHappines , previousHappines = currentHappines, ReloadParameterValue[parameter, previousParamValue]]; happinessesList = Join [happinessesList, {{r, t , s, a , b, c , ey , er, currentHappines}}];} ] ] This simulation allows to obtain learning curve like plot and the from them funniest emoticon configuration. ![learningCurveMC][9] #Future goals From the future perspective the Q-Learning approach will be implemented. A starting point of the further studies will be Monte Carlo tuned policy mentioned before. The second idea is to replaced shallow version of CNN into pre-trained model. # Github Please take a look at the project source code on [github](https://github.com/adendek/DeepLaetitia). ####Stay tuned for further improvements! ####I’m happy about any kind of feedback. Please upvote if you like it ;-) [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Classifier.gif&userId=1075716 [2]: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge "Kaggle" [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=images.png&userId=1075716 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=trainingResult.png&userId=1075716 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=CM.png&userId=1075716 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Rocs.png&userId=1075716 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Reaction.gif&userId=1075716 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Emoticon.png&userId=1075716 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MonteCarlo.png&userId=1075716 Adam Dendek 2017-07-05T22:32:13Z https://community.wolfram.com/groups/-/m/t/1661864 [![enter image description here][1]][46] We're glad to announce that Version 12.0 of Mathematica and the Wolfram Language is here—and it's a big one. This major update introduces several hundred new functions in a range of areas, including mathematics and geometry, visualization, audio and image processing, machine learning, and more, as well as provides a major overhaul to the Mac and Linux notebook interfaces and numerous systemwide performance enhancements. Some key new features include: Mathematics and Geometry ------------------------------------- - Convex optimization solvers such as [LinearOptimization][2], [QuadraticOptimization][3] and [SemidefiniteOptimization][4], with vector inequalities such as [VectorGreater][5] for modeling vector-valued variables - Improved [NDSolve][6] support for nonlinear PDEs using FEM and automated idealized *n*-body interaction using [NBodySimulation][7] - Use [Around][8] to describe uncertainty in measured values, with built-in support in visualization functions - A host of new functions for generating, analyzing and manipulating polygons and polyhedra ([RandomPolygon][9], [RandomPolyhedron][10], [PolygonDecomposition][11], etc.), with added support for holes/voids - Access built-in axiomatic theories for use in symbolic proofs with [AxiomaticTheory][12] Visualization ---------------- - [GeometricScene][13] and [GeometricAssertion][14] for visualizing Euclidean geometry and its theorems - Visualize different components and properties of complex-valued data and functions with [ComplexListPlot][15], [ReImPlot][16] and [AbsArgPlot][17] - Represent geodetic vectors with [GeoVector][18] and visualize geodetic vector fields using [GeoVectorPlot][19] and [GeoStreamPlot][20] - Updated default appearance for [GraphPlot][21] visualizations, as well as new plot themes and updated properties - Create, analyze and modify chemical species with [Molecule][22] and visualize their 2D and 3D structures with [MoleculePlot][23] and [MoleculePlot3D][24] Audio and Image Processing --------------------------------------- - Color-processing functions such as [FindMatchingColor][25], [ImageRecolor][26] and [ColorDetect][27], all powered by the new [ColorsNear][28] wrapper to represent a color neighborhood - Multi-language support for [SpeechSynthesize][29], as well as improved discovery and searching of available voices through [VoiceStyleData][30] - [AudioAnnotate][31] framework for annotating audio objects - [ShortTimeFourier][32] for computing and storing properties of short-time Fourier transforms and [InverseSpectrogram][33] for approximating a signal from a magnitude spectrogram Machine Learning and Neural Networks ------------------------------------ - High-level recognition functions such as [ImageCases][34], [FacialFeatures][35], [AudioIdentify][36] and [SpeechRecognize][37] - Additional pre-trained models, new layers, new encoders and decoders, and multi-GPU support for the neural network framework - Learn a generalized probability distribution from arbitrary data using [LearnDistribution][38] - Replace missing values in data by generated values using [SynthesizeMissingValues][39] and find anomalous examples in data using [FindAnomalies][40] These are just a few of the highlights from this release, which also includes an experimental [Wolfram Compiler][41], a comprehensive new [Microcontroller Kit][42], a bidirectional [Unity][43] game engine interface and expanded knowledge and functionality for [Entity][44] and [EntityStore][45]. You can find out more about these and other Version 12 additions by reading through [Stephen Wolfram's launch-day blog post][46] and looking at the [Summary of New Features in 12][47]. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=12-0-wordcloud-hero.png&userId=11733 [2]: http://reference.wolfram.com/language/ref/LinearOptimization [3]: http://reference.wolfram.com/language/ref/QuadraticOptimization [4]: http://reference.wolfram.com/language/ref/SemidefiniteOptimization [5]: http://reference.wolfram.com/language/ref/VectorGreater [6]: http://reference.wolfram.com/language/ref/NDSolve [7]: http://reference.wolfram.com/language/ref/NBodySimulation [8]: http://reference.wolfram.com/language/ref/Around [9]: http://reference.wolfram.com/language/ref/RandomPolygon [10]: http://reference.wolfram.com/language/ref/RandomPolyhedron [11]: http://reference.wolfram.com/language/ref/PolygonDecomposition [12]: http://reference.wolfram.com/language/ref/AxiomaticTheory [13]: http://reference.wolfram.com/language/ref/GeometricScene [14]: http://reference.wolfram.com/language/ref/GeometricAssertion [15]: http://reference.wolfram.com/language/ref/ComplexListPlot [16]: http://reference.wolfram.com/language/ref/ReImPlot [17]: http://reference.wolfram.com/language/ref/AbsArgPlot [18]: http://reference.wolfram.com/language/ref/GeoVector [19]: http://reference.wolfram.com/language/ref/GeoVectorPlot [20]: http://reference.wolfram.com/language/ref/GeoStreamPlot [21]: http://reference.wolfram.com/language/ref/GraphPlot [22]: http://reference.wolfram.com/language/ref/Molecule [23]: http://reference.wolfram.com/language/ref/MoleculePlot [24]: http://reference.wolfram.com/language/ref/MoleculePlot3D [25]: http://reference.wolfram.com/language/ref/FindMatchingColor [26]: http://reference.wolfram.com/language/ref/ImageRecolor [27]: http://reference.wolfram.com/language/ref/ColorDetect [28]: http://reference.wolfram.com/language/ref/ColorsNear [29]: http://reference.wolfram.com/language/ref/SpeechSynthesize [30]: http://reference.wolfram.com/language/ref/VoiceStyleData [31]: http://reference.wolfram.com/language/ref/AudioAnnotate [32]: http://reference.wolfram.com/language/ref/ShortTimeFourier [33]: http://reference.wolfram.com/language/ref/InverseSpectrogram [34]: http://reference.wolfram.com/language/ref/ImageCases [35]: http://reference.wolfram.com/language/ref/FacialFeatures [36]: http://reference.wolfram.com/language/ref/AudioIdentify [37]: http://reference.wolfram.com/language/ref/SpeechRecognize [38]: http://reference.wolfram.com/language/ref/LearnDistribution [39]: http://reference.wolfram.com/language/ref/SynthesizeMissingValues [40]: http://reference.wolfram.com/language/ref/FindAnomalies [41]: http://reference.wolfram.com/language/guide/CodeCompilation [42]: http://reference.wolfram.com/language/MicrocontrollerKit/guide/MicrocontrollerKit [43]: http://reference.wolfram.com/language/UnityLink/guide/UnityLink [44]: http://reference.wolfram.com/language/ref/Entity [45]: http://reference.wolfram.com/language/ref/EntityStore [46]: http://wolfr.am/launching-12 [47]: http://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn12.html Brian Wood 2019-04-16T16:11:50Z https://community.wolfram.com/groups/-/m/t/401698 I am new to image processing. I want to know that how rasberry pi camera is used? Is it suitable for real time projects? How other camera are interfaced to raspberry pi? Any other methods for doing image processing? I would be greatful for your answer.Thanks! nabin ghimire 2014-12-04T07:49:51Z https://community.wolfram.com/groups/-/m/t/2144200 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/d34dbf01-6652-4598-9cff-6aad9b015495 Wolfram Education Programs 2020-12-21T18:20:08Z https://community.wolfram.com/groups/-/m/t/1893009 According to the [official announcement][1]: > For the first time in its 174-year history, the Smithsonian has released 2.8 million high-resolution two- and three-dimensional images from across its collections onto an open access online platform for patrons to peruse and download free of charge. Featuring data and material from all 19 Smithsonian museums, nine research centers, libraries, archives and the National Zoo, the new digital depot encourages the public to not just view its contents, but use, reuse and transform them into just about anything they choose—be it a postcard, a beer koozie or a pair of bootie shorts. *COLLECTION is HERE*: https://www.si.edu/openaccess There are a lot of interesting and images and 3D models from very diverse corners of life, - here are a few samples: ![enter image description here][2] One of coolest thing is that some of 3D models are given in parts that can be 3D printed separately and then assembled together, thus making cool DIY project. Here is an example you can try with a 3D model of [Mammuthus primigenius / Blumbach][3]. Below the model find and copy the download link: ![enter image description here][4] Import data from the collection (I shortened the URL with `URLShorten`): data = Import["https://wolfr.am/KQ6NWYff", "*"] ![enter image description here][5] You can see these are the pieces that should make up a whole object: Show[data] ![enter image description here][6] It is easy to make 3D printouts with the Wolfram Language -- the parts Printout3D /@ data or whole thing: Printout3D[RegionUnion @@ data] ![enter image description here][7] It is a really neat toy and reminds me of 3D puzzles: you get the parts and have to figure out what the whole is by building it. Could be a fun time with kids :-) If you can get your hands on 3D printer it might be better than a toy store. There are very different types too. Here is, for instance for space-dreamers, the [Apollo 11 Command Module][8] -- apart and together. apollo = Import["https://wolfr.am/KQLE0LOx", "*.obj"] ![enter image description here][9] Show[apollo] ![enter image description here][10] [1]: https://www.smithsonianmag.com/smithsonian-institution/smithsonian-releases-28-million-images-public-domain-180974263/ [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-03-06at9.47.27AM.png&userId=11733 [3]: https://3d.si.edu/object/3d/mammuthus-primigenius-blumbach:341c96cd-f967-4540-8ed1-d3fc56d31f12 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-03-06at10.37.24AM.png&userId=11733 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=esgasd4q3Untitled.png&userId=11733 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=fdsfsdfsdfs.png&userId=11733 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=gfxgfdtey35Untitled.png&userId=11733 [8]: https://3d.si.edu/object/3d/command-module-apollo-11:d8c6457e-4ebc-11ea-b77f-2e728ce88125 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-03-06at10.14.41AM.png&userId=11733 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-03-06at10.25.19AM.png&userId=11733 Vitaliy Kaurov 2020-03-06T05:56:34Z https://community.wolfram.com/groups/-/m/t/1184495 [![enter image description here][1]][27] Mathematica 11.2 is now available! This latest version expands Mathematica's cutting-edge functionality in audio and image processing, mathematical computation and machine learning—plus much more. Here are some of the highlights: **Audio and Image Processing** - Generate spoken audio in different voices with [SpeechSynthesize][2] - The [Audio][3] framework now has stereo support, programmatic playback and new [AudioStream][4] objects - Alter one image to look more like another with [ImageRestyle][5] - Use [CurrentScreenImage][6] or [CurrentNotebookImage][7] to capture the current screen or notebook - Generate 2D and 3D raster images from geometric regions with [RegionImage][8] - Retrieve satellite images of a geographic area with [GeoImage][9] **Mathematical Computation** - [Limit][10] can now calculate undirected, nested and multivariate limits - [RSolveValue][11] and [DSolveValue][12] can now compute limiting values directly - Additional functions and operators for calculating limits: [MinLimit][13], [MaxLimit][14], [DiscreteLimit][15], etc. - Support for 3D Boolean operations through [RegionUnion][16], [RegionIntersection][17], etc. **Machine Learning** - [Classify][18] and [Predict][19] have new training methods, built-in models, progress reporting and [TimeGoal][20] for limiting training times - New feature types for extracting audio, sequences and word vectors from data - [WMLF][21] and [MXNet][22] formats for import/export of neural networks and other machine learning models - Expanded options and methods for various neural network layer types Version 11.2 also includes a number of new import/export formats, interpreter types and low-level performance optimizations. Programming is more flexible than ever with new functionality for [initialization][23], [background tasks][24] and [network programming][25], as well as new interfaces for [external interpreted languages][26]. For more details about these and other new additions, read through Stephen Wolfram's [launch-day blog post][27] and take a look at the [Summary of New Features in 11.2][28]. [![enter image description here][29]][27] [![enter image description here][30]][27] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=112-wordcloud.png&userId=11733 [2]: http://reference.wolfram.com/language/ref/SpeechSynthesize [3]: http://reference.wolfram.com/language/ref/Audio [4]: http://reference.wolfram.com/language/ref/AudioStream [5]: http://reference.wolfram.com/language/ref/ImageRestyle [6]: http://reference.wolfram.com/language/ref/CurrentScreenImage [7]: http://reference.wolfram.com/language/ref/CurrentNotebookImage [8]: http://reference.wolfram.com/language/ref/RegionImage [9]: http://reference.wolfram.com/language/ref/GeoImage [10]: http://reference.wolfram.com/language/ref/Limit [11]: http://reference.wolfram.com/language/ref/RSolveValue [12]: http://reference.wolfram.com/language/ref/DSolveValue [13]: http://reference.wolfram.com/language/ref/MinLimit [14]: http://reference.wolfram.com/language/ref/MaxLimit [15]: http://reference.wolfram.com/language/ref/DiscreteLimit [16]: http://reference.wolfram.com/language/ref/RegionUnion.html [17]: http://reference.wolfram.com/language/ref/RegionIntersection.html [18]: http://reference.wolfram.com/language/ref/Classify [19]: http://reference.wolfram.com/language/ref/Predict [20]: http://reference.wolfram.com/language/ref/TimeGoal [21]: http://reference.wolfram.com/language/ref/format/WMLF.html [22]: http://reference.wolfram.com/language/ref/format/MXNet.html [23]: http://reference.wolfram.com/language/guide/InitializationAndProvisioning.html [24]: http://reference.wolfram.com/language/guide/BackgroundAndScheduledTasks.html [25]: http://reference.wolfram.com/language/guide/NetworkProgramming.html [26]: http://reference.wolfram.com/language/guide/ExternalInterpretedLanguageInterfaces.html [27]: https://wolfr.am/launching-11-2 [28]: http://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn112.html [29]: http://community.wolfram.com//c/portal/getImageAttachment?filename=InOutImg82.png&userId=11733 [30]: http://community.wolfram.com//c/portal/getImageAttachment?filename=InOutImg91.png&userId=11733 Brian Wood 2017-09-14T18:59:23Z https://community.wolfram.com/groups/-/m/t/1732766 After a full 30 mins of some very intellectual talk with Mr.Stephen Wolfram, shortened my Final Project choice list from 7 to just 2. I decided to go with one of the weirdest projects in my list which was Parsing Details from Video Game footage. Now the topic literally made me infer to teach the computer how to understand the game it is showed but Mr. Stephen Wolfram thought it was maybe too easy, I agreed so we both got excited on the practical approach of the project, which was to make a computer algorithm that fully understands a footage given to it and makes a speech output of some key details in such a way that a Visually Impaired person was able to understand and play if the algorithm was to be running realtime.  While working on my project at the program I got to understand the realm of image processing in great detail. Now since image processing is comparatively hard in other languages, it became one of my greatest fears and thus I got a chance to battle it with Mathematica. The trajectory of this project kinda changed when I had a one on one with Mr. Stephen Wolfram, He suggested me to make the Computer understand each frame in such detail that even a blind person can understand it and thus It gave me an Idea on shifting my Project's Purpose from Simply annotation to Analyzing and Pre to Post-processing details. The idea was to use the annotation procedure to analyze the scenario of the game and give out Audio Response (One of the Layers of Output) Among the 3 layers of outputs, the graphical visualization of the processes of the project looked like this: ![enter image description here][1] I began working by first making the game myself on my favorite Game engine UNITY, check it out here and after that, I collected a few screenshots and Videos footages from the game. > The Reason I made the Game myself on Unity was to use the > UnityConnect[ ] in Mathematica, However for a few reasons was not > possible, I would use it later though ![enter image description here][2] The first thing to be done was to make the computer read any useful details that may change the game's state. Among them, "Ready!" and"Game Over" were a few of the Messages that halted the game until the user moves the sprite again. To start TextRecognition on this image above I first had to remove all the layers extra detail from the footage that distorted the result of the text recognition. Definitely, it was the grid behind it, I used ColorDetect on the Green color channel of the Image and deleted that which gave me a result like this: ![enter image description here][3] After Reversing the colors for a better view I applied the text recognize function and got a stable result. Now that we have found the halting point of the code we could skip these frames for program efficiency.  I nested this to an entire Function and along with that I defined a few more Custom Functions that made my code a bit easier to debug. ---------- Before we Continue,  A little Custom Functions. **My Custom Function LegacyTracker[ ]** LegacyTracker[image_, clr_, disp_, imp_] := ( \ pixelize = PixelValuePositions[ ColorNegate[Binarize[ColorDistance[image, clr], imp]], 1]; dop = CoordinateBounds[ pixelize]; \ If[disp == True, Show[image, Graphics[{EdgeForm[{Thick, Green}], Opacity[0], BoundingRegion[pixelize, "MinRectangle"]}]], dop] ) Tracks Keyed Color and return the specific keyed data from the image argument inform of an Image or Coordinates.  **My Custom Function LegacyTrackerMasker[ ]** LegacyTrackerMasker[image_, clr_, disp_, imp_] := ( pixelize = PixelValuePositions[ ColorNegate[Binarize[ColorDistance[image, clr], imp]], 1]; dop = CoordinateBounds[pixelize]; If[disp == True , Show[image, Graphics[{EdgeForm[{Thick, White}], Opacity[1], BoundingRegion[pixelize, "MinRectangle"]}]], dop] ) Masks the Keyed Color to Black in order to Perform Efficient tracking on similar Color key data ---------- I first used this function to Track the yellow life bars in the Game HUD and cropped out the middle row of the image and simply read the Pixel Values and counted each time a yellow group of pixels appeared to determine the number of lives the player has and save them to the main String which would be narrated to you each time you died. f = LegacyTracker[ LegacyTrackerMasker[footage, RGBColor[0/255, 128/255, 248/255], True, 0.1], Yellow, False, 0.1]; tst = ImageTrim[ footage, {Take[f, 1][[1]][[1]], Take[f, 2][[2]][[1]]}, {Take[f, 1][[1]][[2]], Take[f, 2][[2]][[2]]}]; {c, r} = ImageDimensions[tst]; ImageTake[RemoveAlphaChannel[tst], {Round[r/2]}, {1, c}]; count = Length[ Flatten[Length /@ Split [#] & /@ ImageData[ ImageTake[RemoveAlphaChannel[tst], {Round[r/2]}, {1, c}]]]]; If[count == 7, MainStr <> " ," <> "3 Lives Remaining", If[count == 5, MainStr <> " ," <> "2 Lives Remaining", If[count == 3, MainStr <> " ," <> "1 Life Remaining", MainStr <> " ," <> "Game Over"]]] If[SecStr == "Ready!", SpeechSynthesize[MainStr]] After this, I inverted the Previous mask and cropped the grid out and removed all the HUD elements since the have been analyzed for the frame. After that, I ran the Trackers again but now on ghosts for Calculating the angle vectors with respect to Pac-Man for processing the direction they are headed towards and if they were are a safe distance or not.  abt = False; coordEnemyRed = N[Mean[LegacyTracker[a, Red, abt, 0.4]]]; coordEnemyCyan = N[Mean[LegacyTracker[a, Cyan, abt, 0.4]]]; coordEnemyYellow = N[Mean[LegacyTracker[a, RGBColor[255/255, 201/255, 51/255], abt, 0.1]]]; coordEnemyPink = N[Mean[LegacyTracker[a, RGBColor[255/255, 153/255, 204/255], abt, 0.03]]]; coordPacMan = N[Mean[LegacyTracker[a, Yellow, abt, 0.2]]]; I took the mean since it gave me the center of the image which I later was used for calculating the vector angle of the Ghosts with respect to Pac-Man which later was mapped out to give the direction the ghosts were at. distance = Min[N[EuclideanDistance[coordPacMan, coordEnemyCyan]], N[EuclideanDistance[coordPacMan, coordEnemyYellow]], N[EuclideanDistance[coordPacMan, coordEnemyPink]], N[EuclideanDistance[coordPacMan, coordEnemyRed]]] Later, I Calculated the distance between the points and the Mean point of the enemy Mobs that ultimately decided which enemy is most probably a threat to you at the moment and accordingly the program uses a speech synthesizer to produce an output. After the Ghosts, the next thing to detect was the maze, for which I first applied a series of filters to the Cropped Image of the grid (Maze) ColorNegate[ ColorReplace[ ColorNegate[ ColorDetect[FinCrp, RGBColor[0/255, 128/255, 248/255]]], LightGray] // Normal] This ultimately gave me the output like this: ![enter image description here][4] After this, I took the PixelValue position of the White bits and coded a construct that immediately makes a pop sound whenever the Pac-Man hits a wall. Now the next thing I did was to store the coordinates of Pac-man to a List which took a history of the movement and using the existing list data and the Maze pixel values, Calculated above, The computer found probable ways for the Sprite to go and if a way was hazardous for the sprite it would also produce an output.  In similar, all the constructs work in Parallel on passes frame(s) and completely annotates it all.  ---------- At this time the Program is indeed not real-time thus ill be working on this to make it process in the background by connecting my Pacman game with UnityLink and Mathematica. A lot of development yet remains in this project but the prime goal to parse the detail from a frame has been achieved. ---------- As for my future work, I would actually like to work my way towards the accessibility computing and would take this project of mine to a much-advanced level where it actually does all of this in Real-time while producing using music as a key component to communicate with the player. Important Links:   [Project Source Code][5] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=FF.png&userId=1720422 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=1.PNG&userId=1720422 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=image.png&userId=1720422 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=aaaa.png&userId=1720422 [5]: https://github.com/n43ee7/WSS-Final-Project Nabeel Nayyar 2019-07-12T01:36:08Z