https://community.wolfram.com RSS Feed for Wolfram Community showing ideas tagged with Artificial Intelligence sorted by most viewed. https://community.wolfram.com/groups/-/m/t/995095 ![enter image description here][18] The long evolutionary history of humans has optimized the recognition of animals in the human vision system. To escape predators and to find food. Sometimes we believe to see animals and human faces, even if there aren't any, this phenomena is called [pareidolia][1], or more generally apophenia. Classic examples are Jesus on a toast ([Liu2014][2]), the image on the shroud of Turin ([Sheen2016][3]), Elvis in a potato chip ([Voss2011][4]), and the face on Mars ([Martinez-Conde2012][5]). In many cases, in about 100ms ([Naber2012][6]) humans can identify animal shapes. Some animals, e.g. snakes are potentially identified much faster ([VanLe2013][7]). So, an interesting question might be: how often do we (believe to) see animals in an genuinely random image? Here is, in the literal sense of the word, a random image of size 300*300 pixels: SeedRandom[4056380]; dim=300; randomImage = Image[Table[RandomChoice[{2,1}->{0,1}],{dim},{dim}]]; Show[randomImage,ImageSize -> 300] ![enter image description here][8] No animal shapes are directly visible. Using ImageMesh we can get the individual/connected components (some potentially animal-like shaped) as regions (this operation is not fully deterministic, so results below may vary). Show[imesh=ImageMesh[randomImage,Method->"DualMarchingSquares"],ImageSize -> 300] ![enter image description here][9] These are the individual shapes. We ignore too small and too large regions. For our image we find neatly 400 regions. (* this takes a minute *) shapes=Select[ConnectedMeshComponents@imesh,18< Area[#]<2000&]; Length[shapes] `393` These are the shapes positioned in the original image. Show[HighlightMesh[#,Style[2, RandomColor[]]]&/@ shapes] ![enter image description here][10] For later use, we define one function to make a random image and get the shapes. getShapes[{black_,white_}, dim_] :=Select[ConnectedMeshComponents@ ImageMesh[Image[Table[RandomChoice[{black,white}->{0,1}],{dim},{dim}]], Method->"DualMarchingSquares"],18< Area[#]<2000&] As we want to interpret the shapes as animals, we smooth the edges. We switch from regions to graphics. smooth[reg_] :=Graphics[{Lighter[Blue], (ToExpression[ToString[InputForm@reg],StandardForm,Hold] /. HoldPattern[BoundaryMeshRegion[v_,b__,___Rule]]:>GraphicsComplex[v,FilledCurve[{b}/. Line[l_]:> BSplineCurve[DeleteDuplicates[Flatten[l,1]], SplineClosed->True,SplineDegree->2]]])[[1]]}] We also allow changing the orientation and color. rotate[g_,?_] := With[{m={{Cos[?],Sin[?]},{-Sin[?], Cos[?]}}},g/. GraphicsComplex[v_, r__]:> GraphicsComplex[m.#&/@ v,r]] flip[g_] := g/.GraphicsComplex[v_, r__]:> GraphicsComplex[{-1,1}#&/@ v,r] recolor[g_,col_] := g/.c_RGBColor:>col animalize[g_,{f_,?_,col_}] := Graphics[ recolor[rotate[If[f, flip,Identity]@ smooth[g],?],col], PlotRange -> All]&[smooth[g]] Here are some of the shapes that were hiding in the above image.The human eye tries to see fishes, wales, birds, squirrels, frogs, ducks, sea-horses, bunnies, deers, dogs, cats and similar creatures as well as human heads. GraphicsGrid[Partition[Show[#, ImageSize -> 120,Frame->True,FrameTicks->False]&/@#,5]&@ { animalize[shapes[[18]],{False,1.8,GrayLevel[0.1]}], animalize[shapes[[99]],{False,1.1,Darker[Yellow]}], animalize[shapes[[6]],{False,0.7,Darker[Blue]}], animalize[shapes[[14]],{False,0.,Darker[Brown]}], animalize[shapes[[4]],{False,0.5,Darker[Red]}], animalize[shapes[[23]],{False,0,Darker[Orange]}], animalize[shapes[[76]],{False,0,Darker[Orange,0.6]}], animalize[shapes[[95]],{False,-1.5,Darker[Green,0.4]}], animalize[shapes[[9]],{False,0.,Brown}], animalize[shapes[[163]],{False,1.1,Darker[Purple]}]}, Spacings->{5,-20}] ![enter image description here][11] Looking at the last selection shows the importance of one or two eyes in the shapes ([Yang2015][12]). Eyes are often used by nature for deception ([Steven2014][13]). A small Manipulate will allow to optimize the animal-perception by changing color, orientation, and aspect ratio. makeManipulate[shape_]:=Manipulate[Graphics[ recolor[rotate[If[f, flip,Identity]@ #[[1]],?],col], ImageSize -> 120,AspectRatio->ar,PlotRange -> All], OpenerView[{"modify",Column[{ Control[{{?,0,"rotate"},-Pi,Pi,ImageSize->Small}], Control[{{f,False,"reflect"},{True,False},ImageSize->Small}], Control[{{col, Darker[Blue],"color"},Red,ImageSize->Tiny}], Control[{{ar, Automatic,"aspect ratio"},0.2,5,ImageSize->Small}]}]}], SaveDefinitions->True,TrackedSymbols:>True]&[smooth[#]]&@ shape Manipulate[makeManipulate[shapes[[j]]],{{j,141,""},1, Length[shapes],1,Appearance->"Labeled"}] ![enter image description here][14] Here are some more animal-shapes found in other random images. ![enter image description here][15] One could now try to automate the finding of animals using ImageIdentify. In general, the interpretation of ImageIdentify will depend on the image orientation. So, we maximize over different orientations. getOptimalAnimal[shape_] := Module[{s=smooth[shape][[1]],tab,ras,tab2}, tab=Table[{(ras=Rasterize[Graphics[rotate[s,\[Alpha]],ImageSize -> 120,PlotRange -> All]]), Normal@KeyDrop[ImageIdentify[ras,"animal",2,"Probability"],Entity["Concept",#]&/@ {"Person::93r37","Hominid::tt93h","Mammal::nt5bz", "SeaStar::46w97","Shark::632s8","CanisFamiliaris::597qc"}]}, {\[Alpha], 0, 2Pi, 2Pi/24}]; tab2=Sort[{#2[[1,2]], #2[[1,1]], #1}&@@@DeleteCases[ tab,{_,{}}]]; {tab2[[-1,3]]->tab2[[-1,2]],"Probabilty" -> tab2[[-1,1]]}] As ImageIdentify was trained on real photographs, it is not the ideal tool for this task. It is biased towards humans, mammals, and sharks. getOptimalAnimal[shapes[[10]]] ![enter image description here][16] getOptimalAnimal[shapes[[27]]] ![enter image description here][17] To answer the original question: in a random 400x400 pixel image, one easily finds a few dozen animals. Here are four dozen animal-shape like smoothed regions from random 400x400 image. Module[{gs, gs2}, SeedRandom[2222]; gs = getShapes[{2, 1}, 400]; gs2 = {animalize[gs[[3]], {False, 0, GrayLevel[0.1]}], animalize[gs[[6]], {False, 1, GrayLevel[0.1]}], animalize[gs[[9]], {False, 1.3, GrayLevel[0.1]}], animalize[gs[[15]], {False, -1, GrayLevel[0.1]}], animalize[gs[[17]], {False, 0.3, GrayLevel[0.1]}], animalize[gs[[30]], {True, 2.3, GrayLevel[0.1]}], animalize[gs[[39]], {False, 0.3, GrayLevel[0.1]}], animalize[gs[[40]], {True, -0.1, GrayLevel[0.1]}], animalize[gs[[48]], {False, -0.1, GrayLevel[0.1]}], animalize[gs[[53]], {True, -0.1, GrayLevel[0.1]}], animalize[gs[[55]], {True, -0.1, GrayLevel[0.1]}], animalize[gs[[62]], {True, 2.3, GrayLevel[0.1]}], animalize[gs[[64]], {False, 0, GrayLevel[0.1]}], animalize[gs[[65]], {False, 0, GrayLevel[0.1]}], animalize[gs[[68]], {False, -0.2, GrayLevel[0.1]}], animalize[gs[[73]], {True, -0.3, GrayLevel[0.1]}], animalize[gs[[81]], {False, 0.2, GrayLevel[0.1]}], animalize[gs[[86]], {True, 3.4, GrayLevel[0.1]}], animalize[gs[[97]], {False, 0.3, GrayLevel[0.1]}], animalize[gs[[100]], {False, 0.9, GrayLevel[0.1]}], animalize[gs[[101]], {False, 2.6, GrayLevel[0.1]}], animalize[gs[[115]], {False, 4.3, GrayLevel[0.1]}], animalize[gs[[124]], {False, 0, GrayLevel[0.1]}], animalize[gs[[125]], {False, 2.9, GrayLevel[0.1]}], animalize[gs[[139]], {False, 0, GrayLevel[0.1]}], animalize[gs[[146]], {False, -1.6, GrayLevel[0.1]}], animalize[gs[[175]], {False, -1., GrayLevel[0.1]}], animalize[gs[[201]], {True, -0.5, GrayLevel[0.1]}], animalize[gs[[209]], {True, 3.1, GrayLevel[0.1]}], animalize[gs[[210]], {False, -1.4, GrayLevel[0.1]}], animalize[gs[[218]], {True, 3.2, GrayLevel[0.1]}], animalize[gs[[227]], {True, -0.3, GrayLevel[0.1]}], animalize[gs[[229]], {True, 0.9, GrayLevel[0.1]}], animalize[gs[[253]], {True, 0., GrayLevel[0.1]}], animalize[gs[[273]], {True, -0.3, GrayLevel[0.1]}], animalize[gs[[278]], {True, 1., GrayLevel[0.1]}], animalize[gs[[282]], {True, 3, GrayLevel[0.1]}], animalize[gs[[298]], {False, 0, GrayLevel[0.1]}], animalize[gs[[299]], {False, 0.6, GrayLevel[0.1]}], animalize[gs[[318]], {False, 2., GrayLevel[0.1]}], animalize[gs[[326]], {False, 3, GrayLevel[0.1]}], animalize[gs[[340]], {False, -1.3, GrayLevel[0.1]}], animalize[gs[[352]], {True, -13, GrayLevel[0.1]}], animalize[gs[[361]], {True, -13, GrayLevel[0.1]}], animalize[gs[[365]], {False, 2.4, GrayLevel[0.1]}], animalize[gs[[377]], {True, 2.6, GrayLevel[0.1]}], animalize[gs[[395]], {False, 0.3, GrayLevel[0.1]}], animalize[gs[[405]], {True, 0.3, GrayLevel[0.1]}]}; Grid[Partition[Framed[Show[#, ImageSize -> 100]] & /@ SortBy[gs2, FullOptions[#, AspectRatio] &], 6]]] ![enter image description here][18] Using different method options for ImageMesh gives different results. Here are some animal-shaped regions from a 300*300 image and method "MarchingSquares". ![enter image description here][19] [1]: https://en.wikipedia.org/wiki/Pareidolia [2]: https://dx.doi.org/10.1016/j.cortex.2014.01.013 [3]: https://doi.org/10.1163/15736121-12341320 [4]: http://dx.doi.org/10.1093/cercor/bhr315 [5]: https://www.scientificamerican.com/article/a-faithful-resemblance/ [6]: http://dx.doi.org/10.1167/12.1.25 [7]: http://dx.doi.org/10.1073/pnas.1312648110 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=4356yretet465yhdfreg5w4ytrg.png&userId=11733 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fdfgnsgt65whtr.png&userId=11733 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=dfgdhcvbcbcx34tq.png&userId=11733 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=dfg4dsgdfsgdqy546hfgsnbv.png&userId=11733 [12]: http://arxiv.org/pdf/1509.04954 [13]: http://dx.doi.org/10.1093/czoolo/60.1.26 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=weretregdfdgdscbx3454y6rtdfgsb.png&userId=11733 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=sadfert543wyhdfgdhfd5676e565w44.png&userId=11733 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=67567etfhg54yw.png&userId=11733 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=sadf345q4yrthgd.png&userId=11733 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=kfkeighsmf534yrthgfds.png&userId=11733 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fdsg65udfsdstrdgjyt7u56uw56rh.png&userId=11733 Michael Trott 2017-01-15T10:54:26Z https://community.wolfram.com/groups/-/m/t/1093926 ![enter image description here][16] # Introduction Back in [Wolfram Summer School 2016][1] I worked on the project "Image Transformation with Neural Networks: Real-Time Style Transfer and Super-Resolution", which got later [published on Wolfram Community][2]. At the time I had to use the MXNetLink package, but now all the needed functionality is built-in, so here is a top-level implementation of artistic style transfer with Wolfram Language. This is a slightly simplified version of the original method, as it uses a single VGG layer to extract the style features, but a full implementation is of course possible with minor modifications to the code. You can also find this example in the docs: [NetTrain][3] >> Applications >> Computer Vision >> Style Transfer # Code Create a new image with the content of a given image and in the style of another given image. This implementation follows the method described in Gatys et al., *A Neural Algorithm of Artistic Style*. An example content and style image: ![enter image description here][4] To create the image which is a mix of both of these images, start by obtaining a pre-trained image classification network: vggNet = NetModel["VGG-16 Trained on ImageNet Competition Data"]; Take a subnet that will be used as a feature extractor for the style and content images: featureNet = Take[vggNet, {1, "relu4_1"}] ![enter image description here][5] There are three loss functions used. The first loss ensures that the "content" is similar in the synthesized image and the content image: contentLoss = NetGraph[{MeanSquaredLossLayer[]}, {1 -> NetPort["LossContent"]}] ![enter image description here][6] The second loss ensures that the "style" is similar in the synthesized image and the style image. Style similarity is defined as the mean-squared difference between the Gram matrices of the input and target: gramMatrix = NetGraph[{FlattenLayer[-1], TransposeLayer[1 -> 2], DotLayer[]}, {1 -> 3, 1 -> 2 -> 3}]; styleLoss = NetGraph[{gramMatrix, gramMatrix, MeanSquaredLossLayer[]}, {NetPort["Input"] -> 1, NetPort["Target"] -> 2, {1, 2} -> 3, 3 -> NetPort["LossStyle"]}] ![enter image description here][7] The third loss ensures that the magnitude of intensity changes across adjacent pixels in the synthesized image is small. This helps the synthesized image look more natural: l2Loss = NetGraph[{ThreadingLayer[(#1 - #2)^2 &], SummationLayer[]}, {{NetPort["Input"], NetPort["Target"]} -> 1 -> 2}]; tvLoss = NetGraph[<| "dx1" -> PaddingLayer[{{0, 0}, {1, 0}, {0, 0}}, "Padding" -> "Fixed" ], "dx2" -> PaddingLayer[{{0, 0}, {0, 1}, {0, 0}}, "Padding" -> "Fixed"], "dy1" -> PaddingLayer[{{0, 0}, {0, 0}, {1, 0}}, "Padding" -> "Fixed" ], "dy2" -> PaddingLayer[{{0, 0}, {0, 0}, {0, 1}}, "Padding" -> "Fixed"], "lossx" -> l2Loss, "lossy" -> l2Loss, "tot" -> TotalLayer[]|>, {{"dx1", "dx2"} -> "lossx", {"dy1", "dy2"} -> "lossy", {"lossx", "lossy"} -> "tot" -> NetPort["LossTV"]}] ![enter image description here][8] Define a function that creates the final training net for any content and style image. This function also creates a random initial image: createTransferNet[net_, content_Image, styleFeatSize_] := Module[{dims = Prepend[3]@Reverse@ImageDimensions[content]}, NetGraph[<| "Image" -> ConstantArrayLayer["Array" -> RandomReal[{-0.1, 0.1}, dims]], "imageFeat" -> NetReplacePart[net, "Input" -> dims], "content" -> contentLoss, "style" -> styleLoss, "tv" -> tvLoss|>, {"Image" -> "imageFeat", {"imageFeat", NetPort["ContentFeature"]} -> "content", {"imageFeat", NetPort["StyleFeature"]} -> "style", "Image" -> "tv"}, "StyleFeature" -> styleFeatSize ] ] Define a [NetDecoder][9] for visualizing the predicted image: meanIm = NetExtract[featureNet, "Input"][["MeanImage"]] > {0.48502, 0.457957, 0.407604} decoder = NetDecoder[{"Image", "MeanImage" -> meanIm}] ![enter image description here][10] The training data consists of features extracted from the content and style images. Define a feature extraction function: extractFeatures[img_] := NetReplacePart[featureNet, "Input" ->NetEncoder[{"Image", ImageDimensions[img], "MeanImage" ->meanIm}]][img]; Create a training set consisting of a single example of a content and style feature: trainingdata = <| "ContentFeature" -> {extractFeatures[contentImg]}, "StyleFeature" -> {extractFeatures[styleImg]} |> Create the training net whose input dimensions correspond to the content and style image dimensions: net = createTransferNet[featureNet, contentImg, Dimensions@First@trainingdata["StyleFeature"]]; When training, the three losses are weighted differently to set the relative importance of the content and style. These values might need to be changed with different content and style images. Create a loss specification that defines the final loss as a combination of the three losses: perPixel = 1/(3*Apply[Times, ImageDimensions[contentImg]]); lossSpec = {"LossContent" -> Scaled[6.*10^-5], "LossStyle" -> Scaled[0.5*10^-14], "LossTV" -> Scaled[20.*perPixel]}; Optimize the image using [NetTrain][11]. [LearningRateMultipliers][12] are used to freeze all parameters in the net except for the [ConstantArrayLayer][13]. The training is best done on a GPU, as it will take up to an hour to get good results with CPU training. The training can be stopped at any time via Evaluation -> Abort Evaluation: trainedNet = NetTrain[net, trainingdata, lossSpec, LearningRateMultipliers -> {"Image" -> 1, _ -> None}, TrainingProgressReporting -> Function[decoder[#Weights[{"Image", "Array"}]]], MaxTrainingRounds -> 300, BatchSize -> 1, Method -> {"ADAM", "InitialLearningRate" -> 0.05}, TargetDevice -> "GPU" ] ![enter image description here][14] Extract the final image from the [ConstantArrayLayer][15] of the trained net: decoder[NetExtract[trainedNet, {"Image", "Array"}]] ![enter image description here][16] [1]: https://education.wolfram.com/summer/school/alumni/2016/salvarezza/ [2]: http://community.wolfram.com/groups/-/m/t/885941 [3]: http://reference.wolfram.com/language/ref/NetTrain.html [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=I_432.png&userId=95400 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_179.png&userId=95400 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_180.png&userId=95400 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_181.png&userId=95400 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_182.png&userId=95400 [9]: http://reference.wolfram.com/language/ref/NetDecoder.html [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_184.png&userId=95400 [11]: http://reference.wolfram.com/language/ref/NetTrain.html [12]: http://reference.wolfram.com/language/ref/LearningRateMultipliers.html [13]: http://reference.wolfram.com/language/ref/ConstantArrayLayer.html [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_185.png&userId=95400 [15]: http://reference.wolfram.com/language/ref/ConstantArrayLayer.html [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=I_466.png&userId=95400 Matteo Salvarezza 2017-05-15T10:33:59Z https://community.wolfram.com/groups/-/m/t/3288988 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/390b3f6c-6c28-477b-9f08-d74f7f24ea7d Paul Abbott 2024-10-01T17:36:50Z https://community.wolfram.com/groups/-/m/t/884348 The aim of my project for the Wolfram Science Summer School was to build a neural network which could be able to colorize grayscale images in a realistic way. The network has been built following the article [1]. In this paper, the authors propose a fully automated approach for colorization of grayscale images, which uses a combination of global image features, which are extracted from the entire image, and local image features, which are computed from small image patches. Global priors provide information at an image level such as whether or not the image was taken indoors or outdoors, whether it is day or night, etc., while local features represent the local texture or object at a given location. By combining both features, it's possible to leverage the semantic information to color the images without requiring human interaction. The approach is based on Convolutional Neural Networks, which have a strong capacity for learning and is trained to predict the chrominance of a grayscale image using the CIE L*a*b* colorspace. Predicting colors has the nice property that training data is practically free: any color photo can be used as a training example. **Net Layers** The model consists of four main components: a low-level features network, a mid-level features network, a global features network, and a colorization network. First, a common set of shared low-level features are extracted from the image. Using these features, a set of global image features and mid-level image features are computed. Then, the mid-level and the global features are both fused by a "fusion layer" and used as the input to a colorization network that outputs the final chrominance map. Each layer has a ReLu transfer function except for the last convolution of the colorization network, where a sigmoid function is applied.The model is able to process images of any size, but it is most efficient when the input images are 224x224 pixels, as the shared low-level features layers can share outputs. Note that when the input image size is of a different resolution, while the low-level feature weights are shared, a rescaled image of size 224x224 must be used for the global features network.This requires processing both the original image and the rescaled image through the low-level features network, increasing both memory consumption and computation time. For this reason, we trained the model exclusively with images of size 224x224 pixels. *Low-Level Features Network* A 6-layer Convolutional Neural Network obtains low-level features directly from the input image. The convolution filter bank the network represents are shared to feed both the global features network and the mid-level features network. In order to reduce the size of the feature maps, we use convolution layers with increased strides instead of using max-pooling layers (as usual for similar kinds of networks). If padding is added to the layer, the output is effectively half the size of the input layer. We used 3x3 convolution kernels exclusively and a padding of 1x1 to ensure the output is the same size (or half if using a stride of 2) as the input. *Global Features Network* The global image features are obtained by further processing the low-level features with four convolutional layers followed by three fully-connected layers.This results in a 256-dimensional vector representation of the image. *Mid-Level Features Network* The mid-level features are obtained by processing the low-level features further with two convolutional layers. The output is bottlenecked from the original 512-channel low-level features to 256-channel mid-level features. Unlike the global image features, the low-level and mid-level features networks are fully convolutional networks, such that the output is a scaled version of the input. *Fusion Layer* In order to be able to combine the global image features, a 256-dimensional vector, with the (mid-level) local image features, a 28x28x256-dimensional tensor, the authors introduce a fusion layer. This can be thought of as concatenating the global features with the local features at each spatial location and processing them through a small one-layer network.This effectively combines the global feature and the local features to obtain a new feature map that is, as the mid-level features, a 3D volume. *Colorization Network* Once the features are fused, they are processed by a set of convolutions and upsampling layers, which use the nearest neighbour technique so that the output is twice as wide and twice as tall. These layers are alternated until the output is half the size of the original input. The output layer of the colorization network consists of a convolutional layer with a Sigmoid transfer function that outputs the chrominance of the input grayscale image. Finally, the computed chrominance is upsampled and combined with the input intensity/luminance image to produce the resulting color image. In order to train the network, we used the Mean Square Error (MSE) criterion. Given a color image for training, the input of the model is the grayscale image while the target output is the a*b* components of the CIE L*a*b* colorspace. The a*b* components are globally normalized so they lie in the [0,1] range of the Sigmoid transfer function. *Colorization with Classification* While training with only color images using the MSE criterion does give good performance, sometimes it could make obvious mistakes due to not properly learning the global context of the image, e.g., whether it is indoors or outdoors. As learning these networks is an non-convex problem, we facilitated the optimization by also training for classification jointly with the colorization. As we trained the model using a large-scale dataset for classification of N classes (Mathematica ImageIdentify dataset), we had classification labels available for training. These labels correspond to a global image tag and thus can be used to guide the training of the global image features. We did this by introducing another very small neural network that consists of two fully-connected layers: a hidden layer with 256 outputs and an output layer with as many outputs as the number of classes in the dataset. The input of this network is the second to last layer of the global features network with 512 outputs. We trained this network using the cross-entropy loss, jointly with the MSE loss for the colorization network. *Implementation* The aim of my project was to build the network described in the paper using the new NeuralNetworks framework of Mathematica 11. In order to achieve this, some adjustments were needed. First of all, we decided to train and evaluate the network only on images of 224x224 pixels size, in order to use (and train) only one low-level features network, instead of two with shared weights and different outputs. The final network has two inputs: the first one is the colored 224x224 px image, encoded by the "NetEncoder" function in LAB colorspace, the second one the class of the image. The two outputs (named "Loss" and "Output") represent the values of the two loss function used (one for the colorization, the other one for the classification), which are then summed together by the NetTrain function. The three color channels of the input image are split by the split layer: the L channel feeds the "low-level features" network, while the a,b channels are scaled and concatenated in order to obtain a target set for the mean squared loss function comparable with the output of the colorization network. The fusion layer has been replaced by a broadcast layer, which joins the rank 3- tensor, output of the mid-level network, with the vector from the global features network. However, the way they are combined is not exactly the same as the one described in the paper. To evaluate the trained network on a grayscale image it's necessary to drop some branches of the network, such as the classification network and the layers that process the a,b channels of the colored input image in order to produce the target set for the colorization loss function. ![Network described in the paper][1] ![Network implementation with Mathematica NeuralNetworks framework][2] **Results** ![enter image description here][3] **Conclusions** The network described in the paper has been trained on the Places scene dataset [Zhou et al. 2014], which consists of 2,448,872 training images and 20,500 validation images, with 205 classes corresponding to the types of the scene. They filtered the images by removing grayscale images and those that have little color variance with a small automated script. They trained using a batch size of 128 for 200,000 iterations corresponding to roughly 11 epochs. This takes roughly 3 weeks on one core of a NVIDIAR TeslaR K80 GPU. We needed to introduce some new layers in the existing framework and to fix some bugs, so we were able to train our network only for 14 hours on a dataset of 350000 images on one core of a GPU Titan machine. Furthermore, the images in our training set mainly represent specific items, so probably better results may be achieved introducing also images of different types of subjects (landscapes, human created images, indoors, etc). The results we obtained are showed in the section above and are quite good. We are confident that with a deeper and longer training our network would give considerably better results. **Open Problems / Future Developments** Due to the separation between the global and local features, it is possible to use global features computed on one image in combination with local features computed on another image, to change the style of the resulting colorization. One of the more interesting things the model can do is adapting the colorization of one image to the style of another. This is straight-forward to do with this model due to the decorrelation between the global features and the mid-level features. In order to colorize an image A using the style taken from an image B, it's necessary to compute the mid-level local features of image A and the global features from image B. Than it's possible to fuse these features and process them with the colorization network. Both the local and the global features are computed from grayscale images: it's not necessary to use any color information at all. The main limitation of the method lies in the fact that it is datadriven and thus will only be able to colorize images that share common properties with those in the training set. In order to evaluate on significantly different types of images, it would be necessary to train a the model for all type of images (indoor, outdoor, human-created...). In order to obtain good style transfer results, it is important for both images to have some semantic level of similarity between them. **References** [1] Satoshi Iizuka, Edgar Simo - Serra, and Hiroshi Ishikawa."Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification". [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=netwPicture.png&userId=884315 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MyNetwork.png&userId=884315 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=result2.png&userId=884315 Sabrina Giollo 2016-07-07T19:53:19Z https://community.wolfram.com/groups/-/m/t/2108301 # Purpose and Methods: *Some time ago I published on the WFR (Wolfram Function Repository) the function [MusicalScaleSample][1], where a sound scale sample can be generated. Which can be 30 different scales possible with this WFR item. But how to know which scale was the origin that generated the samples themselves? This is what this work is about, where I approach some methods, from semi-mechanics to machine learning with the comment on the results afterwards.* The methods covered here have this order below (of course, there are many other methods, here some in a briefly addressed way and others more detailed): - Semi-mechanical / Pitch Frequency - Pitch Sound Note - Classify / Method Neural Net - Classify / Method Naive Bayes - Customized ResNet These methods are (more or less) in the order of machine complexity x efficiency, the latter, Customized ResNet, was the method that generated the best results (as we going to see). In the course of this work I will comment on each method, it´s limitations and results. # Initial Samples / Sound: The musical scales follow patterns that can be represented by the distances between your notes, so, as a first step, we have these distances below: sequences = {"MajorPentatonic" -> {0, 2, 2, 3, 2}, "MinorPentatonic" -> {0, 3, 2, 2, 3}, "NeopolitanMajor" -> {0, 1, 2, 2, 2, 2, 2}, "NeopolitanMinor" -> {0, 1, 2, 2, 2, 1, 3}, "HungarianMajor" -> {0, 3, 1, 2, 1, 2, 1}, "HungarianMinor" -> {0, 2, 1, 3, 1, 1, 3}, "HungarianGypsy" -> {0, 2, 2, 2, 1, 1, 2}, "HarmonicMajor" -> {0, 2, 2, 1, 2, 1, 3}, "HarmonicMinor" -> {0, 2, 1, 2, 2, 1, 3}, "DoubleHarmonic" -> {0, 1, 3, 1, 2, 1, 3}, "JazzMinor" -> {0, 2, 1, 2, 2, 2, 2}, "Overtone" -> {0, 2, 2, 2, 1, 2, 1}, "BluesScale" -> {0, 3, 2, 1, 1, 3}, "Chromatic" -> {0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, "WholeTone" -> {0, 2, 2, 2, 2, 2}, "DiminishedWholeTone" -> {0, 1, 2, 1, 2, 2, 2}, "Symmetrical" -> {0, 1, 2, 1, 2, 1, 2, 1}, "Arabian" -> {0, 2, 2, 1, 1, 2, 2}, "Balinese" -> {0, 1, 2, 4, 1}, "Byzantine" -> {0, 1, 3, 1, 2, 1, 3}, "Persian" -> {0, 1, 3, 1, 1, 2, 3}, "EastIndianPurvi" -> {0, 1, 3, 2, 1, 1, 3}, "Oriental" -> {0, 1, 3, 1, 1, 3, 1}, "GagakuRyoSenPou" -> {0, 2, 2, 3, 2}, "Zokugaku" -> {0, 3, 2, 2, 3}, "InSenPou" -> {0, 1, 4, 2, 1}, "Okinawa" -> {0, 4, 1, 2, 4}, "Enigmatic" -> {0, 1, 3, 2, 2, 2, 1}, "EightToneSpanish" -> {0, 1, 2, 1, 1, 1, 2, 2}, "Prometheus" -> {0, 2, 2, 2, 3, 1}}; Below, is a sample of a scale, in this case "Enigmatic", as well as its signature through the PitchRecognize command represented in a ListLinePlot chart. It is worth noting that the duration of each note is relatively short, so that the peaks of the notes do not extend like a hill, but rather more defined peaks so that there are better chances of being recognized: Sound@Table[ ResourceFunction["MusicalScaleSample"]["Enigmatic", "Tone" -> "C3", "NoteInterval" -> 0.15, "Playing" -> "Mechanical"], 5] ListLinePlot[PitchRecognize[%], ColorFunction -> Function[{x, y}, Hue[y]], PlotStyle -> Thin, AxesStyle -> Directive[Black, 12]] ![im1][2] # Semi-mechanical / Pitch Frequency First, a function is developed that generates the musical note through PitchRecognize/Frequency; the equation itself is mechanical, although the recognition command already has characteristics of neural networks associated: notes[x_] := Keys@Counts[ Round[-36 + (12 Log[1/55 2^(3/4) #])/Log[2]] & /@ QuantityMagnitude@ DeleteMissing@Values@PitchRecognize[x, "Frequency"]] Below is the conversion of the intervals of the notes in the notes themselves (taking into account 7 different octaves): numb = Table[-48 + # & /@ Flatten[# + Accumulate[Values@sequences[[x]]] & /@ {0, 12, 24, 36, 48, 60, 72}] -> Keys[sequences[[x]]], {x, 1, 30}]; Testing the first method with a “Enigmatic” class scale sample, where we will get the musical notes and the prediction in the results. The result gives values and weights for both notes found and anti-values for notes not found on the scale. The result is this: ![im2][3] Table[Values@numb[[i]] -> Subtract[Values[Counts[MemberQ[Keys@numb[[i]], #] & /@ test]][[1]], Values[Counts[MemberQ[test, #] & /@ Keys@numb[[i]]]][[1]]], {i, 1, 30}] // Dataset[MaximalBy[#, Values, 5]] & ![im3][4] Note that in the case above the scale was correctly identified, but this method is very fragile and for most scales and attempts the result is quite unsatisfactory. # Pitch Sound Note This next method relies more on the ability to recognize of the PitchRecognize command to recognize the frequencies, compared to the first method. The result is also given in valued scores and a prediction of which scale was used (in this case, a “Enigmatic” sample was also used): ![im4][5] Table[Values@numb[[i]] -> Subtract[ Values[Counts[MemberQ[Keys@numb[[i]], #] & /@ test2]][[1]], Values[Counts[MemberQ[test2, #] & /@ Keys@numb[[i]]]][[1]]], {i, 1, 30}] // Dataset[MaximalBy[#, Values, 5]] & ![im5][6] Realize that the result was identical between the first two methods discussed, at least for the musical scale “Enigmatic” sample given. Normally, this method is also not very reliable and most of the time (with most musical scales) is wrong, which was tested, although in this case above I got it right. # Classify / Method Neural Net We will use a Neural Net classification method in this next attempt, to see if by this specific method the result is more satisfactory when working with Audio. First, we will generate a total of 50 samples of each type of musical scale class. Each sample in this case is 25.5 seconds in size formed with 5 random reproductions sequence by the specific scale. The tone will be maintained in all tests in “C3” with two octaves (default) and the playing method will always be mechanical for each note to have the same exact duration: n = 50; samp1 = RandomSample@ Flatten@Table[(Sound@ Table[ResourceFunction["MusicalScaleSample"][#, "Tone" -> "C3", "NoteInterval" -> 0.15, "Playing" -> "Mechanical"], 5] -> #) & /@ Keys@sequences, n]; Above, “samp1” was generated with 1500 samples, then changed to “stest1” with 300 samples to test the result of the classifiers. Now we can use Classify, as well as test it for the result and accuracy of the method: c2 = Classify[samp1, Method -> "NeuralNetwork"] ![im6][7] ClassifierMeasurements[c2, stest1, "Accuracy"] ![im7][8] As we can observe above, this method is far from being the best method to predict which musical scale in the case, it only gave a value of 8% accuracy! # Classify / Method Naive Bayes This next method is another option for the Classify function, an option that usually works best with Audio. Below we will see the result of the classifier with its prediction and accuracy: c3 = Classify[samp1, Method -> "NaiveBayes"] ![im8][9] ClassifierMeasurements[c3, stest1, "Accuracy"] ![im9][10] We see that by this method above (Naive Bayes), the result was already better to predict the original musical scale used, although it is still far from ideal, especially for some of the musical scales classes. # Customized ResNet The last method I tested in this work is by far the best method. First big difference is the transformation (Fourier) of Audio to its spectrum, since the classification of images proved to be a more powerful tool in this case. Below is an example of an Audio sample and its corresponding image to be classified by Machine Learning process: ![im10][11] In order to generate the samples to be performed in the machine learning, some small modifications were made: - Instead of 5 reproductions followed by the scale in each sample, 10 reproductions were made, because more reproductions, it becomes clearer which is the identity of the scale. - A slightly higher value of the note interval was also used to make it easier to identify the sound of each note, this value was adapted together with the reproductions followed, so that, each sample has an exact 60 seconds of duration. - Another change was to directly specify the RasterSize and ImageSize for each sample to have a slightly better definition. Below, 100 samples of each scale class were generated, totaling 3000 samples (the time for this process was approximately 3 hours and the sample set size was just over 1GB): n = 100; samp = RandomSample@ Flatten@Table[(Rasterize[ Spectrogram[ Sound@Table[ ResourceFunction["MusicalScaleSample"][#, "Tone" -> "C3", "NoteInterval" -> 0.17646, "Playing" -> "Mechanical"], 10], Frame -> None], RasterSize -> 600, ImageSize -> 300] -> #) & /@ Keys@sequences, n]; The generator above was used also for a set: "stest" with 30 samples from each class totaling 900 total samples and a smaller set "stest2" of 15 samples from each class totaling 450 total samples. To better store these sample sets above I used DumpSave and the .MX file format: filepath2 = FileNameJoin[{NotebookDirectory[], "samp.mx"}]; DumpSave[filepath2, {samp}]; filepath3 = FileNameJoin[{NotebookDirectory[], "stest.mx"}]; DumpSave[filepath3, {stest}]; filepath4 = FileNameJoin[{NotebookDirectory[], "stest2.mx"}]; DumpSave[filepath4, {stest2}]; Quit[] We can retrieve the sets with the simple Get command: SetDirectory[NotebookDirectory[]]; Get["samp.mx"] Get["stest.mx"] Get["stest2.mx"] Example of how the main set contains the samples and their definitions: samp[[5 ;; 8]] ![im11][12] We will now start to develop the customized NetChain. First, we create a directory for the classifier batches: dirModel = CreateDirectory[]; Than, we downloaded the ResNet, which in this case I used the **ResNet-152** from [Wolfram Neural Repository][13], but we discarded the last 3 layers in order to be able to customize the output and some characteristics of behavior: tempNet = Take[NetModel[ "ResNet-152 Trained on ImageNet Competition Data"], {1, -4}] ![im12][14] We added some layers to increase the complexity and so that it can learn new features. Just as we defined, the 30 different classes of different musical scales to be used by the neural network: newNet = NetChain[<|"pretrainedNet" -> tempNet, "Flatten" -> FlattenLayer[], "Dropout1" -> DropoutLayer[0.2], "linear" -> LinearLayer[1500], "Dropout2" -> DropoutLayer[0.2], "linear2" -> LinearLayer[30], "softmax" -> SoftmaxLayer[]|>, "Output" -> NetDecoder[{"Class", {"MajorPentatonic", "MinorPentatonic", "NeopolitanMajor", "NeopolitanMinor", "HungarianMajor", "HungarianMinor", "HungarianGypsy", "HarmonicMajor", "HarmonicMinor", "DoubleHarmonic", "JazzMinor", "Overtone", "BluesScale", "Chromatic", "WholeTone", "DiminishedWholeTone", "Symmetrical", "Arabian", "Balinese", "Byzantine", "Persian", "EastIndianPurvi", "Oriental", "GagakuRyoSenPou", "Zokugaku", "InSenPou", "Okinawa", "Enigmatic", "EightToneSpanish", "Prometheus"}}]] ![im13][15] The network is ready to be trained. Below is an example of its learning: trainedNet = NetTrain[newNet, samp, ValidationSet -> stest, TrainingProgressMeasurements -> {"ConfusionMatrixPlot", "Accuracy", "Precision", "Recall"}, TrainingProgressCheckpointing -> {"Directory", dirModel}, BatchSize -> 16, MaxTrainingRounds -> 2] ![im14][16] This was the final result of the learning: ![im15][17] We can save the resulting NetChain also in an .MX file with DumpSave so that we can use it when necessary: filepath5 = FileNameJoin[{NotebookDirectory[], "trainedNet.mx"}]; DumpSave[filepath5, {trainedNet}]; With the code below we have, for example, the measures of global accuracy and the result of accuracy for each musical scale independently: NetMeasurements[trainedNet, stest2 , "Accuracy"] NetMeasurements[trainedNet, stest2, "F1Score"] NetMeasurements[trainedNet, stest2, {<|"Measurement" -> "Precision", "ClassAveraging" -> "Macro"|>, <|"Measurement" -> "Recall", "ClassAveraging" -> "Macro"|>}] ![im16][18] The same result of the independent scales measured separately, however, in the form of [ResourceFunction\[“NiceGrid”\]][19]: ![im17][20] Let's see how the result of the ConfusionMatrixPlot of the customized ResNet was performing for the classes of musical scales: NetMeasurements[trainedNet, stest2, "ConfusionMatrixPlot"] ![im18][21] To test the resulting custom NetChain, a function was created that generates a musical scale of choice and checks if the result was accurately predicted: sampleMusic[scale_, opts_ : "Decision"] := Module[{sound}, SetDirectory[NotebookDirectory[]]; Get["trainedNet.mx"]; scale -> {trainedNet[ Rasterize[ Spectrogram[ sound = Sound@ Table[ResourceFunction["MusicalScaleSample"][scale, "Tone" -> "C3", "NoteInterval" -> 0.17646, "Playing" -> "Mechanical"], 10], Frame -> None], RasterSize -> 600, ImageSize -> 300], opts] // Dataset, sound}] For example: sampleMusic["JazzMinor"] ![im19][22] Above we see that it worked, the scale was recognized! Below is another sample of this class of musical scale with its probabilities: sampleMusic["JazzMinor", "Probabilities"] ![im20][23] # Conclusion (and Notes): Although the methods have not been explored extensively (especially the first ones), we can have an idea of the results of each approach. There are many other methods and options for each method, of course, in this work only a few were covered. The best method by far was the customized ResNet, which was more carefully prepared and had very satisfactory results for most musical scales. Only 3 of the 30 scale classes had an accuracy below 50% and with the vast majority (23 of 30) above 70%. It is clear that this work was a relatively simple work, in the sense that the conditions of recognition are controlled so that there is a satisfactory result, but nothing prevents the work from being extended to multiple situations in the future. - **Thanks:** I would like to thank [Siria Sadeddin][24], who gave me tips and guidance on the ResNet part. Thank you Siria! Thank you! [1]: https://resources.wolframcloud.com/FunctionRepository/resources/MusicalScaleSample [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=im1.png&userId=2108079 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=test1mech.png&userId=2108079 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=restest1mech.png&userId=2108079 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=test2.png&userId=2108079 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=restest2.png&userId=2108079 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=baseneuralnet.png&userId=2108079 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=resbaseneuralnet.png&userId=2108079 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=basenaivetest.png&userId=2108079 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=resbasenaivetest.png&userId=2108079 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=spectroini.png&userId=2108079 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=spectrosamp.png&userId=2108079 [13]: https://resources.wolframcloud.com/NeuralNetRepository [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=netcrude.png&userId=2108079 [15]: https://community.wolfram.com//c/portal/getImageAttachment?filename=netcustom.png&userId=2108079 [16]: https://community.wolfram.com//c/portal/getImageAttachment?filename=learner.png&userId=2108079 [17]: https://community.wolfram.com//c/portal/getImageAttachment?filename=trainednet.png&userId=2108079 [18]: https://community.wolfram.com//c/portal/getImageAttachment?filename=netresult.png&userId=2108079 [19]: https://resources.wolframcloud.com/FunctionRepository/resources/NiceGrid [20]: https://community.wolfram.com//c/portal/getImageAttachment?filename=nicegrid.png&userId=2108079 [21]: https://community.wolfram.com//c/portal/getImageAttachment?filename=confusionmatrixplot.png&userId=2108079 [22]: https://community.wolfram.com//c/portal/getImageAttachment?filename=res1.png&userId=2108079 [23]: https://community.wolfram.com//c/portal/getImageAttachment?filename=res2.png&userId=2108079 [24]: https://community.wolfram.com/web/siriasadeddin/home Claudio Chaib 2020-11-04T22:52:47Z https://community.wolfram.com/groups/-/m/t/1787163 Tuseeta-san's [post][1] is how to convert a trained model of TensorFlow to Mathematica. Converting trained models from a language other than Mathematica to Mathematica is very beneficial to Mathematica users. So I'll show how to convert a trained model of PyTorch to Mathematica along with Tuseeta-san's post. ![enter image description here][2] ## Step 1: Figure out the architecture ## The [model][3] to be converted is Pose Estimation that detects the human skeleton (body parts and their connections) from an image. It's called OpenPose.The model consists of Feature map that extracts image features and six Stage maps. Feature map extracts image features from an input image(size:368*368). Each Stage map has two-branch, the first branch predicts confidence and the second predicts PAFs?Part Affinity Fields), along with the image feature .Two-branch are concatenated for next stage. ## Step 2: Coding it in Mathematica ## **Feature map** The Feature map consists of the first 23 layers of VGG-19, followed by 2 sets of Convolution and Ramp. Extract the first 23 layers of VGG-19. vgg19 = NetModel["VGG-19 Trained on ImageNet Competition Data"]; vgg19sub = Take[vgg19, {1, 23}]; Change Encoder. enc = NetExtract[vgg19, "Input"]; enc = NetReplacePart[ enc, {"ImageSize" -> {368, 368}, "VarianceImage" -> {0.229, 0.224, 0.225}, "MeanImage" -> {0.485, 0.456, 0.406}}]; featurefirst = NetReplacePart[vgg19sub, "Input" -> enc]; Add Convolution and Ramp. feature = NetAppend[ featurefirst, {"convadd1" -> ConvolutionLayer[256, 3, "Stride" -> 1, "PaddingSize" -> 1], "reluadd1" -> Ramp, "convadd2" -> ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], "reluadd2" -> Ramp}]; **Stage map** Each Stage map consists only of Convolutions and Ramps. Stage 1: The differences between two branches is that the last output channel number is 38 or 19. blk11 = NetChain[{ ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[512, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[38, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; blk12 = NetChain[{ ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[512, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[19, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; Stage 2?6: The difference between Stage1 and Stage2?6 is the kinds and the numbers of layers. blkx1 = NetChain[{ ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[38, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; blkx2 = NetChain[{ ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[19, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; Finally, create OpenPose. openpose = NetGraph[{ "feature" -> feature,(*feature*) "blk11" -> blk11, "blk12" -> blk12,(*stage 1*) "blk21" -> blkx1, "blk22" -> blkx2,(*stage 2*) "cat12" -> CatenateLayer[], "blk31" -> blkx1, "blk32" -> blkx2,(*stage 3*) "cat23" -> CatenateLayer[], "blk41" -> blkx1, "blk42" -> blkx2,(*stage 4*) "cat34" -> CatenateLayer[], "blk51" -> blkx1, "blk52" -> blkx2,(*stage 5*) "cat45" -> CatenateLayer[], "blk61" -> blkx1, "blk62" -> blkx2,(*stage 6*) "cat56" -> CatenateLayer[] }, {"feature" -> "blk11", "feature" -> "blk12",(*stage 1*) {"blk11", "blk12", "feature"} -> "cat12",(*stage 2*) "cat12" -> "blk21", "cat12" -> "blk22", {"blk21", "blk22", "feature"} -> "cat23",(*stage 3*) "cat23" -> "blk31", "cat23" -> "blk32", {"blk31", "blk32", "feature"} -> "cat34",(*stage 4*) "cat34" -> "blk41", "cat34" -> "blk42", {"blk41", "blk42", "feature"} -> "cat45",(*stage 5*) "cat45" -> "blk51", "cat45" -> "blk52", {"blk51", "blk52", "feature"} -> "cat56",(*stage 6*) "cat56" -> "blk61", "cat56" -> "blk62" }] ![enter image description here][4] ## Step 3: Importing the Weights and the Biases ## Download "[pose_model_scratch.pth][5]" as a trained model of PyTorch. Import the parameters, the weights and the biases. I referred "[How to import python pickle *.pkl?][6]" session = StartExternalSession["Python-NumPy"]; parameters = ExternalEvaluate[session, "import torch import numpy as np import pickle as pkl net_weights = torch.load( 'pose_model_scratch.pth', map_location={'cuda:0': 'cpu'}) keys = list(net_weights.keys()) parameters = {} for i in range(len(keys)): t = net_weights[keys[i]] x = t.numpy() parameters[keys[i]] = x.flatten() parameters"]; DeleteObject[session]; keys = Keys[parameters]; parameters = Values[parameters]; ## Step 4: Parsing the Weights and the Biases ## The parameters is 184 sets of one-dimensional lists. They consists of the Weights and the Biases of 92 Convolution layers in OpenPose. Get a list of layer names for OpenPose with depth level. Then, get a list of 92 names where convolution layer is used in it. layernames = GroupBy[Keys@NetInformation[openpose, "Layers"], First] // Values; convlayernames = (Position[ NetInformation[openpose, "Layers"], _ConvolutionLayer] // Flatten)[[All, 1]] ![enter image description here][7] As you can see in keys, you can see that the order of convolution layers of OpenPose is different from the order of convolution layers of "pose_model_scratch.pth" keys ![enter image description here][8] So, manually sort the order of convolution layers in OpenPose into the order of "pose_model_scratch.pth" convlayernamesGH = {{"feature", "conv1_1"}, {"feature", "conv1_2"}, {"feature", "conv2_1"}, {"feature", "conv2_2"}, {"feature", "conv3_1"}, {"feature", "conv3_2"}, {"feature", "conv3_3"}, {"feature", "conv3_4"}, {"feature", "conv4_1"}, {"feature", "conv4_2"}, {"feature", "convadd1"}, {"feature", "convadd2"}, {"blk11", 1}, {"blk11", 3}, {"blk11", 5}, {"blk11", 7}, {"blk11", 9}, {"blk21", 1}, {"blk21", 3}, {"blk21", 5}, {"blk21", 7}, {"blk21", 9}, {"blk21", 11}, {"blk21", 13}, {"blk31", 1}, {"blk31", 3}, {"blk31", 5}, {"blk31", 7}, {"blk31", 9}, {"blk31", 11}, {"blk31", 13}, {"blk41", 1}, {"blk41", 3}, {"blk41", 5}, {"blk41", 7}, {"blk41", 9}, {"blk41", 11}, {"blk41", 13}, {"blk51", 1}, {"blk51", 3}, {"blk51", 5}, {"blk51", 7}, {"blk51", 9}, {"blk51", 11}, {"blk51", 13}, {"blk61", 1}, {"blk61", 3}, {"blk61", 5}, {"blk61", 7}, {"blk61", 9}, {"blk61", 11}, {"blk61", 13}, {"blk12", 1}, {"blk12", 3}, {"blk12", 5}, {"blk12", 7}, {"blk12", 9}, {"blk22", 1}, {"blk22", 3}, {"blk22", 5}, {"blk22", 7}, {"blk22", 9}, {"blk22", 11}, {"blk22", 13}, {"blk32", 1}, {"blk32", 3}, {"blk32", 5}, {"blk32", 7}, {"blk32", 9}, {"blk32", 11}, {"blk32", 13}, {"blk42", 1}, {"blk42", 3}, {"blk42", 5}, {"blk42", 7}, {"blk42", 9}, {"blk42", 11}, {"blk42", 13}, {"blk52", 1}, {"blk52", 3}, {"blk52", 5}, {"blk52", 7}, {"blk52", 9}, {"blk52", 11}, {"blk52", 13}, {"blk62", 1}, {"blk62", 3}, {"blk62", 5}, {"blk62", 7}, {"blk62", 9}, {"blk62", 11}, {"blk62", 13} }; Get the position of each element of convlayernamesGH in OpenPose. convlayerpos = Flatten[Position[layernames, #] & /@ convlayernamesGH, 1] ![enter image description here][9] Reshape each one-dimensional list of parameters to the dimension of the corresponding weight or bias. getDimB[layer_] := Dimensions@NetExtract[layer, "Biases"] getDimW[layer_] := Dimensions@NetExtract[layer, "Weights"] convs = NetExtract[NetInitialize[openpose], #] & /@ convlayerpos; dimW = getDimW /@ convs; dimB = getDimB /@ convs; dim = Flatten[Transpose[{dimW, dimB}], 1]; parametersReshape = MapThread[ArrayReshape, {parameters, dim}]; ## Step 5: Linking the Weights and the Biases ## Replace the initial values of weights and biases in OpenPose with learned parameters, and finally get trained OpenPose. replacenames = Flatten[ Transpose[{Flatten@{#, "Weights"} & /@ convlayernamesGH, Flatten@{#, "Biases"} & /@ convlayernamesGH}], 1]; rule = Thread[replacenames -> parametersReshape]; trainedOpenPose = NetReplacePart[openpose, rule] ![enter image description here][10] ## Step 6: Making the tests ## For simplification, estimate pose for the image of single-person. The Output2 of OpenPose shows the confidence of 19 body parts in each part where the image is divided into 46 * 46. 1:Nose, 2:Neck, 3:RShoulder, 4:RElbow, 5:RWrist, 6:LShoulder, 7:LElbow, 8:LWrist, 9:RHip, 10:RKnee, 11:RAnkle, 12:LHip, 13:LKnee, 14:LAnkle, 15:REye, 16:LEye, 17:REar, 18:LEar, 19:Bkg Define the function to get the position of max of confidence of each body part. maxpts[img_, confidences_, idex_] := Module[{pos, pts, h}, pos = Reverse@First@Position[h = confidences[[idex]], Max@h]; pts = (pos/46)*ImageDimensions@img; pts = {pts[[1]], (ImageDimensions@img)[[2]] - pts[[2]]} ] Connect the detected body parts and show the result together on the original image. showpose[img_] := Module[{bodylist, size, out, confidences, pts, pose}, bodylist = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14}; size = {368, 368}; out = trainedOpenPose[img]; confidences = out[[2]]; pts = maxpts[img, confidences, #] & /@ bodylist; pose = Graphics[{ Yellow, Thickness[.0125], Line[pts[[#]] & /@ {1, 2}], Green, Line[pts[[#]] & /@ {2, 3, 4, 5}], Cyan, Line[pts[[#]] & /@ {2, 6, 7, 8}], Orange, Line[pts[[#]] & /@ {2, 9, 10, 11}], Magenta, Line[pts[[#]] & /@ {2, 12, 13, 14}], PointSize[Large], Red, Point[pts], White, Point[{{0, 0}, ImageDimensions@img}] }, ImagePadding -> All]; Show[img, pose] ] Let' try. img = Import["ichiro.jpg"]; showpose[img] ![enter image description here][11] ## Future work ## ? Estimate the pose of an image in which multi-person are by using PAFs of output1 of OpenPose. ? Convert a more accurate pose estimation model. [1]: https://community.wolfram.com/groups/-/m/t/1785523 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=00.jpg&userId=1013863 [3]: https://arxiv.org/pdf/1611.08050.pdf [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=752004.jpg&userId=1013863 [5]: https://www.dropbox.com/s/ae071mfm2qoyc8v/pose_model.pth?dl=0 [6]: https://mathematica.stackexchange.com/questions/181273/how-to-import-%5C%20python-pickle-pkl?rq=1 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=691905.jpg&userId=1013863 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=387908.jpg&userId=1013863 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=109406.jpg&userId=1013863 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=179107.jpg&userId=1013863 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=161401.jpg&userId=1013863 Kotaro Okazaki 2019-09-10T21:36:49Z https://community.wolfram.com/groups/-/m/t/1207400 ![found constellations][9] ## Story Time ## At its first meeting in 1922, the International Astronomical Union (IAU), officially adopted the list of 88 constellations that we use today. These include 14 men and women, 9 birds, two insects, 19 land animals, 10 water creatures, two centaurs, one head of hair, a serpent, a dragon, a flying horse, a river and 29 inanimate objects. As many of us have (frustratingly) witnessed first hand while star-gazing - most of these bear little resemblance to their supposed figures. Instead, it is more likely that the ancient constellation-makers meant them to be symbolic, a kind of celestial "Hall of Fame" for their favorite animals or heroes. This begs two questions I sought to answer with this project: 1. Can we 'do better' now with the WL's StarData[] curated data and Machine Learning functionality? 2. What if the ancient constellation-makers were slightly more creative? Say they looked up at the sky, and only saw yoga-poses! Some examples of the found yoga-pose constellations, projected on images of the night sky are shown here, with a walk-through of the code below: [![example yoga-pose constellations][1]][2] ## Yoga Poses ## First things first, finding images for yoga poses. Turns out, the WL has a built in YogaPose Entity Class with 216(!) available entities and their schematics. A lot of these are very similar (e.g. palms facing up/down) and we therefore only select a subset of them, which differ substantially from each other: yogaposes = EntityClass["YogaPose", All] // EntityList; chosenPoses = List /@ {4, 6, 7, 8, 11, 14, 23, 25, 28, 35, 38, 43, 51, 54, 56, 59, 63, 65, 71, 72, 76, 77, 78, 84, 88, 90, 98, 100, 102, 103, 110, 111, 112, 113, 116, 118, 119, 125, 126, 133, 139, 142, 149, 152, 153, 154, 155, 156, 160, 164, 165, 167, 172, 177, 178, 180, 182, 183, 184, 190, 193, 194, 195, 197, 202, 204, 206, 207, 209, 210, 212, 215, 216}; Shallow[Extract[yogaposes, chosenPoses]] We write a wrapper to ensure the schematics are padded to give a square image and visualize our 73 constellations: makeSquare[gr_, size_, bool_: False] := Block[{range, magnitudes, order, padding, newRanges, res}, range = AbsoluteOptions[gr, PlotRange][[1, 2]]; magnitudes = Abs[Subtract @@@ range]; order = Ordering[magnitudes, 2, Greater]; padding = Subtract @@ magnitudes[[order]]/2; newRanges = {{-1, 1} padding + Last[range[[order]]], First[range[[order]]]}; res = Show[gr, PlotRange -> Reverse@newRanges[[order]], ImageSize -> size]; If[bool, Rasterize[res], res]] Multicolumn[ makeSquare[#["SimplifiedSchematic"], 64] & /@ Most[Extract[yogaposes, chosenPoses]], 9, Frame -> All] ![constellations graphic][3] ## Neural Network ## Our problem now can be worded as a classification one: *given 5-10 stars, classify them as one of the 73 yoga-poses constellations* The problem with that formulation of-course is that 5-10 randomly selected points inside the constellation region isn't specific enough to differentiate between constellations. The image below shows that although 1000 sets of 10 points will define the shape, 10 points alone fall short: ![pointsInMesh][4] Instead, we compute the Voronoi diagram of the points, using DistanceTransform to leverage already-optimized convolutional neural nets used for classifying images. Note that, even if (naturally) the results are less clear with fewer points (left to right) - the result with only 10 points at the far right is still quite recognizable to a human eye: ![distanceTransforms][5] With this in-mind, we create a neural network similar to the [VGG16][6] neural network, with successive 5x5 convolutions, ReLu activation functions and Max pooling layers. Finally note that we use an average pooling instead of fully connected layers to reduce the number of parameters and the use of dropout layers. conv[output_] := ConvolutionLayer[output, {5, 5}, "PaddingSize" -> 2] pool[size_] := PoolingLayer[{size, size}, "Stride" -> size] lenet = NetChain[ {conv[32], Ramp, conv[32], Ramp, pool[2], conv[64], Ramp, conv[64], Ramp, pool[2], conv[128], Ramp, conv[128], Ramp, conv[128], Ramp, PoolingLayer[{9, 9}, "Stride" -> 9, "Function" -> Mean], FlattenLayer[], DropoutLayer[], 73, SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", Extract[yogaposes, chosenPoses]}], "Input" -> NetEncoder[{"Image", {64, 64}, ColorSpace -> "Grayscale"}] ] The training set then consists of generating 10-15 points inside the constellation region (resized and padded to allow for rotations later), and taking their DistanceTransform: meshes[n_] := meshes[n] = ImagePad[ColorNegate@ makeSquare[ yogaposes[[chosenPoses[[n, 1]]]]["SimplifiedSchematic"], 44], 10] trainingSet[n_, m_, iter_] := With[{reg = ImageMesh[meshes[n], CornerNeighbors -> False, Method -> "MarchingSquares"]}, Thread[ColorConvert[ ImageAdjust[ DistanceTransform[Image[SparseArray[# -> 0, {64, 64}, 1]]]], "Grayscale"] & /@ Clip[Round@RandomPoint[reg, {iter, m}], {1, 64}] -> yogaposes[[chosenPoses[[n, 1]]]]]] exampleTrainingSet = Flatten[trainingSet[#, 10, 1] & /@ RandomInteger[{1, 73}, 10], 1] The net took a couple of hours to train on ~100,000 examples on my CPU. Here we import the trained net: lenetTrained = Import["constellationsNet.wlnet"] ## Night Sky ## We're almost ready to classify the night sky. We first need the location of the 10,000 brightest stars along with their Right Ascension and Declination: brightest = StarData[EntityClass[ "Star", {EntityProperty["Star", "ApparentMagnitude"] -> TakeSmallest[10000]}], {"RightAscension", "Declination", "ApparentMagnitude"}] We can plot these on the night sky using no projection (i.e. RA Vs Dec), using the sinusoidal projection (taken by Kuba's excellent [answer][7] in SE) or on the celestial sphere given by the following transformations respectively: $$sinusoidal:\{(\alpha -\pi ) \cos (\delta ),\delta \}$$ $$map to3D:\{\cos (\alpha ) \cos (\delta ),\sin (\alpha ) \cos (\delta ),\sin (\delta )\}$$ ![brightestStarsProjections][8] ## Classification ## Finally, we use these 10,000 brightest stars to compute a multivariate smooth kernel distribution out of which to sample from and a Nearest function to compute neighboring stars. We need to of-course use our own distance function on the celestial sphere: wrap[list_] := Block[{xs, ys}, {xs, ys} = Transpose[list]; Thread[{Mod[xs, 2 \[Pi]], Mod[ys, \[Pi], -\[Pi]/2]}]] mapTo3D[\[Alpha]_, \[Delta]_] = {Cos[\[Alpha]] Cos[\[Delta]], Cos[\[Delta]] Sin[\[Alpha]], Sin[\[Delta]]}; dist[{u_, v_}, {x_, y_}] := (#1[[1]] - #2[[1]])^2 + (#1[[2]] - #2[[2]])^2 + (#1[[ 3]] - #2[[3]])^2 & @@ mapTo3D @@@ {{u, v}, {x, y}} pts = wrap@QuantityMagnitude[UnitConvert[brightest[[6 ;;, ;; 2]]]]; nf = Nearest[pts, DistanceFunction -> dist] sm = SmoothKernelDistribution[pts] Our search algorithm is therefore defined as follows: 1. Pick a random position from the night sky distribution 2. Compute its 5-10 nearest neighbors 3. Classify those stars and their rotations by $\frac{2 \pi }{15}$ 4. Select the rotation which gives the highest accuracy 5. Associate constellation to running association and repeat rescale[list_] := Block[{xs, ys}, {xs, ys} = Thread[list]; Thread[Rescale[#, MinMax[#], {11, 54}] & /@ {xs, ys}]] rotate[\[Alpha]_, pts_] := ImageAdjust@ DistanceTransform[ Image@SparseArray[ Round@RotationTransform[\[Alpha] , {65/2, 65/2}][rescale[pts]] -> 0, {64, 64}, 1]] sky = <||>; accumulate[] := Block[{pts = nf[Mod[RandomVariate[sm] + {0, \[Pi]}, 2 \[Pi]] - {0, \[Pi]}, RandomInteger[{5, 10}]], \[Alpha], pred}, {\[Alpha], pred} = Last[SortBy[ First /@ Table[Thread[{\[Alpha], lenetTrained[rotate[\[Alpha], pts], "TopProbabilities"]}], {\[Alpha], 0, 2 \[Pi], (2 \[Pi])/ 15}], Last]]; If[Not[KeyExistsQ[sky, pred[[1]]]] || TrueQ[sky[pred[[1]], "Accuracy"] < pred[[2]]], AssociateTo[sky, pred[[1]] -> <|"Accuracy" -> pred[[2]], "Image" -> HighlightImage[ makeSquare[pred[[1]]["SimplifiedSchematic"], 64], {PointSize[Medium], Red, Round[RotationTransform[\[Alpha], 65/2 {1, 1}][rescale@pts], 0.5]}], "Points" -> pts, "Angle" -> \[Alpha]|>], Nothing];] We can import a precomputed association with 20 such constellations: selectedAsc = Import["skyAsc.m.gz"] ## Results ## Finally, we orient the schematic based on the optimally found angle and (manually) connect the dots. orient[pts_, \[Alpha]_] := Round[RotationTransform[\[Alpha], 65/2 {1, 1}][rescale@pts], 0.5] lines = {{{10, 7, 3, 1, 4, 9, 2, 5, 8}, {2, 6}}, {{7, 3, 1, 9, 6}, {10, 2, 1, 8}, {4, 3, 5}}, {{7, 8, 9, 4, 3, 2, 10}, {1, 2, 6, 5}}, {{10, 2, 1, 3, 9, 5}, {7, 1, 4, 6, 8}}, {{5, 4, 1, 2, 3}}, {{10, 4, 5, 7, 9, 3, 8}, {6, 1, 2, 3}}, {{9, 5, 6, 3, 1, 2, 7}, {7, 4, 8}}, {{7, 6, 2, 3, 5, 8}, {9, 5, 4}, {3, 1}}, {{6, 2, 1, 3, 4}, {2, 5}}, {{5, 6, 4, 1, 2, 8, 3, 7}}, {{10, 6, 3, 2, 7}, {5, 2, 1, 4, 9, 8}}, {{9, 5, 1, 2, 4, 8, 7}, {6, 3, 8}}, {{6, 10, 4, 2, 1, 3, 9, 7, 5, 8}}, {{6, 7, 3, 2, 1, 4, 5}}, {{10, 8, 3, 6, 7, 2, 5, 9}, {5, 1, 4, 3}}, {{6, 5, 2, 1, 3}, {1, 4}}, {{5, 1, 2, 3}, {1, 4}}, {{6, 3, 2, 1, 4}, {3, 5}, {2, 7}}, {{3, 6, 8, 2, 4, 5, 1, 7}}, {{3, 1, 2, 6, 5}}}; makeSquareAndScale[gr_, \[Alpha]_, size_, opacity_] := Block[{range, magnitudes, order, padding, newRanges, s}, range = AbsoluteOptions[gr, PlotRange][[1, 2]]; magnitudes = Abs[Subtract @@@ range]; order = Ordering[magnitudes, 2, Greater]; padding = Subtract @@ magnitudes[[order]]/2; s = size/First[magnitudes[[order]]]; newRanges = {{-1, 1} padding + Last[range[[order]]], First[range[[order]]]}; Graphics[{Opacity[opacity], GeometricTransformation[ GeometricTransformation[ gr[[1]], {ScalingMatrix[{s, s}], -1 First /@ (s Reverse@newRanges[[order]])}], RotationTransform[\[Alpha], (size + 1)/2 {1, 1}]]}, ImageSize -> size]] With[{l = Map[Line, lines, {2}]}, Multicolumn[ Table[Show[ makeSquareAndScale[ Keys[selectedAsc][[i]][ "SimplifiedSchematic"], -selectedAsc[[i, -1]], 64, 0.25], Graphics[ GraphicsComplex[ rescale[selectedAsc[[i, -2]]], {l[[i]], Red, PointSize[Large], Point /@ Sort /@ Flatten /@ lines[[i]]}]], ImageSize -> 150], {i, 20}], 5, Frame -> All, Appearance -> "Horizontal"]] ![found constellations][9] These can also be superimposed on the full night-sky: Graphics[{With[{l = Map[Line, lines, {2}]}, Table[GraphicsComplex[ selectedAsc[[i, -2]], {Orange, Thickness[Large], l[[i]], Red, PointSize[Medium], Point /@ Sort /@ Flatten /@ lines[[i]]}], {i, 20}]], Opacity[0.25], PointSize[Small], Point[Complement[pts, Flatten[Values@selectedAsc[[All, -2]], 1]]]}, ImageSize -> 750] ![found overlay][10] sinusoidal[\[Alpha]_, \[Delta]_] = {(\[Alpha] - \[Pi]) Cos[\[Delta]], \ \[Delta]} Graphics[{With[{l = Map[Line, lines, {2}]}, Table[GraphicsComplex[ sinusoidal @@@ selectedAsc[[i, -2]], {Orange, Thickness[Large], l[[i]], Red, PointSize[Medium], Point /@ Sort /@ Flatten /@ lines[[i]]}], {i, 20}]], Opacity[0.25], PointSize[Small], Point[sinusoidal @@@ Complement[pts, Flatten[Values@selectedAsc[[All, -2]], 1]]]}, ImageSize -> 750] ![found sinusoidal][11] It is then a matter of Overlaying these found constellations to existing images of night skies to produce the images at the beginning of the post: overlay[{img_, constellation_}, {size_, loc_, opac_}] := ImageCompose[imgs[[img]], ImageResize[ Show[makeSquareAndScale[ Keys[selectedAsc][[constellation]][ "SimplifiedSchematic"], -selectedAsc[[constellation, -1]], 64, opac], Graphics[ GraphicsComplex[ rescale[selectedAsc[[constellation, -2]]], {White, Thickness[0.0075], Line /@ lines[[constellation]], White, PointSize[.025], Point /@ Sort /@ Flatten /@ lines[[constellation]]}]], ImageSize -> 1000], size], Scaled[loc]] ## Conclusions / Lessons Learned ## - It IS possible to find collections of stars in the night sky matching all sorts of shapes. Other built-in entities to try are Pokemons, Dinosaurs etc - Not all the found constellation work 'perfectly' - Machine Learning is a powerful tool, and the WL implementation makes it easy to get started - Reformulating the problem to an easier, already solved problem (e.g. points -> 2D image input) can help Classification Accuracy - Perhaps the most interesting aspect of neural networks now, is diversifying its applications - so be creative This work was presented at the 2017 WTC (as part of a larger talk entitled "(De)Generative Art"). <br> I look forward to any comments/suggestions. <br> <br>George <br> <br> PS The editor doesn't seem to let me attach the .wlnet and .m.gz files. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=exampleImages.png&userId=11733 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=exampleImages.png&userId=616023 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=4214constellationsGraphic.png&userId=616023 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=randomPointsInmesh.png&userId=616023 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=distanceTransforms.png&userId=616023 [6]: https://arxiv.org/pdf/1409.1556v6.pdf [7]: https://mathematica.stackexchange.com/questions/89668/geoprojection-for-astronomical-data-wrong-ticks/89792#89792 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=maps.png&userId=616023 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=foundConstellations.png&userId=616023 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=overlaidConstellations.png&userId=616023 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=overlaidConstellationsSinusoidal.png&userId=616023 George Varnavides 2017-10-24T04:17:44Z https://community.wolfram.com/groups/-/m/t/1221098 Introduction ---------- I recently came across a post for a computer program that in some fields intends to compete with the Wolfram Language and which has a toolbox for Machine Learning. I wanted to compare the work described with the workflow in Mathematica. The challenge is to classify old Japanese Characters from texts from the so-called Edo period: Style[StringTake[WikipediaData["Edo period"], 601], 16] ![enter image description here][1] The characters are written in running style and apparently are difficult to read for Japanese speakers today. We will use two approaches of machine learning in the Wolfram Language to tackle this problem. Historical Background ----------------------------- Before we proceed, let's have a look at the Tokugawa shogunate. I will use code borrow from [@Vitaliy Kaurov][at0] from his great article on ["Computational history: countries that are gone"][2]. The following lines compute the historic boundaries of the shogunate: Monitor[tokugawaPOLY = Table[EntityValue[Entity["HistoricalCountry", "TokugawaShogunate"], EntityProperty["HistoricalCountry", "Polygon", {"Date" -> DateObject[{t}]}]], {t, 1600, 1870}];, t] We then only filter for "changes" in the boundaries: tokugawaPOLY // Length tokugawaPOLYcomp = DeleteMissing[ DeleteDuplicates[Transpose[{Range[1600, 1870], tokugawaPOLY}], Last[#1] == Last[#2] &], 1, 2]; tokugawaPOLYcomp // Length In the 271 I consider in the first place there are only 4 changes to the boarders, which we can plot like so: GeoGraphics[{EdgeForm[Red], GeoStyling[Opacity[.1]], #} & /@ tokugawaPOLYcomp[[All, 2]], GeoProjection -> "Mercator", ImageSize -> 800, GeoBackground -> GeoStyling["StreetMap"], GeoRange -> Entity["Country", "Japan"], GeoRangePadding -> Quantity[800, "Kilometers"], GeoZoomLevel -> 6] ![enter image description here][3] So the boarders were indeed very stable for a long period reflecting the "no more wars" philosophy of the day. Arts and culture ---------- The stability led to a very rich art and literature scene. Here are some images that reflect the style of the time: ImageCollage["Thumbnail" /. Normal[Normal[WebImageSearch["Edo Period"]]], Method -> "Rows"] ![enter image description here][4] Our dataset ---------- Also literature flourished during this period. Our dataset will be taken from the [Center for Open Data in the Humanities][5]. The keen reader might have noticed that the page is in Japanese. Mathematica can translate the page, and [so can Google][6]. The dataset is called the "Japanese classic registered glyph data set"; ?????????????. TextTranslation["?????????????"] "Japan classics-type datasets". Following the first link (grey box) on the original Japanese page one gets to t[his page][7], from the bottom of which (look for a download symbol and the text "(TAR+GZ 24.57 MB)" the dataset can be downloaded. This is the google translation of the reference: > Japanese Classic Statement Character Data Set" (Kokugaku Kenkyu Other > Collection / CODH Processing) which in the original is "???????????????????????CODH??". The glyphs come from "bamboo letters" from 15 classic texts; the dataset contains more than 20000 glyphs of the 10 most frequent symbols. These letters were annotated by the "National Institute of Letters of Japan" (google). The dataset was also used in a [data/machine learning challenge][8]. After extracting the file to a folder on the desktop I can import the images with: characterImages = Import["/Users/thiel/Desktop/pmjt_sample_20161116/train_test_file_list.h5", "Data"]; Here is a sample of the glyphs: Grid[Partition[ImageAdjust[Image[#/256.]] & /@ characterImages[[3, 1 ;; 24]], 6], Frame -> All] ![enter image description here][9] Similar to the MNIST dataset, they consist of 28 times 28 pixels. They give the grey level as integers, which is why I divide by 256. This shows the pixelation of the glyphs: Image[characterImages[[3, 12]]/256.] // ImageAdjust ![enter image description here][10] The dataset conveniently contains a training and a test set. There are characterImages[[3]] // Length 19909 glyphs in the training set. All of these are annotated: Flatten[characterImages[[4]]] ![enter image description here][11] We can convert the training data to images like so: Monitor[trainingset = Table[ImageAdjust[Image[characterImages[[3, k]]/256.]] -> Flatten[characterImages[[4]]][[k]], {k, 1, 19909}];, k] The test set contains characterImages[[1]] // Length 3514 glyphs. The annotation of which are found in characterImages[[2]] // Flatten // Short ![enter image description here][12] This is how we prepare the test set data for our machine learning: Monitor[testset = Table[ImageAdjust[Image[characterImages[[1, k]]/256.]] -> Flatten[characterImages[[2]]][[k]], {k, 1, 3514}];, k] What are the 10 most frequent symbols? ---------- In one of the annotation files we find the character codes of the symbols so that we can map the classification to actual symbols. These are the glyphs we will consider: Rasterize /@ (Style[ FromCharacterCode[ ToExpression["16^^" <> StringJoin[Characters[#]]]], 100] & /@ {"3057", "306B", "306E", "3066", "308A", "3092", "304B", "304F", "304D", "3082"}) ![enter image description here][13] We can also make an image collage of them: ImageCollage[ Image /@ (Rasterize /@ (Style[ FromCharacterCode[ ToExpression["16^^" <> StringJoin[Characters[#]]]], 100] & /@ {"3057", "306B", "306E", "3066", "308A", "3092", "304B", "304F", "304D", "3082"})), Method -> "Rows"] ![enter image description here][14] I though it would be nice if they were in a more calligraphic form. So I tried to used ImageRestyle to try and make them more calligraphic. I think that the attempt was not very successful, but I want to show it anyway. First I download a couple of calligraphic symbols: caligraphy = "Thumbnail" /. Normal[Normal[WebImageSearch["japanese calligraphy"]]] I liked the 5th one, which I will use for the style transfer: calisymbols = ImageRestyle[#, caligraphy[[5]], PerformanceGoal -> "Quality"] & /@ (Rasterize /@ (Style[ FromCharacterCode[ ToExpression["16^^" <> StringJoin[Characters[#]]]], 100] & /@ {"3057", "306B", "306E", "3066", "308A", "3092", "304B", "304F", "304D", "3082"})) ![enter image description here][15] We can also remove the background to get an alternative representation: ![enter image description here][16] Machine Learning (Classify) ---------- Our first approach to classify the glyphs will be via Classify. The classification is very fast (only a couple of seconds on my machine) and quite accurate: standardcl = Classify[trainingset, ValidationSet -> testset, PerformanceGoal -> "Quality"] We can first calculate the ClassifierMeasurements cmstandard = ClassifierMeasurements[standardcl, testset] and the accuracy: cmstandard["Accuracy"] With 92.5% this is quite accurate and also much faster than the competitor product. The confusion plot shows that it is a very reasonable accuracy - in particular when considering that native speakers have problems identifying the glyphs. cmstandard["ConfusionMatrixPlot"] ![enter image description here][17] Here are some glyphs that the Classifier identified as class "0": cmstandard["Examples" -> {0, 0}][[1 ;; 20]] ![enter image description here][18] They look quite diverse, but so does the original set: RandomSample[Select[trainingset, #[[2]] == 0 &], 20] ![enter image description here][19] It is quite impressive that the Classify function manages to achieve this level of accuracy. We can, however, do better! Machine Learning (NetTrain) ---------- We can improve the accuracy by training our own custom made network, which in this case will be the standard net that is often used for the MNIST dataset in Mathematica. It turns out that we do not even need a GPU for the training. lenet = NetChain[{ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2], ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2], FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", Range[0, 9]}], "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]] ![enter image description here][20] This we can then train: lenet = NetTrain[lenet, trainingset, ValidationSet -> testset, MaxTrainingRounds -> 20]; ![enter image description here][21] The training is done in about 4:30 minutes, but it appears that 20 training rounds are not necessary. Here are some results when applying the classifier: imgs = Keys@RandomSample[testset, 60]; Thread[imgs -> lenet[imgs]] ![enter image description here][22] Let's see how well it does: cm = ClassifierMeasurements[lenet, testset] cm["ConfusionMatrixPlot"] ![enter image description here][23] This corresponds to an accuracy of cm["Accuracy"] 95.6%, which is really impressive given the dataset. The following table illustrates the difficulty: Grid[Table[ Prepend[Select[trainingset, #[[2]] == k &][[1 ;; 10]][[All, 1]], ImageResize[calisymbols[[k + 1]], 28]], {k, 0, 9}], Frame -> All, Background -> {{Red}, None}] ![enter image description here][24] The first (red) column is derived from the glyph based on the character code - using our "calligraphy-style". The remaining columns show items from the training set, i.e. glyphs that were manually classified to belong to the given group. Conclusion ---------- I do not speak/read Japanese and have no more than Wikipedia knowledge about the Edo period. I still find it quite amazing that using the Wolfram Language's high level Classify function and one of the pre-implemented networks, we can achieve quite a remarkable classification. I am sure that the network can be improved. I would also like to see a more complete dataset with more than 10 different symbols. [at0]: http://community.wolfram.com/web/vitaliyk [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.24.03.png&userId=48754 [2]: http://community.wolfram.com/groups/-/m/t/852277 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.30.18.png&userId=48754 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.33.24.png&userId=48754 [5]: http://codh.rois.ac.jp/char-shape/ [6]: https://translate.google.com/translate?hl=en&sl=auto&tl=en&u=http://codh.rois.ac.jp/char-shape/ [7]: http://codh.rois.ac.jp/char-shape/book/ [8]: https://translate.google.com/translate?hl=en&sl=ja&tl=en&u=http://codh.rois.ac.jp/seminar/japanese-character-challenge-20170210/&sandbox=1 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.56.19.png&userId=48754 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.57.57.png&userId=48754 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.59.36.png&userId=48754 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.01.19.png&userId=48754 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.04.22.png&userId=48754 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.05.04.png&userId=48754 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.07.44.png&userId=48754 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.08.53.png&userId=48754 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.15.09.png&userId=48754 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.17.07.png&userId=48754 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.18.25.png&userId=48754 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.21.43.png&userId=48754 [21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.22.50.png&userId=48754 [22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.27.23.png&userId=48754 [23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.28.27.png&userId=48754 [24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.40.41.png&userId=48754 Marco Thiel 2017-11-14T00:49:30Z https://community.wolfram.com/groups/-/m/t/480226 We extend the discussion on machine learning one step further and focus on predictive analysis offered in the ML domain. Prediction builds on classification and clustering techniques discussed previously and uses pattern detection and similarity features in data to estimate the future outcome. This is particularly relevant to finance where the ability of data groups to predict the values of less-liquid instruments is of high interest. We demonstrate the prediction using CDS data and show the application of non-regression models as superior methods for predictive analysis. ![PA image][1] #Introduction# Classification and clustering which we discussed in previous installments naturally extends into another field of data mining - **prediction**. It is well-known that there will be limited benefit of massive amount of data collection if the data cannot be analysed, decomposed into signals and used to predict future behaviour. Therefore, data analysis in statistics and econometrics can be broken down into four major categories: (i) prediction, (ii) summarisation, (iii) estimation, and (iv) hypothesis testing. Machine learning , as we have stated previously, focuses primarily at data organisation and prediction. However, closely related field is also data summarisation where pattern finding in the data - covered by the classification - is of primary interest. Machine learning is therefore about developing high-performance computer systems that can provide useful predictions in the presence of challenging computational constraints. This whole area of data processing is today routinely described as **data science**, which is primarily looks at both prediction and summarisation, but also extends into data manipulation, visualization, and other similar tasks. In pure statistical terms, the prediction problem reduces to finding *conditional distribution* of some variable y given some other variables $x= (x_ 1,x_ 2,...x_n)$. In data science methodology, variables x are called 'predictors' or 'features'. The prediction point y is generally the highest value (mean) of the calibrated conditional distribution. #Prediction basics# Much of applied econometrics, economics and finance is concerned with detecting and summarising relationships in the data. The most common tool is (linear) regression analysis. Machine learning offers richer set of tools that can usefully summarise various sorts of nonlinear relationships in the data. We will focus on these regression-like tools because they are the most natural for economic and financial applications. The objective of prediction is to find some function that gives a good estimate of y as function of x. In machine learning context the data we work with are generally considered independent and identically distributed. They may be 'fat' which implies more predictors relative to observations or 'tall' which indicated many observations relative to predictors. When we build prediction routines, we start with some observation of y and x and then want to obtain the predicted point for the new value x .'Good' prediction involves finding a function that minimises some error estimation. Sum of squared residuals, mean residual value are generally the most frequently used tools in this context. Tradition all method to solve this problem is regression - typically linear or logistic However, machine learning offers better options especially when there is a lot of data available. These include trees, nearest neighbors, random forests or neural networks. #Working with predictions# Objective of prediction routines is to find good out-of-sample estimation to the data. This task is not always trivial and regressions are generally known to be 'weak' around out-of-sample predictions. This is generally known as 'overfitting' problem. Machine learning addresses this problem through: - Regularisation - preference is given to simpler models - Dataset division into (i) training and (ii) testing sets - Cross-validation of the dataset and the model #Prediction relation to other data mining techniques# Irrespective of the method structure, there is a common confusion between explanatory and predictive models. In a predictive model only the model's accuracy is important and the structure of the model is relatively unimportant, as long as it is robust. In general, models become more robust as the number of predictors declines, and hence there is a greater tendency towards parsimony in predictive models, often using variable selection algorithms. However, if a model is developed for explanatory purposes, the identity and weights of the predictors is the most important part of the model because these are used to infer the nature of the relationship between the class and the values of the predictor variables. The contribution made by the predictors is obviously most important in an explanatory, rather than a predictive, model. In predictive models the role of predictors is less important than the accuracy of the prediction. Consequently, the use of variable selection routines that produce a more parsimonious model are generally more acceptable in predictive models. #Prediction in Finance# The advantages offered by machine learning are well suited for the data analysis in finance, particularly when the larger datasets are available. For years, the regression technique was dominant in economics and finance on the premise of limited amount of data. However,nowadays, in data-rich era, there is no reason not to start applying machine learning methods more frequently in economic and financial industries with data separation or the use of validation sets. ##Problem definition## Let's define the following problem: we have available set of actively traded and liquid CDS rates ( 5 years of daily rates) that we would like to use for prediction of lesser liquid CDS. These types of problem frequently occur in the credit or equity markets where liquidity factor drives certain instruments into specific categories. - 3 CDS will serve as features - We split the data into three sets: (i) training, (ii) testing and (iii) validation - We apply several predictive routines to obtain the future value in the out-of-sample domain ##Data generation## vols = {0.01, 0.008, 0.011, 0.008}; crl = ({ {1, 0.35, 0.45, 0.36}, {0.35, 1, 0.43, 0.32}, {0.45, 0.43, 1, 0.46}, {0.36, 0.32, 0.46, 1} }); cm = Table[vols[[i]]*vols[[j]]*crl[[i, j]], {i, 1, Length[vols]}, {j, 1, Length[vols]}] > {{0.0001, 0.000028, 0.0000495, 0.0000288}, {0.000028, 0.000064, 0.00003784, 0.00002048}, {0.0000495, 0.00003784, 0.000121, 0.00004048}, {0.0000288, 0.00002048, 0.00004048, 0.000064}} init = {0.55, 0.72, 1.25, 1}; mn = MultinormalDistribution[{0, 0, 0, 0}, cm]; data = Accumulate[Prepend[RandomVariate[mn, 1500], init]]; ListLinePlot[Transpose[data], PlotLegends -> {"cds 1", "cds 2", "cds 3", "cds 4"}, PlotLabel -> Style["CDS of observed series", 15]] ![enter image description here][2] traindata = data[[;; 600]]; validata = data[[601 ;; 900]]; testdata = data[[901 ;;]]; trainset = Table[Drop[traindata, None, -1][[i]] -> Flatten[Take[traindata, All, -1]][[i]], {i, 1, Length[traindata]}]; testset = Table[Drop[testdata, None, -1][[i]] -> Flatten[Take[testdata, All, -1]][[i]], {i, 1, Length[testdata]}]; validset = Table[Drop[validata, None, -1][[i]] -> Flatten[Take[validata, All, -1]][[i]], {i, 1, Length[validata]}]; {ListLinePlot[Transpose[traindata], PlotLegends -> {"cds 1", "cds 2", "cds 3", "cds 4"}, PlotLabel -> Style["Training data", 15]], ListLinePlot[Transpose[testdata], PlotLegends -> {"cds 1", "cds 2", "cds 3", "cds 4"}, PlotLabel -> Style["Testing data", 15]], ListLinePlot[Transpose[validata], PlotLegends -> {"cds 1", "cds 2", "cds 3", "cds 4"}, PlotLabel -> Style["Validation data", 15]]} ![enter image description here][3] We can investigate the predictors separately by looking at histogram of the training data: edata = Transpose[Drop[traindata, None, -1]]; {Histogram[edata[[1]], 20, ChartStyle -> "Pastel", PlotLabel -> "Feature 1"], Histogram[edata[[2]], 20, ChartStyle -> "Rainbow", PlotLabel -> "Feature 2"], Histogram[edata[[3]], 20, ChartStyle -> "NeonColors", PlotLabel -> "Feature 3"]} ![enter image description here][4] #Setting the predictors# Having defined the training, testing and validation sets, we can now set up the prediction model: Take[trainset, 10] > {{0.55, 0.72, 1.25} -> 1, {0.571876, 0.719578, 1.24021} -> 0.997824, {0.577651, 0.719425, 1.23595} -> 1.00456, {0.573492, 0.706407, 1.23029} -> 1.00455, {0.564589, 0.721109, 1.2309} -> 0.999223, {0.582398, 0.73248, 1.2497} -> 1.00538, {0.583999, 0.727467, 1.25179} -> 0.999114, {0.592003, 0.721993, 1.24155} -> 0.990911, {0.592474, 0.718474, 1.23417} -> 0.985031, {0.603327, 0.710775, 1.22836} -> 0.98689} Given the size of the training set we select **Random Forest** algorithm for prediction routine pred = Predict[trainset, ValidationSet -> validset, Method -> "RandomForest", PerformanceGoal -> "Quality"] ![enter image description here][5] and obtain the information on the model PredictorInformation[pred] ![enter image description here][6] The automatic routine selects the Random Forest as the most optimal algorithm for this task ##Conditional distribution## We can obtain the information about the distribution of the predicted value: dist = pred[Drop[init, -1], "Distribution"] > NormalDistribution[1.00262, 0.0288765] It is confirmed that the predicted value is normally distributed with mean close to the and low volatility. PDF[dist, x] Plot[PDF[dist, x], {x, 0.8, 1.25}, PlotStyle -> {Thick, Purple}, FillingStyle -> LightGray, Filling -> Axis, PlotLabel -> Style["Conditional PDF of the predicted value", 15]] > 13.8155 E^(-599.627 (-1.00262 + x)^2) ![enter image description here][7] ##Testing predictor accuracy## We may be interested to check how accurate the predictor works relative to the original data. plotdata = Drop[traindata, None, -1]; adata = Transpose[Take[traindata, All, -1]] // Flatten; pdata = Table[pred[plotdata[[i]]], {i, 1, Length[traindata]}]; ListLinePlot[{adata, pdata}, PlotLabel -> Style["Actual and Predicted data", 15], PlotLegends -> {"Actual", "Predicted"}, PlotStyle -> {Blue, Red}] ![enter image description here][8] We can see that the predictor replicates the original data quite well. ##Verification with Testing data set## We can test and validate the classifier with the <ClassifierMeasurements> command. For example, we can used the testing set defined above and look at its subset: Take[testset, 10] > {{0.485579, 0.621485, 1.15959} -> > 1.0466, {0.485904, 0.630628, 1.15086} -> > 1.04336, {0.477018, 0.618847, 1.14833} -> > 1.04416, {0.492894, 0.627871, 1.15071} -> > 1.04294, {0.496431, 0.626622, 1.15025} -> > 1.04363, {0.501258, 0.629071, 1.14728} -> > 1.04505, {0.506147, 0.630316, 1.15071} -> > 1.05803, {0.492122, 0.627663, 1.15387} -> > 1.05949, {0.485113, 0.626491, 1.16395} -> > 1.0621, {0.476587, 0.632491, 1.16424} -> 1.05147} pm = PredictorMeasurements[pred, testset] ![enter image description here][9] The classifier returns the correct class group ping for all categories with high precision. - Resting of residuals pm["ResidualPlot"] ![enter image description here][10] - Testing of comparison pm["ComparisonPlot"] ![enter image description here][11] #Predictor sensitivity to the features variability# We look how the predictor reacts to the features input w.r.t. calibrated conditional distribution {Plot[pred[{x, 0.7, 1.25}], {x, 0.4, 0.7}, PlotStyle -> Blue, Filling -> Axis, FillingStyle -> LightYellow, PlotLabel -> Style["1st CDS sensiitivtiy", 13]], Plot[pred[{0.6, x, 1.25}], {x, 0.6, 0.8}, PlotStyle -> Magenta, Filling -> Axis, FillingStyle -> LightGray, PlotLabel -> Style["2nd CDS sensiitivtiy", 13]], Plot[pred[{0.6, 0.7, x}], {x, 1.1, 1.3}, PlotStyle -> Green, Filling -> Axis, FillingStyle -> LightBrown, PlotLabel -> Style["3rd CDS sensiitivtiy", 13]]} ![enter image description here][12] As charts above suggest, each feature affects the predictor differently: - 1st CDS both decreases and increases the predicted value as the CDS move upwards - 2nd CDS increased the predicted value on the entire range, however the degree of impact varies - 3rd CDS is impact less on the lower side of the range and then increases the predicted CDS rate #Predicting outcomes with new data# We now apply the predictor to the out-of-sample data based on the specification above: - Define new covariance matrix tvols = {0.015, 0.02, 0.03}; tcorr = ({ {1, 0.4, 0.5}, {0.4, 1, 0.45}, {0.5, 0.45, 1} }); tcm = Table[ tvols[[i]]*tvols[[j]]*tcorr[[i, j]], {i, 1, Length[tvols]}, {j, 1, Length[tvols]}] > {{0.000225, 0.00012, 0.000225}, {0.00012, 0.0004, 0.00027}, {0.000225, 0.00027, 0.0009}} - Generate the new CDS data for each feature newinit = {0.68, 0.93, 1.4}; mn = MultinormalDistribution[{0, 0, 0}, tcm]; tdata = Accumulate[Prepend[RandomVariate[mn, 150], newinit]]; ListLinePlot[Transpose[tdata], PlotLegends -> {"new cds 1", "new cds 2", "new cds 3"}, PlotLabel -> Style["Out-of-sample CDS data", 15]] ![enter image description here][13] newdata = Table[predauto[tdata[[i]]], {i, 1, Length[tdata]}]; ListLinePlot[newdata, PlotTheme -> "Web", PlotLabel -> Style["Predicted value of CDS for put-of-sample input", 15]] ![enter image description here][14] We have obtained the new predicted data which we can investigate further: - Data distribution edist = SmoothKernelDistribution[newdata] ![enter image description here][15] Plot[PDF[edist, x], {x, 1, 1.3}, PlotLabel -> "Probability density of the predicted value", Filling -> Axis, PlotRange -> All] ![enter image description here][16] This is the estimated probability density from the predicted data. - Descriptive statistics stats = {Mean, Median, Variance, Min, Max, Skewness, Kurtosis}; TableForm[Through[stats[newdata]], TableHeadings -> {stats, None}] ![enter image description here][17] - Histogram Histogram[newdata, 25, ChartStyle -> "Rainbow"] ![enter image description here][18] #Alternative prediction methods# We can test other prediction methods available in Predict command: - Linear regression plinreg = Predict[trainset, ValidationSet -> validset, Method -> "LinearRegression", PerformanceGoal -> "Quality"] ![enter image description here][19] pmlin = PredictorMeasurements[plinreg, testset]; pmlin["ResidualPlot"] ![enter image description here][20] plinreg[Drop[init, -1], "Distribution"] > NormalDistribution[1.09925, 0.0769071] We observe higher predicted value and higher volatility relative to Random Forest - NearestNeighbors pnn = Predict[trainset, ValidationSet -> validset, Method -> "NearestNeighbors", PerformanceGoal -> "Quality"] ![enter image description here][21] pnntest = PredictorMeasurements[pnn, testset]; pnntest["ResidualPlot"] ![enter image description here][22] pnn[Drop[init, -1], "Distribution"] > NormalDistribution[0.982606, 0.0304784] NearestNeighbors produces excellent fit to the data, but significantly higher volatility - Neural network pneural = Predict[trainset, ValidationSet -> validset, Method -> "NeuralNetwork", PerformanceGoal -> "Quality"] ![enter image description here][23] pneuraltest = PredictorMeasurements[pneural, testset]; pneuraltest["ResidualPlot"] ![enter image description here][24] pneural[Drop[init, -1], "Distribution"] > NormalDistribution[0.974001, 0.0461196] Reasonable fit to the data but with higher volatility than Random Forest. #Conclusion# Prediction is useful technique for finding relationship in data, especially if we have at our disposal larger set of data. Machine learning comes particularly handy in detecting patterns better than traditional linear model. The tools and methods such as decision trees, neutral networks or nearest neighborhood offer more flexibility to find complex relationship Prediction as a method to estimate outcome from supporting features naturally offers practical solutions to finance and economics where the estimation can be quite valuable. Market data generation, growth rates forecast, marketing campaign model testing or inflation prediction are just few examples where machine learning with prediction focus becomes essential tool for decision making.Richer contents and better prediction ability makes machine learning more powerful than traditional regression techniques. [1]: /c/portal/getImageAttachment?filename=PredAnalimage.png&userId=387433 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=62621.png&userId=20103 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=108552.png&userId=20103 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=87493.png&userId=20103 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=90014.png&userId=20103 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=36945.png&userId=20103 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=56476.png&userId=20103 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=61627.png&userId=20103 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=75458.png&userId=20103 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=82379.png&userId=20103 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=323310.png&userId=20103 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=841311.png&userId=20103 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=241412.png&userId=20103 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=725313.png&userId=20103 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=603514.png&userId=20103 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=998015.png&userId=20103 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=880516.png&userId=20103 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=523517.png&userId=20103 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=414018.png&userId=20103 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=497019.png&userId=20103 [21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=189720.png&userId=20103 [22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=458421.png&userId=20103 [23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=741122.png&userId=20103 [24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=756823.png&userId=20103 Igor Hlivka 2015-04-16T00:08:17Z https://community.wolfram.com/groups/-/m/t/1341081 For my research project I had to encounter a thorny problem. But before I tell about the problem I would like to briefly mention something about my research project. Basically I am using embryonic stem cells that self-organize to form spheroids (balls of cells) to study gastrulation events. In order to not bog down the readers with technical jargon, “gastrulation” is a process where the stem cells start to form the different layers; each layer then goes onto form the various tissues/organs, in the process unraveling the developmental plan of the entire organism. I am using experimental techniques and quantitative principles from biophysics and engineering to understand some aspects of this crucial process Now coming back to the problem at hand, the gastruloids (image below) are quite rough in their appearance and not as beautiful as one would like them to be (only a mother can love such an image). Any means of quantifying these gastruloids requires me to initially segment them. When you see a time-lapse images of gastruloids it becomes apparent that they shed a lot of cells (for reasons I do not know yet). This adds considerable noise to the system; oftentimes to the point that – as a human – my eyes are fooled and run into the difficulty of finding the right contours for the spheroids. Here comes the disclosure: classical means/operations in image-processing (gradients and edge detection, filtering, morphological operations etc.. ) prove utterly futile for image segmentation in my case. ![enter image description here][1] (A gastruloid – virtually a ball of cells with many shed around the periphery) So what can you do to address the problem where even the best image processing tool in existence – the human eyes – fails. This is precisely where you take help of neural networks. Neural networks are selling like hotcakes during the recent years and added life and hope to the once dead area of artificial intelligence. Again to avoid underlying technical details, neural networks is a paradigm utilized by the computer to mimic the working of a human brain by taking into account the complex interactions between the cells – but only digitally. There are many flavours of neural networks out there, each one geared towards performing a specific task. With advancements made in the area of deep learning/artificial intelligence, the neural nets have started to surpass humans in tasks that humans have been known to be best for i.e. classification tasks. A few recent examples that come to mind include Google’s AlphaGo beating the former World Go champion and an AI diagnosing skin cancer with an unprecedented accuracy. I utilized one such flavour of neural networks (a deep convolutional network – termed as UNET) to solve my longstanding problem. I constructed the network in Wolfram-Language with external help from Alexey Golyshev. UNET is a deep convolutional network that has a series of convolutional and pooling operations in the contraction phase of the net (wherein the features are extracted) and a sequence of deconvolution & convolution operations in the expansion phase which then yields an output from the network. This output can be subjected to a threshold to ultimately generate a binarized mask (the image segmentation). ![enter image description here][2] The architecture of UNET as provided by the author: https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/ (* ::Package:: *) BeginPackage["UNETSegmentation`"] (* ::Section:: *) (*Creating UNet*) conv[n_]:=NetChain[ { ConvolutionLayer[n,3,"PaddingSize"->{1,1}], Ramp, BatchNormalizationLayer[], ConvolutionLayer[n,3,"PaddingSize"->{1,1}], Ramp, BatchNormalizationLayer[] } ]; pool := PoolingLayer[{2,2},2]; dec[n_]:=NetGraph[ { "deconv" -> DeconvolutionLayer[n,{2,2},"Stride"->{2,2}], "cat" -> CatenateLayer[], "conv" -> conv[n] }, { NetPort["Input1"]->"cat", NetPort["Input2"]->"deconv"->"cat"->"conv" } ]; nodeGraphMXNET[net_,opt: ("MXNetNodeGraph"|"MXNetNodeGraphPlot")]:= net~NetInformation~opt; UNET := NetGraph[ <| "enc_1"-> conv[64], "enc_2"-> {pool,conv[128]}, "enc_3"-> {pool,conv[256]}, "enc_4"-> {pool,conv[512]}, "enc_5"-> {pool,conv[1024]}, "dec_1"-> dec[512], "dec_2"-> dec[256], "dec_3"-> dec[128], "dec_4"-> dec[64], "map"->{ConvolutionLayer[1,{1,1}],LogisticSigmoid} |>, { NetPort["Input"]->"enc_1"->"enc_2"->"enc_3"->"enc_4"->"enc_5", {"enc_4","enc_5"}->"dec_1", {"enc_3","dec_1"}->"dec_2", {"enc_2","dec_2"}->"dec_3", {"enc_1","dec_3"}->"dec_4", "dec_4"->"map"}, "Input"->NetEncoder[{"Image",{160,160},ColorSpace->"Grayscale"}] ] (* ::Section:: *) (*DataPrep*) dataPrep[dirImage_,dirMask_]:=Module[{X, masks,imgfilenames, maskfilenames,ordering, fNames,func}, func[dir_] := (SetDirectory[dir]; fNames = FileNames[]; ordering = Flatten@StringCases[fNames,x_~~p:DigitCharacter.. :> ToExpression@p]; Part[fNames,Ordering@ordering]); imgfilenames = func@dirImage; X = ImageResize[Import[dirImage<>"\\"<>#],{160,160}]&/@imgfilenames; maskfilenames = func@dirMask; masks = Import[dirMask<>"\\"<>#]&/@maskfilenames; {X, NetEncoder[{"Image",{160,160},ColorSpace->"Grayscale"}]/@masks} ] (* ::Section:: *) (*Training UNet*) trainNetwithValidation[net_,dataset_,labeldataset_,validationset_,labelvalidationset_, batchsize_: 8, maxtrainRounds_: 100]:=Module[{}, SetDirectory[NotebookDirectory[]]; NetTrain[net, dataset->labeldataset,All, ValidationSet -> Thread[validationset-> labelvalidationset], BatchSize->batchsize,MaxTrainingRounds->maxtrainRounds, TargetDevice->"GPU", TrainingProgressCheckpointing->{"Directory","results","Interval"->Quantity[5,"Rounds"]}] ]; trainNet[net_,dataset_,labeldataset_, batchsize_:8, maxtrainRounds_: 10]:=Module[{}, SetDirectory[NotebookDirectory[]]; NetTrain[net, dataset->labeldataset,All,BatchSize->batchsize,MaxTrainingRounds->maxtrainRounds, TargetDevice->"GPU", TrainingProgressCheckpointing->{"Directory","results","Interval"-> Quantity[5,"Rounds"]}] ]; (* ::Section:: *) (*Measure Accuracy*) measureModelAccuracy[net_,data_,groundTruth_]:= Module[{acc}, acc =Table[{i, 1.0 - HammingDistance[N@Round@Flatten@net[data[[i]],TargetDevice->"GPU"], Flatten@groundTruth[[i]]]/(160*160)},{i,Length@data} ]; {Mean@Part[acc,All,2],TableForm@acc} ]; (* ::Section:: *) (*Miscellaneous*) saveNeuralNet[net_]:= Module[{dir = NotebookDirectory[]}, Export[dir<>"unet.wlnet",net]]/; Head[net]=== NetGraph; saveInputs[data_,labels_,opt:("data"|"validation")]:=Module[{}, SetDirectory[NotebookDirectory[]]; Switch[opt,"data", Export["X.mx",data];Export["Y.mx",labels], "validation", Export["Xval.mx",data];Export["Yval.mx",labels] ] ] EndPackage[]; The above code can also be found in the repository @ [Wolfram-MXNET GITHUB][3] I trained my network over my laptop GPU (Nvidia GTX 1050) by feeding an augmented data (a set of 300 images constructed from a small dataset) . The training was done in under 3 minutes !. The accuracy (computed as the Hamming Distance between two vectors) of the generated binary masks with respect to the ground truth (unseen data) for a set of 90 images was 98.55 %. And with this a task that previously required me to painstakingly trace the contour of the gastruloids manually can now be performed in a matter of milliseconds. All the saved time and perspiration to be utilized somewhere else? ![enter image description here][4] Below is the results obtained by applying our trained net on one input: ![enter image description here][5] The interesting aspect for me regarding the network was that despite my gastruloids being highly dynamic (changing shape over time) I never had to explicity state it to the network. All the necessary features were learned from the limited number of images that I trained my network with. This is the beauty of the neural network. ![enter image description here][6] Finally the output of the net as applied on a number of unseen images: ![enter image description here][7] Note: I have a python MXNET version of UNET @ [python mxnet GITHUB][8] The wolfram version of UNET however seems to outperform the python version even though it also utilizes MXNET at the back-end for implementing neural networks. It should not come as a surprise because my guess is that the people at Wolfram Research may have done internal optimizations on top of the library [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=gastruloid.png&userId=942204 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=u-net-architecture-initial-authors-implementation.png&userId=942204 [3]: https://github.com/alihashmiii/UNet-Segmentation-Wolfram [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img1.png&userId=942204 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img2.png&userId=942204 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img3.png&userId=942204 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img4.png&userId=942204 [8]: https://github.com/alihashmiii/blobsegmentation Ali Hashmi 2018-05-17T15:06:45Z https://community.wolfram.com/groups/-/m/t/970999 Text and sentiment analysis has become a very popular topic in quantitative research over the last decade, with applications ranging from market research and political science, to e-commerce. In this post I am going to outline an approach to the subject, together with some core techniques, that have applications in investment strategy. In the early days of the developing field of market sentiment analysis, the supply of machine readable content was limited mainly to mainstream providers of financial news such as Reuters or Bloomberg. Over time this has changed with the entry of new competitors in the provision of machine readable news, including, for example, Ravenpack or more recent arrivals like Accern. Providers often seek to sell not only the raw news feed service, but also their own proprietary sentiment indicators that are claimed to provide additional insight into how individual stocks, market sectors, or the overall market are likely to react to news. There is now what appears to be a cottage industry producing white papers seeking to demonstrate the value of these services, often accompanied by some impressive pro-forma performance statistics for the accompanying strategies, which include long-only, long/short, market neutral and statistical arbitrage. For the purpose of demonstration I intend to forego the blandishments of these services, although many are no doubt are excellent, since the reader is perhaps unlikely to have access to them. Instead, in what follows I will focus on a single news source, albeit a highly regarded one: the Wall Street Journal. This is, of course, a simplification intended for illustrative purposes only – in practice one would need to use a wide variety of news sources and perhaps subscribe to a machine readable news feed service. But similar principles and techniques can be applied to any number of news feeds or online sites. **The WSJ News Archive** We are going to access the Journal’s online archive, which presents daily news items in a convenient summary format, an example of which is shown below. The archive runs from the beginning of 2012 through to the current day, providing ample data for analysis. In what follows, I am going to make a few important assumptions, neither of which is likely to be 100% accurate – but which will not detract too much from the validity of the research, I hope. The first assumption is that the news items shown in each daily archive were reported prior to the market open at 9:30 AM. This is likely to be true for the great majority of the stories, but there are no doubt important exceptions. Since we intend to treat the news content of each archive as antecedent to the market action during the corresponding trading session, exceptions are likely to introduce an element of look-ahead bias. The second assumption is that the archive for each day is shown in the form in which it would have appeared on the day in question. In reality, there are likely to have been revisions to some of the stores made subsequent to their initial publication. So, here too, we must allow for the possibility of look-ahead bias in the ensuing analysis. ![enter image description here][1] With those caveats out of the way, let’s proceed. We are going to be using broad market data for the S&P 500 index in the analysis to follow, so the first step is to download daily price series for the index. Note that we begin with daily opening prices, since we intend to illustrate the application of news sentiment analysis with a theoretical day-trading strategy that takes positions at the start of each trading session, exiting at market close. ![enter image description here][2] From there we calculate the intraday return in the index, from market open to close, as follows: ![enter image description here][3] **Text Analysis & Classification** Next we turn to the task of reading the news archive and categorizing its content. Mathematica makes the importation of html pages very straightforward, and we can easily crop the raw text string to exclude page headers and footers. The approach I am going to take is to derive a sentiment indicator based on an analysis of the sentiment of each word in the daily archive. Before we can do that we must first convert the text into individuals words, stripping out standard stop-words such as "the" and "in" and converting all the text to lower case. Naturally one can take this pre-processing a great deal further, by identifying and separating out proper nouns, for example. Once the text processing stage is complete we can quickly summarize the content, for example by looking at the most common words, or by representing the entire archive in the form of a word cloud. Given that we are using the archive for the first business day of 2012, it is perhaps unsurprising that we find that "2012", "new" and "year" feature so prominently! archive = Import[StringJoin["http://www.wsj.com/public/page/archive-", DateString[ datelist[[1]], {"Year", "-", "MonthShort", "-", "DayShort"}], ".html"]]; archive = StringDrop[archive, StringPosition[archive, DateString[ datelist[[1]], {"MonthName", " ", "DayShort", ", ", "Year"}]][[1, 2]]]; archive = StringTake[ archive, -1 + StringPosition[archive, "ARCHIVE FILTER"][[1, 1]]]; archivewords = ToLowerCase[DeleteStopwords[TextWords[archive]]]; TakeLargest[Counts[archivewords], 20] WordCloud[archivewords] <|"2012" -> 45, "new" -> 41, "year" -> 39, "u.s." -> 33, "2011" -> 21, "asia" -> 16, "iowa" -> 16, "crisis" -> 15, "biggest" -> 14, "news" -> 14, "markets" -> 14, "india" -> 13, "said" -> 13, "economic" -> 13, "market" -> 13, "oil" -> 13, "europe's" -> 13, "look" -> 12, "day" -> 12, "price" -> 12|> ![enter image description here][4] The subject of sentiment analysis is a complex one and I only touch on it here. For those interested in the subject I can recommend The Text Mining Handbook, by Feldman and Sanger, which is a standard work on the topic. Here I am going to employ a machine learning classifier provided with Mathematica 11. It is not terribly sophisticated (or, at least, has not been developed with financial applications especially in mind), but will serve for the purposes of this article. For those unfamiliar with the functionality, the operation of the sentiment classification algorithm is straightforward enough. For instance: ![enter image description here][5] We apply the algorithm to classify each word in the daily news archive and arrive at a sentiment indicator based on the proportion of words that are classified as "positive". The sentiment reading for the archive for Jan-3, 2012, for example, turns out to be 67.4%: ![enter image description here][6] **Sentiment Index Analytics** We can automate the process of classifying the entire WSJ archive with just a few lines of code, producing a time series for the daily sentiment indicator, which has an average daily value of 68.5% - the WSJ crowd tends to be bullish, clearly! Note how the 60-day moving average of the indicator rises steadily over the period from 2012 through Q1 2015, then abruptly reverses direction, declining steadily thereafter - even somewhat precipitously towards the end of 2016. WSJSentimentIndicator[date_] := Module[{d = date, archive, archivewords, WSJSI}, archive = Import[StringJoin["http://www.wsj.com/public/page/archive-", DateString[d, {"Year", "-", "MonthShort", "-", "DayShort"}], ".html"]]; archive = StringDrop[archive, StringPosition[archive, DateString[d, {"MonthName", " ", "DayShort", ", ", "Year"}]][[1, 2]]]; archive = StringTake[ archive, -1 + StringPosition[archive, "ARCHIVE FILTER"][[1, 1]]]; archivewords = ToLowerCase[DeleteStopwords[TextWords[archive]]]; WSJSI = #Positive/(#Negative + #Positive) &@ Counts[Classify["Sentiment", archivewords]] // N; {WSJSI, archivewords, archive}] ![enter image description here][7] As with most data series in investment research, we are less interested in the level of a variable, such as a stock price, than we are in the changes in level. So the next step is to calculate the daily percentage change in the sentiment indicator and examine the correlation with the corresponding intraday return in the S&P 500 Index. At first glance our sentiment indicator appears to have very little predictive power - the correlation between indicator changes and market returns is negligibly small overall - but we shall later see that this is not the last word. ![enter image description here][8] **Conditional Distributions** Thus far the results appear discouraging; but as is often the case with this type of analysis we need to look more closely at the conditional distribution of returns. Specifically, we will examine the conditional distribution of S&P 500 Index returns when changes in the sentiment index are in the upper and lower quantiles of the distribution. This will enable us to isolate the impact of changes in market sentiment at times when the swings in sentiment are strongest. In the analysis below, we begin by examining the upper and lower third of the distribution of changes in sentiment: ![enter image description here][9] The analysis makes clear that the distribution of S&P 500 Index returns is very different on days when the change in market sentiment is large and positive vs. large and negative. The difference is not just limited to the first moment of the conditional distribution, where the difference in the mean return is large and statistically significant, but also in the third moment. The much larger, negative skewness means that there is a greater likelihood of a large decline in the market on days in which there is a sizable drop in market sentiment, than on days in which sentiment significantly improves. In other words, the influence of market sentiment changes is manifest chiefly through the mean and skewness of the conditional distributions of market returns. **A News Trading Algorithm** We can capitalize on these effects using a simple trading strategy in which we increase the capital allocated to a long-SPX position on days when market sentiment improves, while reducing exposure on days when market sentiment falls. We increase the allocation by a factor - designated the leverage factor - on days when the change in the sentiment indicator is in the upper 1/3 of the distribution, while reducing the allocation by 1/leveragefactor on days when the change in the sentiment indicator falls in lower 1/3 of the distribution. The allocation on other days is 100%. The analysis runs as follows: period = QuantityMagnitude@ DateDifference[First@datelist, Last@datelist, "Year"]; AnnStd = Sqrt[252]* StandardDeviation[ Transpose@{tsSPXreturns["Values"], strategyreturns}]; cf = {tsVTDSPX[Last@datelist]/1000 - 1, tsVTDstrategy[Last@datelist]/1000 - 1}; CAGR = -1 + (1 + cf)^(1/period); IR = CAGR/AnnStd; Print[Style["News Sentiment Strategy", "Subsection"]]; P1 = Style[ NumberForm[ TableForm[{CAGR, AnnStd, IR}, TableHeadings -> {{"CAGR", "Ann. StDev.", "IR"}, {Style["SP500 Index", Bold], Style["Strategy", Bold]}}], {6, 2}], FontSize -> 14]; P2 = DateListPlot[{tsVTDSPX, tsVTDstrategy}, Filling -> Axis, PlotLegends -> {"S&P500 Index", "Strategy"}, PlotLabel -> Style["Value of $1,000", Bold], ImageSize -> Medium]; Print[P1]; Print[P2]; ![enter image description here][10] It turns out that, using a leverage factor of 2.0, we can increase the CAGR from 10% to 21% over the period from 2012-2016 using the conditional distribution approach. This performance enhancement comes at a cost, since the annual volatility of the news sentiment strategy is 17% compared to only 12% for the long-only strategy. However, the overall net result is positive, since the risk-adjusted rate of return increases from 0.82 to 1.28. We can explore the robustness of the result, comparing different quantile selections and leverage factors using Mathematica's interactive Manipulate function: Manipulate[ percentiles = Quantile[tsSIchange, {quantilefrac, 1 - quantilefrac}]; bottompercentile = Flatten[Position[tsSIchange["Values"], x_ /; x < percentiles[[1]]]]; toppercentile = Flatten[Position[tsSIchange["Values"], x_ /; x > percentiles[[2]]]]; strategyreturns = tsSPXreturns["Values"]; strategyreturns[[bottompercentile]] = (1/leveragefactor)* strategyreturns[[bottompercentile]]; strategyreturns[[toppercentile]] = leveragefactor*strategyreturns[[toppercentile]]; tsVTDstrategy = TimeSeries[ Transpose[{datelist, 1000*FoldList[Times, 1, 1 + strategyreturns]}]]; DateListPlot[{tsVTDSPX, tsVTDstrategy}, Filling -> Axis, PlotLegends -> {"S&P500 Index", "Strategy"}, PlotLabel -> Style["Value of $1,000", Bold], ImageSize -> Medium], {leveragefactor, 1, 3}, {quantilefrac, 0.1, 0.5}] ![enter image description here][11] **Conclusion** A simple market sentiment indicator can be created quite easily from publicly available news archives, using a standard machine learning sentiment classification algorithm. A market sentiment algorithm constructed using methods as straightforward as this appears to provide the capability to differentiate the conditional distribution of market returns on days when changes in market sentiment are significantly positive or negative. The differences in the higher moments of the conditional distribution appears to be as significant as the differences in the mean. In principle, we can use the insight provided by the sentiment indicator to enhance a long-only day-trading strategy, increasing leverage and allocation on days when changes to market sentiment are positive and reducing them on days when sentiment declines. The performance enhancements resulting from this approach appear to be significant. Several caveats apply. The S&P 500 index is not tradable, of course, and it is not uncommon to find trading strategies that produce interesting theoretical results. In practice one would be obliged to implement the strategy using a tradable market proxy, such as a broad market ETF or futures contract. The strategy described here, which enters and exits positions daily, would incur substantial trading costs, that would be further exacerbated by the use of leverage. Of course there are many other uses one can make of news data, in particular with firm-specific news and sentiment analytics, that fall outside the scope of this article. Hopefully, however, the methodology described here will provide a sign-post towards further, more practically useful research. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig1.png&userId=773999 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig2.png&userId=773999 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig3.png&userId=773999 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig4a.png&userId=773999 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig5.png&userId=773999 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig6.png&userId=773999 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig8.png&userId=773999 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig9.png&userId=773999 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig10.png&userId=773999 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig10a.png&userId=773999 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fig15.png&userId=773999 Jonathan Kinlay 2016-11-28T20:41:07Z https://community.wolfram.com/groups/-/m/t/1646303 # UNET [](https://zenodo.org/badge/latestdoi/137186334) [](https://github.com/dwyl/esta/issues) ---------- ![Automated 3D muscle segmentation using UNET / RESNET using DIXON MRI data][17] A package to generate and train a UNET deep convolutional network for 2D and 3D image segmentation. Some code was based on [work][1] by [@Ali Hashmi][at0], which was also dicussed in [this post][2] The full version of the toolbox can be found on my [github page][3]. * [Information](#information) * [Install toolbox](#install-toolbox) * [Using the toolbox](#using-the-toolbox) * [Functionality](#functionality) * [Visualization](#visualization) * [Example](#example) ## Information UNET is developed for [Mathematica](https://www.wolfram.com/mathematica/). It contains the following toolboxes: - UnetCore - UnetSupport Documentation of all functions and their options is fully integrated in the Mathematica documentation. The toolbox always works within the latest version of Mathematica and does not support any backward compatibility. All code and documentation is maintained and uploaded to github using [Workbench](https://www.wolfram.com/workbench/). ## Install toolbox Install the toolbox in the Mathematica UserBaseDirectory > Applications. FileNameJoin[{$UserBaseDirectory, "Applications"}] ## Using the toolbox The toolbox can be loaded by using <<UNET` The notbook ``UNET.nb`` shows examples of how to use the toolbox on artificially generated 2D data. There are also examples how to visualize the layer of your trained network and how to visualize the training itself. ## Functionality The network supports multi channel inputs and multi class segmentation. * UNET generates a UNET convolutional network. * 2D UNET ![UNET 2D][4] * 3D UNET ![UNET 3D][5] * Loss Layers: Training the data is done using three loss layers: a SoftDiceLossLayer, BrierLossLayer and a CrossEntropyLossLayer. ![SoftDiceLossLayer, BrierLossLayer and a CrossEntropyLossLayer][6] * Convolution Blocks: The toobox contains five different convolution blocks that build up the network: [UNET][7], UResNet, [RestNet][8], UDenseNet, [DensNet][9]. ![Convolution blocks][10] * SplitTrainData splits the data and labels into training, validation and test data. ![split train Data][11] * TrainUNET trains the network. ![Train UNET][12] ## Visualization * Visualize the network and results. * Visualize the features of the layers. ![Visualize layer features][13] * Visualize the results. ![Visualize the results][14] * Animate the training process. ![UNET 2D animation][15] ![UNET 3D animation][16] ## Example * Example: 3D segmentation of lower legg muscles using MRI data. ![Automated 3D muscle segmentation using UNET / RESNET using DIXON MRI data][17] [1]: https://github.com/alihashmiii/UNet-Segmentation-Wolfram [2]: https://community.wolfram.com/groups/-/m/t/1341081?p_p_auth=w8PIeeiA [3]: https://github.com/mfroeling [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=UNET2D.PNG&userId=1332602 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=UNET3D.PNG&userId=1332602 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Loss.PNG&userId=1332602 [7]: https://arxiv.org/abs/1505.04597 [8]: https://arxiv.org/abs/1512.03385 [9]: https://arxiv.org/abs/1608.06993 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=convblocks.PNG&userId=1332602 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Split.PNG&userId=1332602 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Train.PNG&userId=1332602 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Visualize1.PNG&userId=1332602 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Visualize2.PNG&userId=1332602 [15]: https://community.wolfram.com//c/portal/getImageAttachment?filename=amin0-v2.gif&userId=1332602 [16]: https://community.wolfram.com//c/portal/getImageAttachment?filename=amin4-v2.gif&userId=1332602 [17]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Muscle_Segmentation.jpg&userId=1332602 [at0]: https://community.wolfram.com/web/alihashmi87 Martijn Froeling 2019-04-03T20:01:26Z https://community.wolfram.com/groups/-/m/t/465774 # Abstract We discuss classification as the data-mining technique in patterns recognition and supervised machine-learning. Comparison of classification to other data exploration methods is presented and the importance of this approach is further discussed. We demonstrate the machine-learning method in finance and show how credit risk categorisation can be easily achieved with classification technique. ![Classification][1] # Introduction Searching for patterns in economic and financial data has a long history. One of the oldest approached explored by econometricians was the data arrangement into groups based on their similarities and differences. As computer resources improved there was a growth in the scope and availability of new computational methods. However, the greatest data analysis challenges are undoubtedly more recent. Advances in observational and collection methods have generated, and continue to generate enormous volumes of information that present significant storage, retrieval and analysis challenges. The analytical tools that extract information from data can be placed into two broad and overlapping categories: (i) **cluster** and (ii) **classification** methods. Using these techniques, datasets can be classified according to a specific property by measurements indirectly related to the property of interest. An empirical relationship on the classification rule can be developed from a dataset for which the property of interest and the measurements are known. The classification rule can then be used to predict the property in samples that are not part of the original set. In view of the above, we view classification as a data-mining technique that comes particularly helpful in **finding patterns** in the data that can dramatically reduce its dimensionality. It helps identifying class membership from prediction variables and identify features that either contribute or can be used to predict class membership. # Machine learning Machine learning is a subset of artificial intelligence that incorporates a large number of statistical ideas and methods and uses algorithms that 'learn' from data. Learning is based around finding the relationship between a set of features in the data structure. **Induction** and **deduction** are two types of reasoning used in the learning process. Induction stems from the particular to the general and can be used to generate classification rules from examples by establishing 'cause-effect' relationships between existing facts. Deduction, on the other hand, is concerned with reasoning using existing knowledge. The requirement for previously acquired knowledge is one of the main differences between deduction and induction. Because deduction can be used to predict the consequences of events or to determine the prerequisite conditions for an observed event, it is at the heart of most expert systems. Machine learning is primarily based on induction. Whilst this is natural consequence of the method itself, it leads to paternally two problems: - Learning by induction requires potentially many examples to 'acquire knowledge' - It is not guaranteed to be 'future-proof' since the main principle is based on what happened in the past These difficulties are not restricted to machine learning. it is well known that relationships models based on regression should not be extrapolated beyond the limits of the original data. A good news, however, is that noise does not have consistent effects on machine learning algorithms; for example, neural networks, and instance-based methods all use 'distributed' representations This is important because distributed representations should be less susceptible to noise hence they have the potential to be more robust classifiers. ## Supervised learning Supervised learning applies to algorithms in which a 'teacher' continually assesses, or supervises, the classifier's performance during training by marking predictions as correct or incorrect. Naturally, this assumes that each case has a valid class label. Learning in this sense relates to the way in which a set of parameter values are adjusted to improve the classifiers' performance. ## Unsupervised learning On the other hand, in unsupervised learning there is no teacher and the classifier is left to find its own classes - known as clusters. The biggest benefits from an unsupervised classification are likely in knowledge-poor environments, particularly when there are large amounts of unlabeled data. As such clustering are viewed as a means of generating, rather than testing, hypotheses and related classifications. # Classification Classification is the placing of objects into predefined groups. This type of process is also called 'pattern recognition'. Why? The set of samples for which the property of interest and measurements is known is called the training set. The set of measurements that describe each sample in the data set is called a pattern. The determination of the property of interest by assigning a sample to its respective category is called recognition, hence the term pattern recognition. . As such, it is a collection of methods for 'supervised learning'. Clustering is related to classification. Both techniques place objects into groups or classes. The important difference is that the classes are not predefined in a cluster analysis. Instead objects are placed into 'natural' groups.Therefore, clustering algorithms are unsupervised. When we deal with large sets of multidimensional data, we can apply standard methods such as PCA (principal component analysis) or clustering. Both techniques attempt to analyse data without directly using information about the class assignment of the samples. Although both are powerful methods for uncovering relationships in large multivariate data sets, they are not sufficient for developing a classification rule that can accurately predict the class-membership of an unknown sample. Pattern recognition techniques fall into one of two categories: - Nonparametric discriminants - Similarity-based classifiers Nonparametric discriminants,.such as neural networks, attempt to divide a data space into different regions. In the simplest case, that of a binary classifier, the data space is divided into two regions. Samples that share a common property (e.g. risk type) will be found on one side of the decision surface, whereas those samples comprising the other category will be found on the other side. Classification based on random or chance separation.can be a serious problem if the data set is not sample rich. Because economic datasets usually contain more variables than samples, similarity-based classifiers are generally preferred. A basic assumption in similarity-base classification is that distances between points in the measurement space will be inversely related to their degree of similarity. Using a similarity-based classifier we can determine the class-membership of a sample by examining the class label of the data point closest to it in the measurement space. Classification in *Mathematica* 10 can be efficiently performed with the [Classify][2] command which offers good degree of flexibility and classification method selection. # Classification techniques in Finance Classification can be effectively used in the risk management and large dataset processing where group information is of particular interest. Consider the following case: financial institution has large portfolio of loans to its clients. It wants to establish the method for assessing the quality of lending portfolio by linking the maturity and credit spread on each loan to internal rating categories. We apply the Classification method to analyse the portfolio and to assign each loan to the rating class. Finally we produce the classified loans status based on the rating group ping. # Training set definition The financial institution can define the training set (the template for sample assignment) in the following way: - 6 Rating categories: AAA,, AA, A, BBB, BB, B - 3 maturity sets: 3 years / 5 years / 10 years - Credit spreads for each maturity / rating trainset = {{3, 50} -> "AAA", {5, 70} -> "AAA", {10, 85} -> "AAA", {3, 85} -> "AA", {5, 145} -> "AA", {10, 200} -> "AA", {3, 120} -> "A", {5, 200} -> "A", {10, 260} -> "A", {3, 210} -> "BBB", {5, 300} -> "BBB", {10, 390} -> "BBB", {3, 290} -> "BB", {5, 370} -> "BB", {10, 440} -> "BB", {3, 360} -> "B", {5, 470} -> "B", {10, 550} -> "B"}; By isolating credit spreads, we can visualise the training set credit spreads levels below: sprd = {{50, 70, 85}, {85, 145, 200}, {120, 200, 260}, {210, 300, 390}, {290, 370, 440}, {360, 470, 550}}; BarChart[sprd, ChartLegends -> {3 Y, 5 Y, 10 Y}, PlotLabel -> Style["Credit spreads per maturity/rating class", 15], ChartLabels -> {{"AAA", "AA", "A", "BBB", "BB", "B"}, None}] ![enter image description here][3] ## Setting the classifier We use the training set defined above to set up the classifier based on similarity measure using k-nearest neighbor method clsf = Classify[trainset, Method -> "NearestNeighbors"] ![enter image description here][4] and obtain the information on the classifier ClassifierInformation[clsf] ![enter image description here][5] We can investigate the classifier further by obtaining additional Information: ClassifierInformation[clsf, "MethodDescription"] ![enter image description here][6] # Classifier testing We can test and validate the classifier with the <ClassifierMeasurements> command. For example, we can define the validation test set as follows: vset = {{3, 46} -> "AAA", {5, 81} -> "AAA", {10, 92} -> "AAA", {3, 91} -> "AA", {5, 155} -> "AA", {10, 202} -> "AA", {3, 126} -> "A", {5, 215} -> "A", {10, 259} -> "A", {3, 219} -> "BBB", {5, 311} -> "BBB", {10, 399} -> "BBB", {3, 292} -> "BB", {5, 378} -> "BB", {10, 442} -> "BB", {3, 350} -> "B", {5, 460} -> "B", {10, 568} -> "B"}; ### Testing accuracy of the classifier cm = ClassifierMeasurements[clsf, vset] cm["Accuracy"] ![enter image description here][7] `1.` The classifier returns the correct class group ping for all categories with high precision. ### Measuring F-Score cm["FScore"] `<|"A" -> 1., "AA" -> 1., "AAA" -> 1., "B" -> 1., "BB" -> 1., "BBB" -> 1.|>` ### Generating Confusion matrix cm["ConfusionMatrixPlot"] ![enter image description here][8] The matrix confirms that the classifier assigns the sample test data to the rating classes correctly. There are no overlaps in the assignment. # Applying classifier to the financial data Applying classifier to the financial data Having defined and tested the classifier, we can start applying it to the financial data. Let's assume that financial institution has 50,000 loans with remaining maturity / market spread information. The sample of the loan portfolio is shown below: dat2 = Table[{RandomReal[10], RandomInteger[{70, 620}]}, {50000}]; Take[dat2, 75] ListPlot[%, PlotStyle -> Magenta, PlotLabel -> Style["Sample loan spreads per maturity", Blue, 15]] {{8.518356726448882`,454},{9.144298651160256`,608},{2.855364088031351`,184},{6.0566791640110615`,282},{1.5475654397289134`,266},{2.6417875173168994`,315},{4.966702524403626`,231},{0.19793692995662404`,605},{4.478200060877295`,213},{9.292431559943076`,294},{9.28184530814859`,447},{4.204004132022266`,234},{2.4319873689548466`,296},{0.34979911441219613`,499},{4.188999084728859`,139},{9.94074933345011`,392},{3.4739994419209292`,429},{8.95426428111189`,206},{8.554799166603996`,367},{1.4762114253028358`,498},{3.1821905760430784`,520},{9.366754802446565`,473},{1.6701704343156525`,244},{1.376279011603522`,232},{9.566319795447356`,333},{0.8703633914472597`,564},{6.71837887079888`,452},{7.302098986957059`,441},{2.6238815271294573`,239},{1.5336733840535572`,157},{1.4641136928541592`,146},{0.6465197801125662`,198},{1.222074695630404`,315},{5.702826552095841`,339},{6.731378893601473`,552},{8.808804286652013`,195},{0.6739420557112723`,314},{6.378960901113356`,393},{5.72547626395229`,445},{9.460117372811034`,417},{7.24575011914737`,376},{3.542651788089211`,280},{9.266105852374011`,473},{0.07409813721899816`,238},{6.042215952909746`,412},{4.264033025081259`,176},{2.6031280721072108`,541},{0.30796996377340946`,240},{7.365932988439827`,339},{6.898537558573874`,269},{7.588189071934799`,80},{7.64691230466779`,173},{4.097033804059016`,456},{7.365705637020458`,141},{6.499421726202534`,578},{8.431948572861224`,351},{6.241593889798619`,540},{5.458576916155449`,551},{1.2166050064107576`,340},{5.5963915988722395`,146},{5.958260855340793`,235},{1.5241468668957854`,595},{9.71283120935691`,382},{3.604748368454281`,226},{6.595824594015976`,431},{5.125769853255001`,619},{1.9166780841799138`,121},{9.506908583817971`,526},{2.793767635980249`,574},{5.83433511143844`,186},{7.708892750256794`,484},{4.099308138686737`,568},{5.6066580959611265`,527},{4.377895268594408`,146},{6.840495656634236`,380}} ![enter image description here][9] We apply the classifier to the spread data: clsloans = clsf[#] & /@ dat2; and group the results into rating classes rtdgroup = {"AAA", "AA", "A", "BBB", "BB", "B"}; rtdfreq = Table[Count[clsloans, i], {i, rtdgroup}] BarChart[rtdfreq, ChartStyle -> "Rainbow", ChartLegends -> rtdgroup, ChartLabels -> rtdgroup, LabelingFunction -> Above, PlotLabel -> Style["Rating classification of 50,000 loans portfolio", Blue, 15]] `{720, 9003, 6481, 8878, 6867, 18051}` ![enter image description here][10] We can see that the largest proportion of loans falls into B-rated category and the smallest to the AAA category. This is naturally expected. Further aggregation of the classified results can be done on Investment grade (>BB) and Non-Investment grade scale: {Total[rtdfreq] - Total[Take[rtdfreq, -2]], Total[Take[rtdfreq, -2]]}; BarChart[%, ChartStyle -> "Pastel", ChartLegends -> {"IG Loans", "Non-IG Loans"}, LabelingFunction -> Above, PlotLabel -> Style["Investment grade / Non-investment grade split", Blue, 15]] ![enter image description here][11] We can see equal split between IG and non-IG loans. ## Dynamic update of the portfolio The set-up classifier can work with dynamically changing portfolio. For example, we can add 100 new loans to the existing portfolio and visualise the result: newport=Table[Count[Join[clsloans,clsf[#]&/@ Table[{RandomReal[10],RandomInteger[{70,620}]},{50}]],i],{i,rtdgroup}]; {BarChart[rtdfreq,ChartStyle->"Rainbow",ChartLegends->rtdgroup,ChartLabels->rtdgroup, LabelingFunction->Above,PlotLabel->Style["Existing portfolio",Blue,15],ImageSize->290], BarChart[newport,ChartStyle->"TemperatureMap",ChartLegends->rtdgroup,ChartLabels->rtdgroup, LabelingFunction->Above,PlotLabel->Style["Portfolio with added 100 new assets",Blue,15],ImageSize->290]} ![enter image description here][12] We can also simulate the generation of new load assets and see how they are dynamically classified using the classification defined on the training set. Manipulate[BarChart[Table[Count[clsf[#] & /@ Table[{RandomReal[10], RandomInteger[{70, 620}]}, {\[Alpha]}], i], {i, rtdgroup}], ChartStyle -> "DarkRainbow", ChartLegends -> rtdgroup, ChartLabels -> rtdgroup, LabelingFunction -> Above, PlotLabel -> Style["Classified new loans in the portfolio", Blue, 15]], {{\[Alpha], 10, "Nb of new loans"}, 0, 100, 1}] # Review of the classification algorithms We can further investigate the behaviour of the classification algorithm by looking at the probability assignment boundaries for each class. We select the 5 year horizon for presentation purposes and pick the IG group: {Plot[clsf[{5,x},{"Probability","AAA"}],{x,0,150},PlotStyle->Blue,Filling->Axis,PlotLabel->Style["AAA Rated",13]], Plot[clsf[{5,x},{"Probability","AA"}],{x,0,240},PlotStyle->Magenta,Filling->Axis,PlotLabel->Style["AA Rated",13]], Plot[clsf[{5,x},{"Probability","A"}],{x,0,300},PlotStyle->Green,Filling->Axis,PlotLabel->Style["A Rated",13]], Plot[clsf[{5,x},{"Probability","BBB"}],{x,0,400},PlotStyle->Orange,Filling->Axis,PlotLabel->Style["BBB Rated",13]]} ![enter image description here][13] This 'digital-alike' behaviour of the NearestNeighbors algorithm is the one we actually desire in the probabilistic setting. This ensures limited overlap / spillage between the class boundaries. When we selected other classification algorithms, the result can be quite different. This is demonstrated below: ## Random Forest classification clsfRF = Classify[trainset, Method -> "RandomForest"]; method=clsfRF; {Plot[method[{5,x},{"Probability","AAA"}],{x,0,150},PlotStyle->Blue,Filling->Axis,PlotLabel->Style["AAA Rated",13]], Plot[method[{5,x},{"Probability","AA"}],{x,0,240},PlotStyle->Magenta,Filling->Axis,PlotLabel->Style["AA Rated",13]], Plot[method[{5,x},{"Probability","A"}],{x,0,300},PlotStyle->Green,Filling->Axis,PlotLabel->Style["A Rated",13]], Plot[method[{5,x},{"Probability","BBB"}],{x,0,400},PlotStyle->Orange,Filling->Axis,PlotLabel->Style["BBB Rated",13]]} ![enter image description here][14] ## Neural Networks classification clsfNN = Classify[trainset, Method -> "NeuralNetwork"]; method=clsfNN; {Plot[method[{5,x},{"Probability","AAA"}],{x,0,150},PlotStyle->Blue,Filling->Axis,PlotLabel->Style["AAA Rated",13]], Plot[method[{5,x},{"Probability","AA"}],{x,0,240},PlotStyle->Magenta,Filling->Axis,PlotLabel->Style["AA Rated",13]], Plot[method[{5,x},{"Probability","A"}],{x,0,300},PlotStyle->Green,Filling->Axis,PlotLabel->Style["A Rated",13]], Plot[method[{5,x},{"Probability","BBB"}],{x,0,400},PlotStyle->Orange,Filling->Axis,PlotLabel->Style["BBB Rated",13]]} ![enter image description here][15] ## Support Vector Machine clsfSVM = Classify[trainset, Method -> "SupportVectorMachine"]; method=clsfSVM; {Plot[method[{5,x},{"Probability","AAA"}],{x,0,150},PlotStyle->Blue,Filling->Axis,PlotLabel->Style["AAA Rated",13]], Plot[method[{5,x},{"Probability","AA"}],{x,0,240},PlotStyle->Magenta,Filling->Axis,PlotLabel->Style["AA Rated",13]], Plot[method[{5,x},{"Probability","A"}],{x,0,300},PlotStyle->Green,Filling->Axis,PlotLabel->Style["A Rated",13]], Plot[method[{5,x},{"Probability","BBB"}],{x,0,400},PlotStyle->Orange,Filling->Axis,PlotLabel->Style["BBB Rated",13]]} We observe very different classification schemes when the algorithm changes from NearestNeighbors to other alternatives. Whilst RandomForest generates patchy classification pattern, artificial NeuralNetworks or SupportVectorMachine produce smooth probabilistic curves. However, such smoothness may be less desirable in the rating assignment context. This is demonstrated below: ### Example: we want to test 4.5 years maturity and 110 bp spread. NearestNeighbors sees this as A-rated asset given the proximity measure clsf[{4.5, 110}, "Probabilities"] `<|"A" -> 0.722222, "AA" -> 0.0555556, "AAA" -> 0.0555556, "B" -> 0.0555556, "BB" -> 0.0555556, "BBB" -> 0.0555556|>` RandomForest generates multiple set of statistically important probabilities with less clear-cut assignment clsfRF[{4.5, 110}, "Probabilities"] `<|"A" -> 0.254235, "AA" -> 0.212853, "AAA" -> 0.167341, "B" -> 0.119395, "BB" -> 0.110873, "BBB" -> 0.135303|>` NeuralNetworks sees this spread as primarily BBB-rates asset, but there is still strong support for A-rating as well clsfNN[{4.5, 110}, "Probabilities"] `<|"A" -> 0.316899, "AA" -> 0.0229639, "AAA" -> 0.00319862, "B" -> 0.00256616, "BB" -> 0.0709275, "BBB" -> 0.583445|>` SupportVectorMachine provides similarly a set of statistically significant probabilities - seeing this as A-rates asset with other still being significant for quick conclusion clsfSVM[{4.5, 110}, "Probabilities"] `<|"A" -> 0.415746, "AA" -> 0.210153, "AAA" -> 0.105478, "B" -> 0.0119384, "BB" -> 0.0444652, "BBB" -> 0.212219|>` The **NearestNeighbors** classification scheme produces the most consistent result in terms of data association with the rating class. In the context of this particular assignment this is the preferred method. # Conclusion Classification is a powerful method for data organisation and dimensionality reduction in supervised machine learning. It is particularly useful when data has to be organised into class structure or given certain association with structural hierarchy. This is ideal for finding patterns in large datasets. Classification is well-suited technique for financial data analysis. This was demonstrated on the credit rating case where simple implementation led to a power full and robust scheme that was able to classify large portfolios with real-time updates. As such, classification is therefore essential for prudent risk management. [1]: /c/portal/getImageAttachment?filename=Classifcnimage.jpg&userId=387433 [2]: http://reference.wolfram.com/language/ref/Classify.html [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=sadfe34q5tyrhsgfdsa.png&userId=11733 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=dg34qgre.png&userId=11733 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=dfg5q4ywhntfgbsear4t54hr.png&userId=11733 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=34erwth.png&userId=11733 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=asdfq35yfdhgs.png&userId=11733 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=sdfacbxnmjguityer54.png&userId=11733 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=wer345tehrwrg5q3t.png&userId=11733 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=gfjhglkghjfg4356yrteaw.png&userId=11733 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=g5qy46whrt5q34hte.png&userId=11733 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=edffnbfg54hqt.png&userId=11733 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=sadft5rnhfgbd.png&userId=11733 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=fgt57697oiujtyhr.png&userId=11733 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=gdft45645y54yhrtghfsd.png&userId=11733 Igor Hlivka 2015-03-24T16:20:34Z https://community.wolfram.com/groups/-/m/t/1732399 Description ----------- I created a [website][1] that automatically makes a spot-the-difference challenge from various images. When given an image, I first find the specific items within the picture using ImageContents. I then remove or edit each of those contents and put the edited subimage back onto the original image, or in the case of removal, use InPaint to cover the gap. When the user arrives at the site, they choose a difficulty level, whether they would like one image or all of the images in that difficulty range, and an image. They can click to the very right in order to reveal the correct solution. General Image Functions ----------------------- The heart of the functionality in my project is ImageContents, a function which uses machine learning to find and identify things in a given image. Though ImageContents uses the same idea as ImageIdentify, it can find items in subimages and not only the entire image. For example, let’s take the ImageContents of this elephant image. It finds each elephant separately. ![enter image description here][2] One of the first problems I ran into was that Inpaint will sometimes use parts of the original image which contain other objects: for example, Inpaint might successfully cover the large elephant, but catch the small elephant in its mask, covering the large elephant with an unsettling floating baby elephant head. ![enter image description here][3] In order to solve this, I created my first function: `mask` which solves the issue of not being able to auto-generate masks based solely on their bounding boxes. boundingboxes[image_]:=Quiet[Check[ List @@@ Flatten@ Values@Normal@ImageContents[image, All, "BoundingBox"], {}]] The next function I created is called `expunge`, which utilizes `mask` to Inpaint the image contents. mask[{{col1_, row1_}, {col2_, row2_}}, {nrows_, ncols_}, previous_: None] := Table[If[nrows - row2 < row <= nrows - row1 + 1 && col1 < col <= col2, 1, If[previous === None, 0, previous[[row, col]]]], {row, nrows}, {col, ncols}] The next function I made is called ImageMapAt. ImageTrim trims the contents out of the image and applies the given function to the trimmed contents. ImageMapAt uses ImageCompose to shove the edited contents back into the original image, one at a time. Also, ImageMapAt uses the parameters {Left,Top} to align the edited contents correctly when pasting them back into the image, so it doesn't return outputs like this: ![enter image description here][4] ## Image Difference Functions ## Once all my general functions were working, my next step was to create functions to be applied to the images to automate the differences. These functions were based on built-in functions in the Wolfram Language, but specified to visible yet not too obvious variations. These include `blurred`, which blurs an image by 2 units, `darker` which darkens dominant colors, `aquarecolor` which replaces a dominant color with part of the gradient “Aquamarine”, and various others. ## Displaying Image Differences ## Once I had created differences, my next step was to display them. The idea is simple, merely subtract the original image from the one with differences. The difficult part was finding a way to display them nicely as the default had the differences placed on a black background making them hard to see and the ColorNegated version of it was too bright, so I used ImageAdjust. Here are some of the results. ![enter image description here][5] ## Image Distance Sorting ## Now that I had created and displayed the differences, I sorted the images into the order of larger changes to smaller ones using ImageDistance, because the larger the distance is, the bigger the change. For example, compare these two distances. Wouldn’t you agree the first one is roughly ten times easier than the second? ![enter image description here][6] ![enter image description here][7] ## Website Code ## I then created LibraryAdd, a function that is used to add images into my curated image library. Lastly, I created my website. The basic idea is that next to the original (left) and the new image (right), there will be a blank/white image that I refer to as "blank." The user clicks on the "empty space" once they think they know the difference, and then the correct difference is shown. If it is clicked again, the image returns to its original blank state. The code is shown below. CloudDeploy[ FormFunction[{"difficulty" -> {"easy" -> {0, 1/4}, "medium" -> {1/4, 1/2}, "hard" -> {1/2, 3/4}, "impossible" -> {3/4, 1}}, "selection" -> {"random", "all"}, "image" -> (CloudImport[First@# <> "/1.png"] -> First[#] & /@ Select[CloudObjects[ "gallery"], ! StringContainsQ[First@#, ".png"] &])}, With[{total = Length[CloudObjects[#image <> "/"]]/2, original = #image <> "/1"}, Grid[With[{orderly = Table[With[{new = #image <> "/" <> ToString[i]}, "<img height=\"200px\" src=\"" <> If[#2, blank, #] <> "\"" <> If[#2, " onclick=\"javascript:this.src=(this.src=='" <> # <> "' ? '" <> blank <> "' : '" <> # <> "');\"", ""] <> ">" & @@@ {original <> ".png" -> False, new <> ".png" -> False, new <> "a.png" -> True}], {i, TakeQuantile[Range[2, total], #difficulty]}]}, If[#["selection"] === "random", {RandomChoice[orderly]}, orderly]]]] &, AppearanceRules -> <|"Title" -> "Spot the Difference", "Description" -> "Choose a difficulty, whether you would like to see the \ solutions or not, and an image, then hit submit to try to spot the \ difference between the original image on the left and the edited one \ on the right. <i><a \ href=\"mailto:stella@maymin.com\">stella@maymin.com</a></i>"|>], \ "spot.me", Permissions -> "Public"] ## Future Work ## In the future, as an extension of my website, I would like to find a way to create more of a game, perhaps by using dynamics. Another extension may be to create a timer counting in seconds how long it took the user to click the correct difference, or a counter of how many incorrect clicks on the Locator there were. Using those high scores, I think it might be possible to create a Leaderboard across all users. This would create a competition between users all over the world, so they can try to beat each other’s high scores in the race to the top. Lastly, a simpler extension is to create the option of allowing the user to upload their own images, which would be saved in my master library. This would let users play with images others have uploaded. ## Acknowledgements ## I would like to thank my mentor, Rory Fougler, for helping me with my code every step of the way. It really made a huge difference. I would also like to thank Chip Hurst for providing various insights and helpful functions throughout my project. Lastly, I would like to thank Mads Bahrami, Kyle Keane, and Anna Musser for making this entire experience truly unforgettable. Overall, I am so grateful for my knowledge and capabilities now, thanks to my stay here at the Wolfram High School Summer Camp. [1]: https://www.wolframcloud.com/obj/stella/spot.me [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ImageContentselephants.png&userId=1720407 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=OopsPost.png&userId=1720407 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=oopsididitagainpost.png&userId=1720407 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=orderlydiffs.png&userId=1720407 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ImageDistance01.PNG&userId=1720407 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ImageDistance03.PNG&userId=1720407 Stella Maymin 2019-07-12T01:28:57Z https://community.wolfram.com/groups/-/m/t/1016315 ![enter image description here][1] # Introduction So here is a simple approach that applies Wolfram Language machine learning functions to a classification problem for finding possible Higgs particles. It uses a labeled data set with 30 numerical physical attributes (things like measured spins, angles, energies, etc.) and with labels being either 'signal' (s) or 'background' (b). The attached notebook runs through a sample analysis in the Wolfram Language: importing the training data, cleaning it up for using it, setting up a neural network, training the network with the data, and finally checking how well the trained neural network does at making predictions. Here is the description of data from the source website at KAGGLE: > Discovery of the long awaited Higgs boson was announced July 4, 2012 and confirmed six months later. 2013 saw a number of prestigious awards, including a Nobel prize. But for physicists, the discovery of a new particle means the beginning of a long and difficult quest to measure its characteristics and determine if it fits the current model of nature. > A key property of any particle is how often it decays into other particles. ATLAS is a particle physics experiment taking place at the Large Hadron Collider at CERN that searches for new particles and processes using head-on collisions of protons of extraordinarily high energy. The ATLAS experiment has recently observed a signal of the Higgs boson decaying into two tau particles, but this decay is a small signal buried in background noise. > The goal of the Higgs Boson Machine Learning Challenge is to explore the potential of advanced machine learning methods to improve the discovery significance of the experiment. No knowledge of particle physics is required. Using simulated data with features characterizing events detected by ATLAS, your task is to classify events into "tau tau decay of a Higgs boson" versus "background." > The winning method may eventually be applied to real data and the winners may be invited to CERN to discuss their results with high energy physicists. # References and sources - [Learning to discover: the Higgs boson machine learning challenge][2] - [KAGGLE: Higgs Boson Machine Learning Challenge][3] - [Opendata ATLAS][4] # Training data Import the training data: training = Import["D:\\machinelearning\\higgs\\training\\training.csv", "Data"]; Dimensions[training] `{250001, 33}` Look at the data fields (they are described in the pdf link above). "EventId" should not be used as part of the training, since it has no predictive value. The last column "Label" is the classification (s=signal, b=background) training[[1]] `{"EventId", "DER_mass_MMC", "DER_mass_transverse_met_lep", \ "DER_mass_vis", "DER_pt_h", "DER_deltaeta_jet_jet", \ "DER_mass_jet_jet", "DER_prodeta_jet_jet", "DER_deltar_tau_lep", \ "DER_pt_tot", "DER_sum_pt", "DER_pt_ratio_lep_tau", \ "DER_met_phi_centrality", "DER_lep_eta_centrality", "PRI_tau_pt", \ "PRI_tau_eta", "PRI_tau_phi", "PRI_lep_pt", "PRI_lep_eta", \ "PRI_lep_phi", "PRI_met", "PRI_met_phi", "PRI_met_sumet", \ "PRI_jet_num", "PRI_jet_leading_pt", "PRI_jet_leading_eta", \ "PRI_jet_leading_phi", "PRI_jet_subleading_pt", \ "PRI_jet_subleading_eta", "PRI_jet_subleading_phi", "PRI_jet_all_pt", \ "Weight", "Label"}` Sample vector: training[[2]] `{100000, 138.47, 51.655, 97.827, 27.98, 0.91, 124.711, 2.666, 3.064, \ 41.928, 197.76, 1.582, 1.396, 0.2, 32.638, 1.017, 0.381, 51.626, \ 2.273, -2.414, 16.824, -0.277, 258.733, 2, 67.435, 2.15, 0.444, \ 46.062, 1.24, -2.475, 113.497, 0.00265331, "s"}` Set up a simple neural network (this can be tinkered with to improve the results): net=NetInitialize[ NetChain[{ LinearLayer[3000],Ramp,LinearLayer[3000],Ramp,LinearLayer[2],SoftmaxLayer[] }, "Input"->{30}, "Output"->NetDecoder[{"Class",{"b","s"}}] ]] ![enter image description here][5] Set up the training data: data = Map[Take[#, {2, 31}] -> Last[#] &, Drop[training, 1]]; Numerical vectors that each point to a classification (s or b): RandomSample[data, 3] `{{87.06, 23.069, 67.711, 162.488, -999., -999., -999., 0.903, 4.245, 318.43, 0.523, 0.839, -999., 105.019, -0.404, 1.612, 54.943, -0.898, 0.855, 13.541, 1.729, 409.232, 1, 158.469, -1.363, -1.762, -999., -999., -999., 158.469} -> "b", {163.658, 55.559, 116.84, 50.019, -999., -999., -999., 2.855, 38.623, 130.132, 0.906, 1.359, -999., 51.993, 0.36, 2.142, 47.112, -0.932, -1.596, 22.782, 2.663, 191.557, 1, 31.026, -2.333, 0.739, -999., -999., -999., 31.026} -> "b", {100.248, 27.109, 60.729, 132.094, -999., -999., -999., 1.405, 10.063, 218.474, 0.541, 1.414, -999., 62.519, -0.401, -1.974, 33.817, -0.893, -0.657, 54.857, -1.298, 396.228, 1, 122.137, -2.369, 1.689, -999., -999., -999., 122.137} -> "s"}` # Training Length[data] `250000` {tdata,vdata}=TakeDrop[data,240000]; result=NetTrain[net,tdata,TargetDevice->"GPU",ValidationSet->Scaled[0.1],MaxTrainingRounds->1000] ![enter image description here][6] DumpSave["D:\\machinelearning\\higgs\\higgs.mx", result]; # Testing This is the test data (unlabeled): test = Import["D:\\machinelearning\\higgs\\test\\test.csv", "Data"]; Extract the validation data: validate = Map[Take[#, {2, 31}] &, Drop[test, 1]]; Predictions made on the unlabeled data: result /@ RandomSample[validate, 5] `{"b", "b", "s", "b", "b"}` Sample from the labeled data and compute classifier statistics: cm = ClassifierMeasurements[result, RandomSample[vdata, 1000]] ![enter image description here][7] cm["Accuracy"] `0.846` Plot the confusion matrix: cm["ConfusionMatrixPlot"] ![enter image description here][8] [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ATLASEXP_image.png&userId=20103 [2]: https://higgsml.lal.in2p3.fr/files/2014/04/documentation_v1.8.pdf [3]: https://www.kaggle.com/c/higgs-boson [4]: http://opendata.cern.ch/about/ATLAS [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=5393ty56yetjhw4.png&userId=20103 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=5256567urytere.png&userId=20103 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=rtyee567rutyrd.png&userId=20103 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ghf56utyehw65eyhrw.png&userId=20103 Arnoud Buzing 2017-02-17T14:16:25Z https://community.wolfram.com/groups/-/m/t/1647625 I recently wanted to learn more about recurrent neural networks (RNNs) and how to use them for regression problems. In particular, I wanted to try and combine RNNs with the ideas for Bayesian error bar estimation I discussed in my blog/community posts a while back: https://blog.wolfram.com/2018/05/31/how-optimistic-do-you-want-to-be-bayesian-neural-network-regression-with-prediction-errors/ https://community.wolfram.com/groups/-/m/t/1319745 I also found the found the following answer on StackExchange about sequence learning with `SequenceAttentionLayer`, which helped me quite a bit to understand how to use to use these layers: https://mathematica.stackexchange.com/a/143672/43522 With those resources in hand, I wanted to give time-series regression a shot. First, let's use `FinancialData` as an easy source of a test dataset. Here I download 5 years of data from 5 different indices and turn them into a regularly sampled matrix of vectors and then split the dataset into 80/20% training/test data: indices = {"GOOGL", "AS:ASML", "MSFT", "AAPL", "GE"}; data = FinancialData[indices, {Today - 5 Quantity[1, "Years"], Today}]; DateListPlot[data] With[{ matrix = N@Normal[ TimeSeriesRescale[ TimeSeriesResample[TimeSeriesThread[Identity, TimeSeries /@ data]], {0, Max[Length /@ data] - 1}] ][[All, 2]] }, {fe, data} = FeatureExtraction[matrix, "StandardizedVector", {"ExtractorFunction", "ExtractedFeatures"}] ]; {trainingData, testData} = TakeDrop[data, Round[0.8*Length[data]]]; Dimensions /@ {trainingData, testData} ![enter image description here][1] Next, we define a function that can generate training data for the network from the data matrix we just made: Clear[genFun]; genFun[data_, { seqLengthIn_, seqLengthout_, offset_}] := With[{ dim = Dimensions[data][[2]], max = Max[1, Length[data] - seqLengthout - offset - seqLengthIn - 2] }, Function[ With[{ri = RandomInteger[{1, max}, #BatchSize]}, <| "Input" -> Function[data[[# ;; # + seqLengthIn - 1]]] /@ ri, "Target" -> Join[ ConstantArray[{ConstantArray[0., dim]}, #BatchSize], Function[With[{j = # + offset + seqLengthIn - 1}, data[[j ;; j + seqLengthout - 1]]]] /@ ri, 2 ] |>]]]; Each time the function is called, it generates a batch of sequences that will be used as training input and target sequences that the network should try to learn and reproduce, but the targets are prepended with a blank start-of-sequence vector: Dimensions /@ genFun[data, {10, 6, 1}][<|"BatchSize" -> 3|>] Out[72]= <|"Input" -> {3, 10, 5}, "Target" -> {3, 7, 5}|> Next we put together our neural network: Clear[encoderNetAttend, decoderNet, lossLayer, trainingNetAttention]; encoderNetAttend[n_, dropout_] := NetGraph[<| "lstm" -> LongShortTermMemoryLayer[n, "Dropout" -> dropout], "last" -> SequenceLastLayer[], "post" -> NetChain[{LinearLayer[n], ElementwiseLayer["SELU"], DropoutLayer[], LinearLayer[]}] |>, {"lstm" -> "last" -> "post" -> NetPort["StateInit"], "lstm" -> NetPort["States"]} ]; decoderNet[n_, dropout_] := NetGraph[ <| "part1" -> PartLayer[{All, 1}], "part2" -> PartLayer[{All, 2}], "lstm" -> LongShortTermMemoryLayer[n, "Dropout" -> dropout], "shape1" -> ReshapeLayer[{n, 1}], "shape2" -> ReshapeLayer[{n, 1}], "catState" -> CatenateLayer[2] |>, { NetPort["Input"] -> NetPort["lstm", "Input"], NetPort["lstm", "Output"] -> NetPort["Output"], NetPort["StateIn"] -> "part1" -> NetPort["lstm", "State"], NetPort["StateIn"] -> "part2" -> NetPort["lstm", "CellState"], NetPort["lstm", "State"] -> "shape1", NetPort["lstm", "CellState"] -> "shape2", {"shape1", "shape2"} -> "catState" -> NetPort["StateOut"] }, "StateIn" -> {n, 2} ]; lossLayer["LogPrecision"] = ThreadingLayer@Function[{yObserved, yPredicted, logPrecision}, (yPredicted - yObserved)^2*Exp[logPrecision] - logPrecision ]; regressionNet[n_, dimIn_, dropout : _?NumericQ : 0.5] := NetInitialize@NetGraph[ <| "encoder" -> encoderNetAttend[n, dropout], "decoder" -> decoderNet[n, dropout], "attention" -> SequenceAttentionLayer["Bilinear"], "catenate" -> CatenateLayer[2], "regression" -> NetMapOperator[LinearLayer[{2, dimIn}]], "part1" -> PartLayer[{All, 1}], "part2" -> PartLayer[{All, 2}] |>, { NetPort["Input"] -> "encoder", NetPort["encoder", "StateInit"] -> NetPort["decoder", "StateIn"], NetPort["encoder", "States"] -> NetPort["attention", "Input"], NetPort["Prediction"] -> NetPort["decoder", "Input"], NetPort[{"decoder", "Output"}] -> NetPort["attention", "Query"], {"decoder", "attention"} -> "catenate" -> "regression", "regression" -> "part1" -> NetPort["mu"], "regression" -> "part2" -> NetPort["logTau"] }, "Input" -> {"Varying", dimIn}, "Prediction" -> {"Varying", dimIn} ]; trainingNetAttention[n_, dimIn_, dropout : _?NumericQ : 0.5] := NetInitialize@NetGraph[ <| "regression" -> regressionNet[n, dimIn, dropout], "loss" -> lossLayer["LogPrecision"], "total" -> SummationLayer[], "most" -> SequenceMostLayer[], "rest" -> SequenceRestLayer[] |>, { NetPort["Input"] -> NetPort["regression", "Input"], NetPort["Target"] -> "most" -> NetPort["regression", "Prediction"], NetPort["Target"] -> "rest", {"rest", NetPort["regression", "mu"], NetPort["regression", "logTau"]} -> "loss" -> "total" -> NetPort["Loss"] }, "Input" -> {"Varying", dimIn}, "Target" -> {"Varying", dimIn} ]; nHidden = 250; pDrop = 0.25; NetInformation[encoderNetAttend[nHidden, pDrop], "SummaryGraphic"] NetInformation[decoderNet[nHidden, pDrop], "SummaryGraphic"] NetInformation[regressionNet[nHidden, 2, 0.25], "SummaryGraphic"] NetInformation[ trainingNetAttention[nHidden, 2, 0.25], "SummaryGraphic"] ![enter image description here][2] The idea here is that the encoder produces a state vector that the decoder uses to start making predictions. The decoder, in turn, accepts the networks' own predictions to iteratively crank out the next one. During the training, though, we don't have the actual predictions from the network, so instead we feed in the sequence of actual target values and offset the target sequence at the loss layer by one with a `SequenceRest` layer. This is called *teacher forcing*. Time to train the network: seqConfig = {15, 7, 1}; trainedTrainObject = NetTrain[ trainingNetAttention[nHidden, Length[indices], pDrop], genFun[trainingData, seqConfig], All, TargetDevice -> "GPU", LossFunction -> "Loss", TimeGoal -> 30*60, BatchSize -> 200, ValidationSet -> genFun[testData, seqConfig][<|"BatchSize" -> 200|>] ]; trainedNet = trainedTrainObject["TrainedNet"]; trainedTrainObject["EvolutionPlots"] ![enter image description here][3] To make predictions, the network needs to eat its own results iteratively, which we can do with a `NestList` makePredictorFunction[trainedNet_, evaluationMode : ("Test" | "Train") : "Test", device : ("CPU" | "GPU") : "CPU"] := With[{ dim = Last@NetExtract[trainedNet, {"Input"}], encoder = NetExtract[trainedNet, {"regression", "encoder"}], regressionNet = NetDelete[NetExtract[trainedNet, "regression"], "encoder"] }, Function[{input, n}, Module[{ encodedInput = encoder[input, NetEvaluationMode -> evaluationMode, TargetDevice -> device], vectorInputQ = Depth[input] === 4, length = Length[input], reshape }, reshape = If[vectorInputQ, ConstantArray[#, Length[input]] &, Identity]; GeneralUtilities`AssociationTranspose@NestList[ regressionNet[ <|"Input" -> encodedInput["States"], "Prediction" -> #mu, "StateIn" -> #StateOut|>, NetEvaluationMode -> evaluationMode, TargetDevice -> device] &, <| "mu" -> reshape@{ConstantArray[0, dim]}, "StateOut" -> encodedInput["StateInit"] |>, n ][[2 ;;, {"mu", "logTau"}, If[vectorInputQ, All, Unevaluated[Sequence[]]], 1]] ] ] ]; For example, we can predict 5 steps ahead based on 30 points of historical data: predictor = makePredictorFunction[trainedNet, "Train", "CPU"]; Dimensions /@ predictor[data[[;; 30]], 7] Out[299]= <|"mu" -> {7, 5}, "logTau" -> {7, 5}|> Here's a manipulate to visualize the predictions against the actual data. We sample the network several times to get a sense of the uncertainty from the training weights and combine that uncertainty with the variance predicted by the network: Manipulate[ Manipulate[ Quiet@With[{ predictions = Dot[ {{1., 1.}, {1., 0.}, {1., -1.}}, {Mean[#mu], nstdev*Sqrt[Variance[#mu] + Mean[Exp[-#logTau]]]} &@Map[ Transpose, predictor[Table[dat[[Max[1, n - t] ;; UpTo[n]]], s], l] ][[{"mu", "logTau"}, All, All, index]] ], trueData = ListPlot[ MapIndexed[{First[#2] + Max[1, n - t] - 1, #1} &, Flatten[dat[[Max[1, n - t] ;; UpTo[n + l], index]]] ], Joined -> True, PlotStyle -> Black ] }, Show[ ListPlot[ Thread@Legended[ MapIndexed[ {First[#2] + Max[1, n - t] - 1, #1} &, Flatten[{ dat[[Max[1, n - t] ;; UpTo[n], index]], # }] ] & /@ predictions, {StringForm["\[Mu] + `1` \[Sigma]", Dynamic[nstdev]], "\[Mu]", StringForm["\[Mu] - `1` \[Sigma]", Dynamic[nstdev]]} ], Joined -> True, PlotStyle -> (Directive[#, Dashed] & /@ {Blue, Red, Blue}), ImageSize -> 500, Filling -> {1 -> {2}, 3 -> {2}} ], trueData, ImageSize -> Large, PlotRange -> All ] ], {{index, 1, "Index"}, Thread[Range[Length[indices]] -> indices]}, {{n, Round[Length[dat]/10], "Position"}, 1, Length[dat], 1}, {{t, 20, "History length"}, 1, 30, 1}, {{l, 30, "Extrapolation"}, 1, 50, 1}, {{s, 5, "Number of samples"}, 1, 20, 1}, {{nstdev, 2, "Standard deviations"}, 0, 4, 0.1}, Paneled -> False, ContinuousAction -> True, TrackedSymbols :> {index, n, t, l, s, nstdev}, SynchronousInitialization -> False ], {{dat, trainingData, "Dataset"}, {trainingData -> "Training data", testData -> "Test data"}}, TrackedSymbols :> {dat}, SynchronousInitialization -> False ] ![enter image description here][4] The results on the validation set do not look great, but the comparison between the first 4 years of the data and the last one may not be fair. Or it may simply be necessary to involve many more indices before it becomes feasible to do extrapolations of this type. At any rate, I hope that this code is useful to other users here! I will probably tweak bits of code here and there over the next few days and maybe add more commentary, so be sure to check back occasionally if you're interested. **Update** Some of the code needed to be updated slightly to work in the recently released Mathematica V12. The `SequenceAttentionLayer` is being phased out, but still works (see the "Properties & Relations" documentation for `AttentionLayer` for information about how to replace `SequenceAttentionLayer` with `AttentionLayer`). The financial data also needed a few more processing steps in V12 before it can be used. Please see the attached notebook for the V12 version of the code. **Update 2 (10 June 2019)** I attached another notebook with code for a neural network that does not use teacher forcing but instead feeds its own predictions back into itself during training. This makes sure that the network is trained in exactly the same way as it is used afterwards. This also removes the need to use a `NestList` afterwards to generate the predictions. Training the network becomes trickier, though, and you need to experiment more with learning rate multipliers and gradient clipping to get convergence. **Update 3 (13 July 2020)** I just updated the notebooks to make sure that all the code works correctly in Mathematica V12.1 To make this code work, you may need to update your neural network paclet first by evaluating: PacletUpdate["NeuralNetworks"] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=8936datelistplot.png&userId=839741 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=1613networks.png&userId=839741 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=progressplot.png&userId=839741 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=manipulate.png&userId=839741 Sjoerd Smit 2019-04-05T15:43:41Z https://community.wolfram.com/groups/-/m/t/1379001 # Punctuation Restoration With Recurrent Neural Networks Mengyi Shan, Harvey Mudd College, mshan@hmc.edu ![flow][1] All codes posted on GitHub: [https://github.com/Shanmy/Summer2018Starter/tree/master/Project][2]. Raw results in the attached notebook. ---------- ## Introduction In natural language processing problems such as automatic speech recognition (ASR), the generated text is normally unpunctuated, which is hard for further recognition or analysis. Thus punctuation restoration is a small but crucial problem that deserves our attention. This project aims to build an automatic "punctuation adding" tool for plain English text with no punctuation. Since the input text could be considered as a sequence in which context is important for every single word's properties, recurrent characteristics of neural networks are considered to be a good method. Traditional approaches to this problem include usage of various recurrent neural networks (RNN), especially long short-term memory layers (LSTM). This project examines several models built from different layers and introduces bidirectional operators which can significantly improve the result compared with old methods. ## Methods ![Basic steps of the method][3] There're four basic steps in the whole process. First, we get the corpus of articles (with punctuations). Then, we keep the periods and commas in the corpus but change the question marks, exclamation marks, and colons to periods and commas, while removing all other punctuations. With this pure text, we tag each word as one of {NONE, COMMA, PERIOD} by judging if it is followed by a punctuation or not. And this set of tagging rules are sent to a neural network model for training. Finally, we test the result on another piece of articles, which is the test set. ### Data Basically, we have two pieces of data. The first one is the Wikipedia text of 4000 nouns (deleting missing), and the second is 50 novels from Wolfram data repository. (*Get wikipedia text of 4000 nouns*) nounlist = Select[WordList[], WordData[#, "PartsOfSpeech"][[1]] == "Noun" &]; rawData = StringJoin @@ DeleteCases[Flatten[WikipediaData[#] & /@ Take[nounlist, {1, 4000}], 2], _Missing] (*Get text of 50 novels*) books = StringJoin @@ Get /@ ResourceSearch["novels", 50]; ### Pre-processing The first goal of the preprocessing step is to purify the text. That is, since we only consider commas and periods, we should either delete or replace other characters and punctuations. Also, for convenience, all numbers are replaced with 1 first. All other characters are removed from the text. (*Show sets of characters replaced with comma, period, whitespace, one and null respectively*) toComma = Characters[":;"]; toPeriod = Characters["!?"]; toWhiteSpace = {"-", "\n"}; toOne = {"0", "1", "2", "3", "4", "5", "6", "7", "8", "9"}; toNull[x_String] := Complement[Union[Characters[x]], ToUpperCase@Alphabet[], Alphabet[], toOne, toComma, toPeriod, toWhiteSpace, {".", ",", " "}]; Then complete the replacement and modify it to pure form. And we include a validation test to examine its purity. (*Replacement and modification. End with lowercase text with only periods, alphabets and commas.*) toPureText[x_String] := StringReplace[#, ".," .. -> ". "] &@ StringReplace[#, ". " .. -> ". "] &@ StringReplace[#, {" ," -> ",", " ." -> "."}] &@ StringReplace[#, {"1" .. -> "one", " " .. -> " "}] &@ StringReplace[#, {"1. 1" -> "", "1, 1" -> "1"}] &@ StringReplace[#, {" " .. -> " "}] &@ StringReplace[#, {"," -> ", ", "." -> ". "}] &@ ToLowerCase@ StringReplace[{toComma -> ",", toPeriod -> ".", toNull[x] -> "", toWhiteSpace -> " ", toOne -> "1"}][x]; (*validation test*) VerificationTest[Length@StringSplit@x == Length@TextWords@x] Then we define a function fPuncTag that can generate the corresponding tagging given a piece of text with punctuation. (*Define the tagging function, and maps it to original text. Original text is partitioned into pieces of 200 words*) fPuncTag := Switch[StringTake[#, -1], ".", "a", ",", "b", _, "c"] &; fWordTag[x_String] := Map[fPuncTag, Partition[StringSplit[x], 200], {2}]; And we can thus remove the punctuation, and build a set of rules between the unpunctuated text and the generated tagging. fWordText[x_String] := StringReplace[#, {"," -> "", "." -> ""}] & /@ StringRiffle /@ Partition[StringSplit[x], 200]; fWordTrain[x_String] := Normal@AssociationThread[fWordText[x], fWordTag[x]]; totalData = fWordTrain@toPureText@rawText With the total data, we want to divide it into three groups: the training set, the validation set, and the test set. (* First we know that the length is 63252, then we divide it by 15:3:1*) order = RandomSample[Range[63252]]; trainingSet = totalData[[Take[order, 50000]]]; validationSet = totalData[[Take[order, {50001, 60000}]]]; testSet = totalData[[Take[order, {60001, -1}]]]; ### Train During neural network training, I used 8 different combinations of layers, out of which 4 are worth considering. They are listed as followed. LSTM layer, gate recurrent layer, and basic recurrent layer are three types of recurrent layers, each representing a net that takes a sequence of vectors and outputs a sequence of the same length. LSTM is commonly used in natural language processing problems, so we start with it as a penetrating point. (*Pure LSTM*) net1 = NetChain[{ embeddingLayer, LongShortTermMemoryLayer[100], LongShortTermMemoryLayer[60], LongShortTermMemoryLayer[30], LongShortTermMemoryLayer[10], NetMapOperator[LinearLayer[3]], SoftmaxLayer["Input" -> {"Varying", 3}]}, "Output" -> NetDecoder[{"Class", {"a", "b", "c"}}] ]; (*Gate Recurrent*) net2 = NetChain[{ embeddingLayer, LongShortTermMemoryLayer[100], GatedRecurrentLayer[60], LongShortTermMemoryLayer[30], GatedRecurrentLayer[10], NetMapOperator[LinearLayer[3]], SoftmaxLayer["Input" -> {"Varying", 3}]}, "Output" -> NetDecoder[{"Class", {"a", "b", "c"}}] ]; (Basic Recurrent) net3 = NetChain[{ embeddingLayer, LongShortTermMemoryLayer[100], BasicRecurrentLayer[60], LongShortTermMemoryLayer[30], BasicRecurrentLayer[10], NetMapOperator[LinearLayer[3]], SoftmaxLayer["Input" -> {"Varying", 3}]}, "Output" -> NetDecoder[{"Class", {"a", "b", "c"}}] ]; (*Bidirectional*) net4 = NetChain[{ embeddingLayer, LongShortTermMemoryLayer[100], NetBidirectionalOperator[{LongShortTermMemoryLayer[40], GatedRecurrentLayer[40]}], NetBidirectionalOperator[{LongShortTermMemoryLayer[20], GatedRecurrentLayer[20]}], LongShortTermMemoryLayer[10], NetMapOperator[LinearLayer[3]], SoftmaxLayer["Input" -> {"Varying", 3}]}, "Output" -> NetDecoder[{"Class", {"a", "b", "c"}}] ]; The embedding layer is used to change words into vectors that represent their semantic characteristics. (*The embedding layer here*) embeddingLayer = NetModel["GloVe 100-Dimensional Word Vectors Trained on Wikipedia and Gigaword 5 Data"] With all those neural network models set up, we can train each neural network. To save time, I first trained all models with a small data set of only 3 million words to compare their behaviors. (*train the neural network while saving the training object*) NetTrain[net, trainingSet, All, ValidationSet -> validationSet] ### Test Since this classification problem is a problem of a skewed dataset, that is, most of the words should have the tag "None", it doesn't make sense to use "accuracy" to measure the models' behavior. Even if it simply do nothing and always return "None", it will have a high accuracy that is the percentage of "None" in the whole tagging set. Instead, to evaluate the behavior of the models, we introduce the concept of precision, recall, and f1-score. (*Precision and recall*) precision = truePrediction/allTrue recall = truePrediction/allPrediction F1 = HarmonicMean[{precision, recall}] ![PR][4] For a given test set, first, we want to remove its punctuations and run the trained model on it. (*romve punctuation and run the model*) noPuncTest = Keys /@ testSet result = net["TrainedNet"] /@ noPuncTest; Then we changed the tags to 1,2 and 0. And we calculate the elementwise product of realTag and resultTag. If an element is 4, it means that both the realTag and resultTag is 2, which counts as a successful prediction of a comma. An element of 2 represents a successful prediction of a period. ![Tag][5] (*Change the tags to numerical values and count 1s and 4s*) realTag = Replace[Flatten[Values /@ Take[testSet, 3252]], {"a" -> 1, "b" -> 2, "c" -> 0}, {1}]; resultTag = Replace[Flatten[result], {"a" -> 1, "b" -> 2, "c" -> 0}, {1}]; totalTag = realTag*resultTag; Now we can use totalTag, resultTag, and realTag to calculate precision, recall, and f1-score. (*Precision*) PrecPeriod = N@Count[totalTag, 1]/Count[resultTag, 1] PrecComma = N@Count[totalTag, 4]/Count[resultTag, 2] (*Recall*) RecPeriod = N@Count[totalTag, 1]/Count[realTag, 1] RecComma = N@Count[totalTag, 4]/Count[realTag, 2] (*F1*) F1Period = (2*RecPeriod*PrecPeriod)/(RecPeriod + PrecPeriod) F1Comma = (2*RecComma*PrecComma)/(RecComma + PrecComma) ## Result Ten neural networks are trained based on a small dataset with different layers. Only using Long Short-Term Memory layers gives an f1 score of 13% and 11% for periods and commas. Introducing dropout parameters, pooling layers, elementwise layers, basic recurrent layers and gate recurrent layers all produce an f1 score between 10% and 30%, showing no significant improvement. Introduction of the bidirectional operator (combining two recurrent layers) improves the scores to 53% and 47%, and to 72% and 60% respectively when training on a larger dataset of 10M words. Here are the results for the three different neural networks trained with a 3M small dataset, and bidirectional neural network (which has the best performance in the small dataset) trained with a larger dataset of 10M words. The first figure is of the period and the second is of the comma. ![Period][6] ![Comma][7] We can easily observe the advantage of the bidirectional operator in terms of both periods and commas, precision and recall. Instead of the sequence to sequence learning, "tagging" is a significantly more efficient and accurate way to restore punctuation in plain text. Since every words' tags ("None", "Comma", "Period") is influenced by its context, it makes sense that recurrent neural networks and bidirectional operators show great potential in this research. Generally, the recall score is significantly lower than the precision score, suggesting that the model generates too many punctuations than it should. This could be due to the dataset of Wikipedia which is not clean enough. In the Wikipedia text, sometimes there're equations, translations, or other strange characters that we simply delete. This changed the ratio of punctuations to words and produces some segments of text that is "full of" punctuations since all words are not recognized and simply deleted. One example of those "not clean segment" is shown below. ![wiki][8] Also, the overall performance on commas is slightly worse than on periods. This also makes sense from a linguistics point of view. There seems to be a concrete linguistics set of rules for the period, but the usage of comma greatly depends on personal writing style. For example, you could say either *"I like apples but I don't like bananas."*, or *"I like apples, but I don't like bananas."* In this way, it's really hard to build a model for comma prediction with such high accuracy. But fortunately, sometimes adding commas or not doesn't really influence the overall meaning of the sentence. So it's okay to be tolerant to a slightly worse performance on commas. ## Future Works 70% f1-score is still not enough for the application. Planned future work focuses on improving accuracy to a level suitable for usage in industry. The most urgent and important future work is using a larger data size. We can observe great improvement when changing from 3M to 10M dataset, but it's still far less than enough. ![plot][9] If we take a closer look at the evolution plots during training, we can see that the error rate and loss of training set are continuously decreasing, while the error rate and loss of the validation set soon reaches a stable state and doesn't change too much. The gap between those two curves suggests the possibility of overfitting, and it should greatly help if we introduce better and more data. Also, punctuation restoration should not be limited to periods and comma. A more rigorous study of the question mark, exclamation mark, colon, and quotation mark is expected. However, we should note that the choice of most punctuations is not restricted to one possibility. In cases like distinguishing a period with an exclamation mark, we cannot expect a high f1-score. But it's still an interesting topic, may be useful for topics like sentimental analysis. ## Acknowledgement I would like to thank the summer school for providing the environment and background skills for me to finish this project. Especially, I want to thank my mentor for helping me with neural network problems and debugging. ## Data and Reference - [Wolfram Data Repository][10] - [Wikipedia][11] - Tilk O, et al. "Lstm for Punctuation Restoration in Speech Transcripts." Proceedings of the Annual Conference of the International Speech Communication Association, Interspeech, 2015-January, 2015, pp. 683\687. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-10at7.10.22PM.png&userId=1362824 [2]: https://github.com/Shanmy/Summer2018Starter/tree/master/Project [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-11at10.22.30AM.png&userId=1362824 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=PR1.png&userId=1362824 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-11at11.45.44AM.png&userId=1362824 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-10at8.38.40PM.png&userId=1362824 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-10at8.38.49PM.png&userId=1362824 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-11at12.13.54PM.png&userId=1362824 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2018-07-10at8.59.14PM.png&userId=1362824 [10]: https://datarepository.wolframcloud.com/category/Text-Literature/ [11]: https://www.wikipedia.org Mengyi Shan 2018-07-11T17:20:45Z https://community.wolfram.com/groups/-/m/t/1732586 ![Feature Space Plot for speech data][1] ### *Figure Above: Feature Space Plot for speech data* ---------- ## Abstract In this project, I aim to design a system which is capable of detecting mood in human speech. Specifically, the system can be trained on a single user's voice given samples of emotional speech labeled as being angry, happy, or sad. The system is then able to classify future audio clips of the user's speech as having one of these three moods. I collected voice samples of myself and trained a Classifier Function based on this data. I experimented with many different available methods for the Classifier Function and for audio preprocessing to obtain the most accurate Classifier. ## Obtaining Data In this project, I focused on detecting mood for only one user (specifically, myself) since different people may express mood in different ways, leading to confusion for the Classifier. I initially recorded ten speech clips in each mood as training data, and ten different clips in each mood as testing data. I used [Audacity](https://www.audacityteam.org/) to record audio clips in 32-bit floating point resolution, exported as WAV files. If you're looking to replicate this project, you can use your own data, or use mine, available on [my GitHub repository](https://github.com/vedadehhc/MoodDetector). I later recorded additional clips, as discussed below. ## Feature Extraction In order to produce the most accurate Classifier, I extracted features which I thought were most useful to detecting mood. Specifically, I extracted the amplitudes, fundamental frequencies, and formant frequencies of each clip over the length of the clip using [AudioLocalMeasurements](https://reference.wolfram.com/language/ref/AudioLocalMeasurements.html). That is, I took multiple measurements of each of these values, over many partitions of the clip. I partitioned each clip into 75 parts, but feel free to experiment with different values. I also found the word rate, using Wolfram Language's experimental [SpeechRecognize](https://reference.wolfram.com/language/ref/SpeechRecognize.html) function, and the ratio of pausing time to total time of the clip using [AudioIntervals](https://reference.wolfram.com/language/ref/AudioIntervals.html). The final function takes the location of the audio file (could be local or on the web) and the number of partitions, and returns an association with the extracted features. It's important that the function return an association, since this makes things much easier when constructing the Classifier Function. The initial function for feature extraction is included below. extractFeatures[fileLocation_, parts_] := Module[{audio, assoc, pdur, amp, freq, form}, audio = Import[fileLocation]; pdur = AudioMeasurements[audio, "Duration"]/parts; amp = AudioLocalMeasurements[audio, "RMSAmplitude", PartitionGranularity -> pdur] // Normal; freq = AudioLocalMeasurements[audio, "FundamentalFrequency", MissingDataMethod -> {"Interpolation", InterpolationOrder -> 1}, PartitionGranularity -> pdur] // Normal; form = AudioLocalMeasurements[audio, "Formants", PartitionGranularity -> pdur] // Normal; assoc = <| Table["amplitude" <> ToString[i] -> amp[[i]][[2]], {i, Length[amp]}], Table["frequency" <> ToString[i] -> freq[[i]][[2]], {i, Length[freq]}], "wordrate" -> Length[TextWords[Quiet[SpeechRecognize[audio]]]]/ AudioMeasurements[audio, "Duration"], Table[Table["formant" <> ToString[i] <> "-" <> ToString[j] -> form[[i]][[j]], {j, Length[form[[i]]]}], {i, Length[form]}], "pauses" -> Total[Abs /@ Subtract @@@ AudioIntervals[audio, "Quiet"]]/ AudioMeasurements[audio, "Duration"] |>; Map[Normal, assoc, {1}] ] ### Get Training Data Using this function, I was able get the features for all of my training data. Here, I'll import my data from [my GitHub repository](https://github.com/vedadehhc/MoodDetector). angryTrainingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TrainingData/angry" <> ToString[i] <> ".wav", 75], {i, 10}]; happyTrainingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TrainingData/happy" <> ToString[i] <> ".wav", 75], {i, 10}]; sadTrainingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TrainingData/sad" <> ToString[i] <> ".wav", 75], {i, 10}]; ### Get Testing Data I also imported my testing data in the same way, so that it could be passed as an argument to the Classifier Function. I'll get this from GitHub as well. Note that this is the regular test data on GitHub. angryTestingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/Regular/a" <> ToString[i] <> ".wav", 75], {i, 10}]; happyTestingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/Regular/h" <> ToString[i] <> ".wav", 75], {i, 10}]; sadTestingFeats = Table[extractFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/Regular/s" <> ToString[i] <> ".wav", 75], {i, 10}]; ## Training a Classifier I constructed a Classifier Function using [Classify](https://reference.wolfram.com/language/ref/Classify.html). classifier = Classify[<|"angry" -> angryTrainingFeats, "happy" -> happyTrainingFeats, "sad" -> sadTrainingFeats|>, Method -> "LogisticRegression"] I ran the classifier on each set of training data. classifier[angryTrainingFeats] classifier[happyTrainingFeats] classifier[sadTrainingFeats] I experimented with all of the built-in options for [Method](https://reference.wolfram.com/language/ref/Method.html) to construct a Classifier Function with the best accuracy. I found that [LogisticRegression](https://reference.wolfram.com/language/ref/method/LogisticRegression.html) gave the best accuracy at 77% accuracy on the test data. However, I wanted to improve the accuracy further. ## Audio Pre-Processing One of the things that improved the Classifier's accuracy substantially was cleaning the audio before extracting features and classifying. Specifically, I trimmed the audio using [AudioTrim](https://reference.wolfram.com/language/ref/AudioTrim.html), and filtered each clip using [HighpassFilter](https://reference.wolfram.com/language/ref/HighpassFilter.html) before extracting features. The audio cleaning function is included below. cleanAudio[fileLocation_] := Module[{audio, trimmed, filtered}, audio = Import[fileLocation]; trimmed = AudioTrim[audio]; filtered = HighpassFilter[trimmed, Quantity[300, "Hertz"]] ] ### New Feature Extraction Function I updated the feature extraction function to include audio cleaning. extractCleanFeatures[fileLocation_, parts_] := Module[{audio, assoc, pdur, amp, freq, form}, audio = cleanAudio[fileLocation]; pdur = AudioMeasurements[audio, "Duration"]/parts; amp = AudioLocalMeasurements[audio, "RMSAmplitude", PartitionGranularity -> pdur] // Normal; freq = AudioLocalMeasurements[audio, "FundamentalFrequency", MissingDataMethod -> {"Interpolation", InterpolationOrder -> 1}, PartitionGranularity -> pdur] // Normal; form = AudioLocalMeasurements[audio, "Formants", PartitionGranularity -> pdur] // Normal; assoc = <| Table[ "amplitude" <> ToString[i] -> amp[[i]][[2]], {i, Length[amp]}], Table[ "frequency" <> ToString[i] -> freq[[i]][[2]], {i, Length[freq]}], "wordrate" -> Length[TextWords[Quiet[SpeechRecognize[audio]]]]/ AudioMeasurements[audio, "Duration"], Table[ Table["formant" <> ToString[i] <> "-" <> ToString[j] -> form[[i]][[j]], {j, Length[form[[i]]]}], {i, Length[form]}], "pauses" -> Total[Abs /@ Subtract @@@ AudioIntervals[audio, "Quiet"]]/ AudioMeasurements[audio, "Duration"] |>; Map[Normal, assoc, {1}] ] ### New Results Using audio cleaning, the accuracy of the classifier improved to 90%, much better than before. The results can be seen in the Confusion Matrix Plot below, generated using [ClassifierMeasurements](https://reference.wolfram.com/language/ref/ClassifierMeasurements.html). ![Confusion Matrix Plot for Regular Data][2] ## Testing with Neutral Statements Now, up until now, the statements used for recordings were all emotional in nature. For example, the clips recorded in an angry mood also had an angry statement being said. In order to control for this, I recorded neutral statements in each mood. That is, I recorded each of ten emotionally neutral statements in each of the three moods. If the Classifier still works on this data, that would show that it is not relying on the content of the speech, but on other audio features, which is the intended method. I imported these data using the clean feature extractor and, once again, I'll download them from my GitHub here. Note that this is the Neutral Statements data. nAngryCleanTestingFeats = Table[extractCleanFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/NeutralStatements/a" <> ToString[i] <> ".wav", 75], {i, 10}]; nHappyCleanTestingFeats = Table[extractCleanFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/NeutralStatements/h" <> ToString[i] <> ".wav", 75], {i, 10}]; nSadCleanTestingFeats = Table[extractCleanFeatures[ "https://github.com/vedadehhc/MoodDetector/raw/master/AudioData/ TestingData/NeutralStatements/s" <> ToString[i] <> ".wav", 75], {i, 10}]; ### Results for Neutral Statements I ran the clean Classifier on the new data, and the results were very positive. The classifier achieved an accuracy of 97% on the Neutral Statements data. The results are displayed in the Confusion Matrix Plot below, generated using [ClassifierMeasurements](https://reference.wolfram.com/language/ref/ClassifierMeasurements.html). ![Confusion Matrix Plot for Neutral Statements Data][3] ## Conclusions Overall, the Classifier was able to identify mood at 93% accuracy, even when the same statements were spoken in different moods. The composite Confusion Matrix Plot for all testing data can be seen below. ![Confusion Matrix Plot for all test data][4] ## Future Work In the future, I hope to improve the accuracy of the Classifier by providing additional training data, and to test it further with additional testing data. I also hope to expand the range of moods that the Classifier handles, including moods such as fear, calmness, and excitement. This, of course, would require the aforementioned additional data, and perhaps, more complex structures for the Classifier Function. In the future I would also like to experiment with multiple speakers, and determine whether classifiers for one speaker's moods can be used to determine those of another speaker. ## Acknowledgements I would like to thank my mentor Faizon Zaman for his guidance and assistance on this project. ## GitHub You can find my code for this project on [my GitHub repository](https://github.com/vedadehhc/MoodDetector), with all code available beginning July 12th, 2019. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=featureSpacePlotNoLabel.png&userId=1724789 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=CMP1.png&userId=1724789 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=CMP2.png&userId=1724789 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=CMP3.png&userId=1724789 Dev Chheda 2019-07-12T01:16:42Z https://community.wolfram.com/groups/-/m/t/1383518 ![A Representation of the emotion categorization system][1] ---------- #Abstract This project aims to develop a machine learning application to identify the sentiments in a music clip. The data set I used consists of one hundred 45-second clips from the Database for Emotional Analysis of Music and an additional 103 gathered by myself. I manually labeled all 203 clips and used them as training data for my program. This program works best with classical-style music, which is the main component of my data set, but also works with other genres to an reasonable extent. #Introduction One of the most important functions of music is to affect emotion, but the experience of emotion is ambiguous and subjective to individual. The same music may induce a diverse range of feelings in people as a result of different context, personality, or culture. Many musical features, however, usually lead to the same effect on the human brain. For example, louder music correlates more with excitement or anger, while softer music corresponds to tenderness. This consistency makes it possible to train a supervised machine learning program based on musical features. #Background This project is based on James Russell's circumplex model, in which a two-dimensional emotion space is constructed from the x-axis of valence level and y-axis of arousal level, as shown above in the picture. Specifically, valence is a measurement of an emotion's pleasantness, whereas arousal is a measurement of an emotion's intensity. Russell's model provides a metric on which different sentiments can be compared and contrasted, creating four main categories of emotion: Happy (high valence, high arousal), Stressed (low valence, high arousal), Sad (low valence, low arousal), and Calm (high valence, low arousal). Within these main categories there are various sub-categories, labeled on the graph above. Notably, "passionate" is a sub-category that does not belong to any main category due to its ambiguous valence value. ---------- #Program Structure The program contains a three-layer structure. The first layer is responsible for extracting musical features, the second for generating a list of numerical predictions based on different features, and the third for predicting and displaying the most probable emotion descriptors based on the second layer's output. ![enter image description here][2] ##First Layer The first layer consists of 23 feature extractors that generate numerical sequence based on different features: (*A list of feature extractors*) feMin[audio_] := Normal[AudioLocalMeasurements[audio, "Min", List]] feMax[audio_] := Normal[AudioLocalMeasurements[audio, "Max", List]] feMean[audio_] := Normal[AudioLocalMeasurements[audio, "Mean", List]] feMedian[audio_] := Normal[AudioLocalMeasurements[audio, "Median", List]] fePower[audio_] := Normal[AudioLocalMeasurements[audio, "Power", List]] feRMSA[audio_] := Normal[AudioLocalMeasurements[audio, "RMSAmplitude", List]] feLoud[audio_] := Normal[AudioLocalMeasurements[audio, "Loudness", List]] feCrest[audio_] := Normal[AudioLocalMeasurements[audio, "CrestFactor", List]] feEntropy[audio_] := Normal[AudioLocalMeasurements[audio, "Entropy", List]] fePeak[audio_] := Normal[AudioLocalMeasurements[audio, "PeakToAveragePowerRatio", List]] feTCent[audio_] := Normal[AudioLocalMeasurements[audio, "TemporalCentroid", List]] feZeroR[audio_] := Normal[AudioLocalMeasurements[audio, "ZeroCrossingRate", List]] feForm[audio_] := Normal[AudioLocalMeasurements[audio, "Formants", List]] feHighFC[audio_] := Normal[AudioLocalMeasurements[audio, "HighFrequencyContent", List]] feMFCC[audio_] := Normal[AudioLocalMeasurements[audio, "MFCC", List]] feSCent[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralCentroid", List]] feSCrest[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralCrest", List]] feSFlat[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralFlatness", List]] feSKurt[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralKurtosis", List]] feSRoll[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralRollOff", List]] feSSkew[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralSkewness", List]] feSSlope[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralSlope", List]] feSSpread[audio_] := Normal[AudioLocalMeasurements[audio, "SpectralSpread", List]] feNovelty[audio_] := Normal[AudioLocalMeasurements[audio, "Novelty", List]] <br/> ##Second Layer Using data generated from the first layer, the valence and arousal predictors of the second layer provide 46 predictions for the audio input, based on its different features. (*RMSAmplitude*) (*Feature extractor*) feRMSA[audio_] := Normal[AudioLocalMeasurements[audio, "RMSAmplitude", List]] dataRMSA = Table[First[takeLast[feRMSA[First[Take[musicFiles, {n}]]]]], {n, Length[musicFiles]}]; (*Generating predictor*) pArousalRMSA = Predict[dataRMSA -> arousalValueC] ![Sample predictor function][3] <br/> ##Third Layer The two parts of the third layer, main category classifier and sub-category classifier, each utilize the tensors generated in the second layer to make a prediction within their realm of emotion. The output consists of two parts, a main category emotion and a sub-category emotion. (*Main*) emotionClassify1 = Classify[classifyMaterial -> emotionList1, PerformanceGoal -> "Quality"] (*Sub*) emotionClassify2 = Classify[classifyMaterial -> emotionList2, PerformanceGoal -> "Quality"] ![enter image description here][4] <br/> ##Output If the program receives an input that is longer than 45 second, it will automatically clip the audio file into 45 second segments and return the result for each. If the last segment is less than 45 seconds, the program would still work fine on it, though with reduced accuracy. The display for each clip includes a main-category and a sub-category descriptor, with each of their associated probability also printed. ###Sample testing: Debussy's Clair de Lune ![enter image description here][5] <br/> ---------- #Conclusion The program gives very reasonable result for most music in the classical style. However, the program have three shortcomings that I plan to fix in later versions of the this program. Firstly, the program may give contradictory result (ex. happy and depressed) if the sentiment dramatically changes in the middle of a 45 second segment, perhaps reflecting the music's changing emotional composition. The current 45 second clipping window is rather long and thus prone to capture contradicting emotions. In the next version of this program, the window will probably be shortened to 30 or 20 seconds to reduce prediction uncertainty. Secondly, the program's processing speed has a lot of room of improvement. It currently takes about one and half minutes to compute an one minute audio file. In future versions I will remove relative ineffective feature extractors to speed things up. Lastly, the data used in creating this application is solely from myself, and therefore it is prone to my human biases. I plan to expand the data set with more people's input and more genres of music. I have attached the application to this post so that everyone can try out the program. #Acknowledgement I sincerely thank my mentor, Professor Rob Morris, for providing invaluable guidance to help me carry out the project. I also want to thank Rick Hennigan for giving me crucial support with my code. [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=8714Emotion2DSpace.PNG&userId=1371765 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2406DataStructure.PNG&userId=1371765 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=8990capture1.PNG&userId=1371765 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=5765capture2.PNG&userId=1371765 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=3607SampleTesting.PNG&userId=1371765 William Yicheng Zhu 2018-07-14T02:40:20Z https://community.wolfram.com/groups/-/m/t/3394441 [![Wolfram Language for biosciences research and teaching][2]][1] &[Wolfram Notebook][3] [1]: https://www.wolfram.com/biosciences/ [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Poster.png&userId=20103 [3]: https://www.wolframcloud.com/obj/93eb3e71-f5fe-46eb-a5b1-a1243a7fca0a Dimitar Krastev 2025-02-14T19:53:57Z