https://community.wolfram.com RSS Feed for Wolfram Community showing any discussions tagged with Artificial Intelligence sorted by most likes. https://community.wolfram.com/groups/-/m/t/3336417 Wolfram Notebook Assistant + LLM Kit is now rolling out! -------------------------------------------------------- > [**GET IT HERE»**][1] ![enter image description here][2] (For Stephen Wolfram's full introduction, see his blog: https://wolfr.am/NotebookAssistant) Notice: With this new subscription, we're rolling out updates to our notebook interfaces for version 14.1. When your system updates, you'll see new items on your standard toolbar. ---------- We're excited to introduce this product that provides new interfaces for computational AI assistance and also provides LLM access for programmatic uses as well. Over the course of its development we've already found a lot of good (and fun) uses for it and we're sure you'll find many more. Let me give a quick summary of the product's two parts: The **Notebook Assistant** is aimed at making working in Wolfram Notebooks easier. It provides context-aware suggestions to streamline your workflow based on the content of your notebook. Whether you're coding or writing, the assistant can help you with tasks such as auto-completing text, suggesting functions, and offering debugging tips. The **Wolfram LLM Kit** lets you use Wolfram as your provider, letting you use [Chat Notebooks][3] and [programmatic LLM functions][4] as simply as possible. You won't need to set up accounts or give your email to third-party providers like Open AI or Anthropic. We take care of all of that. **Who is this subscription really for?** If you use Wolfram Notebooks, then there's something in here for you. - Its not just for code! Like any good LLM, it can help with writing tasks by completing text or outlining documents. And unlike most, it can understand your data and computations while doing it. - If you're just starting with Wolfram Language code, tell the assistant what you're trying to do in plain language and it will help with the first steps toward coding your ideas. It isn't exactly a tutor but it can tell you to functions to use. If you hit problems on the way, it can help find alternatives. - If you're a veteran, it can help you by suggesting rarely-used functions or Options that you might not have thought of. It can access the stack while helping you debug your code. You can use the auto-complete to breeze through the drudgery of mundane code sections. We encourage you to check out Stephen Wolfram's blog and the gallery of examples to get inspired. We can't wait to see how you use these new tools to enhance your creativity and productivity in Wolfram Notebooks. Please share your experiences with the Wolfram Community so we can learn new and different ways to use these tools. - Get Notebook Assistant + LLM Kit: https://www.wolfram.com/notebook-assistant-llm-kit/ - Configuration help: https://support.wolfram.com/67504 - Chat notebook tutorial: https://reference.wolfram.com/language/tutorial/ChatNotebooks.html - LLM Functions: https://reference.wolfram.com/language/guide/LLMFunctions.html - Stephen Wolfram's blog announcement: https://wolfr.am/NotebookAssistant [1]: https://www.wolfram.com/notebook-assistant-llm-kit/ [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Example2.png&userId=462781 [3]: https://reference.wolfram.com/language/tutorial/ChatNotebooks.html [4]: https://reference.wolfram.com/language/guide/LLMFunctions.html Bradley Ashby 2024-12-09T20:51:01Z https://community.wolfram.com/groups/-/m/t/3062832 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=w5qgsdf.jpg&userId=11733 [2]: https://www.wolframcloud.com/obj/04458d24-aacf-4cd7-8a5d-46efde37e927 Marco Thiel 2023-11-09T22:51:38Z https://community.wolfram.com/groups/-/m/t/884348 The aim of my project for the Wolfram Science Summer School was to build a neural network which could be able to colorize grayscale images in a realistic way. The network has been built following the article [1]. In this paper, the authors propose a fully automated approach for colorization of grayscale images, which uses a combination of global image features, which are extracted from the entire image, and local image features, which are computed from small image patches. Global priors provide information at an image level such as whether or not the image was taken indoors or outdoors, whether it is day or night, etc., while local features represent the local texture or object at a given location. By combining both features, it's possible to leverage the semantic information to color the images without requiring human interaction. The approach is based on Convolutional Neural Networks, which have a strong capacity for learning and is trained to predict the chrominance of a grayscale image using the CIE L*a*b* colorspace. Predicting colors has the nice property that training data is practically free: any color photo can be used as a training example. **Net Layers** The model consists of four main components: a low-level features network, a mid-level features network, a global features network, and a colorization network. First, a common set of shared low-level features are extracted from the image. Using these features, a set of global image features and mid-level image features are computed. Then, the mid-level and the global features are both fused by a "fusion layer" and used as the input to a colorization network that outputs the final chrominance map. Each layer has a ReLu transfer function except for the last convolution of the colorization network, where a sigmoid function is applied.The model is able to process images of any size, but it is most efficient when the input images are 224x224 pixels, as the shared low-level features layers can share outputs. Note that when the input image size is of a different resolution, while the low-level feature weights are shared, a rescaled image of size 224x224 must be used for the global features network.This requires processing both the original image and the rescaled image through the low-level features network, increasing both memory consumption and computation time. For this reason, we trained the model exclusively with images of size 224x224 pixels. *Low-Level Features Network* A 6-layer Convolutional Neural Network obtains low-level features directly from the input image. The convolution filter bank the network represents are shared to feed both the global features network and the mid-level features network. In order to reduce the size of the feature maps, we use convolution layers with increased strides instead of using max-pooling layers (as usual for similar kinds of networks). If padding is added to the layer, the output is effectively half the size of the input layer. We used 3x3 convolution kernels exclusively and a padding of 1x1 to ensure the output is the same size (or half if using a stride of 2) as the input. *Global Features Network* The global image features are obtained by further processing the low-level features with four convolutional layers followed by three fully-connected layers.This results in a 256-dimensional vector representation of the image. *Mid-Level Features Network* The mid-level features are obtained by processing the low-level features further with two convolutional layers. The output is bottlenecked from the original 512-channel low-level features to 256-channel mid-level features. Unlike the global image features, the low-level and mid-level features networks are fully convolutional networks, such that the output is a scaled version of the input. *Fusion Layer* In order to be able to combine the global image features, a 256-dimensional vector, with the (mid-level) local image features, a 28x28x256-dimensional tensor, the authors introduce a fusion layer. This can be thought of as concatenating the global features with the local features at each spatial location and processing them through a small one-layer network.This effectively combines the global feature and the local features to obtain a new feature map that is, as the mid-level features, a 3D volume. *Colorization Network* Once the features are fused, they are processed by a set of convolutions and upsampling layers, which use the nearest neighbour technique so that the output is twice as wide and twice as tall. These layers are alternated until the output is half the size of the original input. The output layer of the colorization network consists of a convolutional layer with a Sigmoid transfer function that outputs the chrominance of the input grayscale image. Finally, the computed chrominance is upsampled and combined with the input intensity/luminance image to produce the resulting color image. In order to train the network, we used the Mean Square Error (MSE) criterion. Given a color image for training, the input of the model is the grayscale image while the target output is the a*b* components of the CIE L*a*b* colorspace. The a*b* components are globally normalized so they lie in the [0,1] range of the Sigmoid transfer function. *Colorization with Classification* While training with only color images using the MSE criterion does give good performance, sometimes it could make obvious mistakes due to not properly learning the global context of the image, e.g., whether it is indoors or outdoors. As learning these networks is an non-convex problem, we facilitated the optimization by also training for classification jointly with the colorization. As we trained the model using a large-scale dataset for classification of N classes (Mathematica ImageIdentify dataset), we had classification labels available for training. These labels correspond to a global image tag and thus can be used to guide the training of the global image features. We did this by introducing another very small neural network that consists of two fully-connected layers: a hidden layer with 256 outputs and an output layer with as many outputs as the number of classes in the dataset. The input of this network is the second to last layer of the global features network with 512 outputs. We trained this network using the cross-entropy loss, jointly with the MSE loss for the colorization network. *Implementation* The aim of my project was to build the network described in the paper using the new NeuralNetworks framework of Mathematica 11. In order to achieve this, some adjustments were needed. First of all, we decided to train and evaluate the network only on images of 224x224 pixels size, in order to use (and train) only one low-level features network, instead of two with shared weights and different outputs. The final network has two inputs: the first one is the colored 224x224 px image, encoded by the "NetEncoder" function in LAB colorspace, the second one the class of the image. The two outputs (named "Loss" and "Output") represent the values of the two loss function used (one for the colorization, the other one for the classification), which are then summed together by the NetTrain function. The three color channels of the input image are split by the split layer: the L channel feeds the "low-level features" network, while the a,b channels are scaled and concatenated in order to obtain a target set for the mean squared loss function comparable with the output of the colorization network. The fusion layer has been replaced by a broadcast layer, which joins the rank 3- tensor, output of the mid-level network, with the vector from the global features network. However, the way they are combined is not exactly the same as the one described in the paper. To evaluate the trained network on a grayscale image it's necessary to drop some branches of the network, such as the classification network and the layers that process the a,b channels of the colored input image in order to produce the target set for the colorization loss function. ![Network described in the paper][1] ![Network implementation with Mathematica NeuralNetworks framework][2] **Results** ![enter image description here][3] **Conclusions** The network described in the paper has been trained on the Places scene dataset [Zhou et al. 2014], which consists of 2,448,872 training images and 20,500 validation images, with 205 classes corresponding to the types of the scene. They filtered the images by removing grayscale images and those that have little color variance with a small automated script. They trained using a batch size of 128 for 200,000 iterations corresponding to roughly 11 epochs. This takes roughly 3 weeks on one core of a NVIDIAR TeslaR K80 GPU. We needed to introduce some new layers in the existing framework and to fix some bugs, so we were able to train our network only for 14 hours on a dataset of 350000 images on one core of a GPU Titan machine. Furthermore, the images in our training set mainly represent specific items, so probably better results may be achieved introducing also images of different types of subjects (landscapes, human created images, indoors, etc). The results we obtained are showed in the section above and are quite good. We are confident that with a deeper and longer training our network would give considerably better results. **Open Problems / Future Developments** Due to the separation between the global and local features, it is possible to use global features computed on one image in combination with local features computed on another image, to change the style of the resulting colorization. One of the more interesting things the model can do is adapting the colorization of one image to the style of another. This is straight-forward to do with this model due to the decorrelation between the global features and the mid-level features. In order to colorize an image A using the style taken from an image B, it's necessary to compute the mid-level local features of image A and the global features from image B. Than it's possible to fuse these features and process them with the colorization network. Both the local and the global features are computed from grayscale images: it's not necessary to use any color information at all. The main limitation of the method lies in the fact that it is datadriven and thus will only be able to colorize images that share common properties with those in the training set. In order to evaluate on significantly different types of images, it would be necessary to train a the model for all type of images (indoor, outdoor, human-created...). In order to obtain good style transfer results, it is important for both images to have some semantic level of similarity between them. **References** [1] Satoshi Iizuka, Edgar Simo - Serra, and Hiroshi Ishikawa."Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification". [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=netwPicture.png&userId=884315 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=MyNetwork.png&userId=884315 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=result2.png&userId=884315 Sabrina Giollo 2016-07-07T19:53:19Z https://community.wolfram.com/groups/-/m/t/1787163 Tuseeta-san's [post][1] is how to convert a trained model of TensorFlow to Mathematica. Converting trained models from a language other than Mathematica to Mathematica is very beneficial to Mathematica users. So I'll show how to convert a trained model of PyTorch to Mathematica along with Tuseeta-san's post. ![enter image description here][2] ## Step 1: Figure out the architecture ## The [model][3] to be converted is Pose Estimation that detects the human skeleton (body parts and their connections) from an image. It's called OpenPose.The model consists of Feature map that extracts image features and six Stage maps. Feature map extracts image features from an input image(size:368*368). Each Stage map has two-branch, the first branch predicts confidence and the second predicts PAFs?Part Affinity Fields), along with the image feature .Two-branch are concatenated for next stage. ## Step 2: Coding it in Mathematica ## **Feature map** The Feature map consists of the first 23 layers of VGG-19, followed by 2 sets of Convolution and Ramp. Extract the first 23 layers of VGG-19. vgg19 = NetModel["VGG-19 Trained on ImageNet Competition Data"]; vgg19sub = Take[vgg19, {1, 23}]; Change Encoder. enc = NetExtract[vgg19, "Input"]; enc = NetReplacePart[ enc, {"ImageSize" -> {368, 368}, "VarianceImage" -> {0.229, 0.224, 0.225}, "MeanImage" -> {0.485, 0.456, 0.406}}]; featurefirst = NetReplacePart[vgg19sub, "Input" -> enc]; Add Convolution and Ramp. feature = NetAppend[ featurefirst, {"convadd1" -> ConvolutionLayer[256, 3, "Stride" -> 1, "PaddingSize" -> 1], "reluadd1" -> Ramp, "convadd2" -> ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], "reluadd2" -> Ramp}]; **Stage map** Each Stage map consists only of Convolutions and Ramps. Stage 1: The differences between two branches is that the last output channel number is 38 or 19. blk11 = NetChain[{ ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[512, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[38, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; blk12 = NetChain[{ ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[128, 3, "Stride" -> 1, "PaddingSize" -> 1], Ramp, ConvolutionLayer[512, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[19, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; Stage 2?6: The difference between Stage1 and Stage2?6 is the kinds and the numbers of layers. blkx1 = NetChain[{ ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[38, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; blkx2 = NetChain[{ ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 7, "Stride" -> 1, "PaddingSize" -> 3], Ramp, ConvolutionLayer[128, 1, "Stride" -> 1, "PaddingSize" -> 0], Ramp, ConvolutionLayer[19, 1, "Stride" -> 1, "PaddingSize" -> 0]}]; Finally, create OpenPose. openpose = NetGraph[{ "feature" -> feature,(*feature*) "blk11" -> blk11, "blk12" -> blk12,(*stage 1*) "blk21" -> blkx1, "blk22" -> blkx2,(*stage 2*) "cat12" -> CatenateLayer[], "blk31" -> blkx1, "blk32" -> blkx2,(*stage 3*) "cat23" -> CatenateLayer[], "blk41" -> blkx1, "blk42" -> blkx2,(*stage 4*) "cat34" -> CatenateLayer[], "blk51" -> blkx1, "blk52" -> blkx2,(*stage 5*) "cat45" -> CatenateLayer[], "blk61" -> blkx1, "blk62" -> blkx2,(*stage 6*) "cat56" -> CatenateLayer[] }, {"feature" -> "blk11", "feature" -> "blk12",(*stage 1*) {"blk11", "blk12", "feature"} -> "cat12",(*stage 2*) "cat12" -> "blk21", "cat12" -> "blk22", {"blk21", "blk22", "feature"} -> "cat23",(*stage 3*) "cat23" -> "blk31", "cat23" -> "blk32", {"blk31", "blk32", "feature"} -> "cat34",(*stage 4*) "cat34" -> "blk41", "cat34" -> "blk42", {"blk41", "blk42", "feature"} -> "cat45",(*stage 5*) "cat45" -> "blk51", "cat45" -> "blk52", {"blk51", "blk52", "feature"} -> "cat56",(*stage 6*) "cat56" -> "blk61", "cat56" -> "blk62" }] ![enter image description here][4] ## Step 3: Importing the Weights and the Biases ## Download "[pose_model_scratch.pth][5]" as a trained model of PyTorch. Import the parameters, the weights and the biases. I referred "[How to import python pickle *.pkl?][6]" session = StartExternalSession["Python-NumPy"]; parameters = ExternalEvaluate[session, "import torch import numpy as np import pickle as pkl net_weights = torch.load( 'pose_model_scratch.pth', map_location={'cuda:0': 'cpu'}) keys = list(net_weights.keys()) parameters = {} for i in range(len(keys)): t = net_weights[keys[i]] x = t.numpy() parameters[keys[i]] = x.flatten() parameters"]; DeleteObject[session]; keys = Keys[parameters]; parameters = Values[parameters]; ## Step 4: Parsing the Weights and the Biases ## The parameters is 184 sets of one-dimensional lists. They consists of the Weights and the Biases of 92 Convolution layers in OpenPose. Get a list of layer names for OpenPose with depth level. Then, get a list of 92 names where convolution layer is used in it. layernames = GroupBy[Keys@NetInformation[openpose, "Layers"], First] // Values; convlayernames = (Position[ NetInformation[openpose, "Layers"], _ConvolutionLayer] // Flatten)[[All, 1]] ![enter image description here][7] As you can see in keys, you can see that the order of convolution layers of OpenPose is different from the order of convolution layers of "pose_model_scratch.pth" keys ![enter image description here][8] So, manually sort the order of convolution layers in OpenPose into the order of "pose_model_scratch.pth" convlayernamesGH = {{"feature", "conv1_1"}, {"feature", "conv1_2"}, {"feature", "conv2_1"}, {"feature", "conv2_2"}, {"feature", "conv3_1"}, {"feature", "conv3_2"}, {"feature", "conv3_3"}, {"feature", "conv3_4"}, {"feature", "conv4_1"}, {"feature", "conv4_2"}, {"feature", "convadd1"}, {"feature", "convadd2"}, {"blk11", 1}, {"blk11", 3}, {"blk11", 5}, {"blk11", 7}, {"blk11", 9}, {"blk21", 1}, {"blk21", 3}, {"blk21", 5}, {"blk21", 7}, {"blk21", 9}, {"blk21", 11}, {"blk21", 13}, {"blk31", 1}, {"blk31", 3}, {"blk31", 5}, {"blk31", 7}, {"blk31", 9}, {"blk31", 11}, {"blk31", 13}, {"blk41", 1}, {"blk41", 3}, {"blk41", 5}, {"blk41", 7}, {"blk41", 9}, {"blk41", 11}, {"blk41", 13}, {"blk51", 1}, {"blk51", 3}, {"blk51", 5}, {"blk51", 7}, {"blk51", 9}, {"blk51", 11}, {"blk51", 13}, {"blk61", 1}, {"blk61", 3}, {"blk61", 5}, {"blk61", 7}, {"blk61", 9}, {"blk61", 11}, {"blk61", 13}, {"blk12", 1}, {"blk12", 3}, {"blk12", 5}, {"blk12", 7}, {"blk12", 9}, {"blk22", 1}, {"blk22", 3}, {"blk22", 5}, {"blk22", 7}, {"blk22", 9}, {"blk22", 11}, {"blk22", 13}, {"blk32", 1}, {"blk32", 3}, {"blk32", 5}, {"blk32", 7}, {"blk32", 9}, {"blk32", 11}, {"blk32", 13}, {"blk42", 1}, {"blk42", 3}, {"blk42", 5}, {"blk42", 7}, {"blk42", 9}, {"blk42", 11}, {"blk42", 13}, {"blk52", 1}, {"blk52", 3}, {"blk52", 5}, {"blk52", 7}, {"blk52", 9}, {"blk52", 11}, {"blk52", 13}, {"blk62", 1}, {"blk62", 3}, {"blk62", 5}, {"blk62", 7}, {"blk62", 9}, {"blk62", 11}, {"blk62", 13} }; Get the position of each element of convlayernamesGH in OpenPose. convlayerpos = Flatten[Position[layernames, #] & /@ convlayernamesGH, 1] ![enter image description here][9] Reshape each one-dimensional list of parameters to the dimension of the corresponding weight or bias. getDimB[layer_] := Dimensions@NetExtract[layer, "Biases"] getDimW[layer_] := Dimensions@NetExtract[layer, "Weights"] convs = NetExtract[NetInitialize[openpose], #] & /@ convlayerpos; dimW = getDimW /@ convs; dimB = getDimB /@ convs; dim = Flatten[Transpose[{dimW, dimB}], 1]; parametersReshape = MapThread[ArrayReshape, {parameters, dim}]; ## Step 5: Linking the Weights and the Biases ## Replace the initial values of weights and biases in OpenPose with learned parameters, and finally get trained OpenPose. replacenames = Flatten[ Transpose[{Flatten@{#, "Weights"} & /@ convlayernamesGH, Flatten@{#, "Biases"} & /@ convlayernamesGH}], 1]; rule = Thread[replacenames -> parametersReshape]; trainedOpenPose = NetReplacePart[openpose, rule] ![enter image description here][10] ## Step 6: Making the tests ## For simplification, estimate pose for the image of single-person. The Output2 of OpenPose shows the confidence of 19 body parts in each part where the image is divided into 46 * 46. 1:Nose, 2:Neck, 3:RShoulder, 4:RElbow, 5:RWrist, 6:LShoulder, 7:LElbow, 8:LWrist, 9:RHip, 10:RKnee, 11:RAnkle, 12:LHip, 13:LKnee, 14:LAnkle, 15:REye, 16:LEye, 17:REar, 18:LEar, 19:Bkg Define the function to get the position of max of confidence of each body part. maxpts[img_, confidences_, idex_] := Module[{pos, pts, h}, pos = Reverse@First@Position[h = confidences[[idex]], Max@h]; pts = (pos/46)*ImageDimensions@img; pts = {pts[[1]], (ImageDimensions@img)[[2]] - pts[[2]]} ] Connect the detected body parts and show the result together on the original image. showpose[img_] := Module[{bodylist, size, out, confidences, pts, pose}, bodylist = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14}; size = {368, 368}; out = trainedOpenPose[img]; confidences = out[[2]]; pts = maxpts[img, confidences, #] & /@ bodylist; pose = Graphics[{ Yellow, Thickness[.0125], Line[pts[[#]] & /@ {1, 2}], Green, Line[pts[[#]] & /@ {2, 3, 4, 5}], Cyan, Line[pts[[#]] & /@ {2, 6, 7, 8}], Orange, Line[pts[[#]] & /@ {2, 9, 10, 11}], Magenta, Line[pts[[#]] & /@ {2, 12, 13, 14}], PointSize[Large], Red, Point[pts], White, Point[{{0, 0}, ImageDimensions@img}] }, ImagePadding -> All]; Show[img, pose] ] Let' try. img = Import["ichiro.jpg"]; showpose[img] ![enter image description here][11] ## Future work ## ? Estimate the pose of an image in which multi-person are by using PAFs of output1 of OpenPose. ? Convert a more accurate pose estimation model. [1]: https://community.wolfram.com/groups/-/m/t/1785523 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=00.jpg&userId=1013863 [3]: https://arxiv.org/pdf/1611.08050.pdf [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=752004.jpg&userId=1013863 [5]: https://www.dropbox.com/s/ae071mfm2qoyc8v/pose_model.pth?dl=0 [6]: https://mathematica.stackexchange.com/questions/181273/how-to-import-%5C%20python-pickle-pkl?rq=1 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=691905.jpg&userId=1013863 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=387908.jpg&userId=1013863 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=109406.jpg&userId=1013863 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=179107.jpg&userId=1013863 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=161401.jpg&userId=1013863 Kotaro Okazaki 2019-09-10T21:36:49Z https://community.wolfram.com/groups/-/m/t/3062403 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=fg45qw.jpg&userId=11733 [2]: https://www.wolframcloud.com/obj/34ac42c4-10de-4201-a158-6af47eda2bd8 Marco Thiel 2023-11-08T21:48:25Z https://community.wolfram.com/groups/-/m/t/2166833 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/cdf7d474-f4fb-4cbd-bbd5-f1fac8699f7a Siria Sadeddin 2021-01-18T22:46:11Z https://community.wolfram.com/groups/-/m/t/1221098 Introduction ---------- I recently came across a post for a computer program that in some fields intends to compete with the Wolfram Language and which has a toolbox for Machine Learning. I wanted to compare the work described with the workflow in Mathematica. The challenge is to classify old Japanese Characters from texts from the so-called Edo period: Style[StringTake[WikipediaData["Edo period"], 601], 16] ![enter image description here][1] The characters are written in running style and apparently are difficult to read for Japanese speakers today. We will use two approaches of machine learning in the Wolfram Language to tackle this problem. Historical Background ----------------------------- Before we proceed, let's have a look at the Tokugawa shogunate. I will use code borrow from [@Vitaliy Kaurov][at0] from his great article on ["Computational history: countries that are gone"][2]. The following lines compute the historic boundaries of the shogunate: Monitor[tokugawaPOLY = Table[EntityValue[Entity["HistoricalCountry", "TokugawaShogunate"], EntityProperty["HistoricalCountry", "Polygon", {"Date" -> DateObject[{t}]}]], {t, 1600, 1870}];, t] We then only filter for "changes" in the boundaries: tokugawaPOLY // Length tokugawaPOLYcomp = DeleteMissing[ DeleteDuplicates[Transpose[{Range[1600, 1870], tokugawaPOLY}], Last[#1] == Last[#2] &], 1, 2]; tokugawaPOLYcomp // Length In the 271 I consider in the first place there are only 4 changes to the boarders, which we can plot like so: GeoGraphics[{EdgeForm[Red], GeoStyling[Opacity[.1]], #} & /@ tokugawaPOLYcomp[[All, 2]], GeoProjection -> "Mercator", ImageSize -> 800, GeoBackground -> GeoStyling["StreetMap"], GeoRange -> Entity["Country", "Japan"], GeoRangePadding -> Quantity[800, "Kilometers"], GeoZoomLevel -> 6] ![enter image description here][3] So the boarders were indeed very stable for a long period reflecting the "no more wars" philosophy of the day. Arts and culture ---------- The stability led to a very rich art and literature scene. Here are some images that reflect the style of the time: ImageCollage["Thumbnail" /. Normal[Normal[WebImageSearch["Edo Period"]]], Method -> "Rows"] ![enter image description here][4] Our dataset ---------- Also literature flourished during this period. Our dataset will be taken from the [Center for Open Data in the Humanities][5]. The keen reader might have noticed that the page is in Japanese. Mathematica can translate the page, and [so can Google][6]. The dataset is called the "Japanese classic registered glyph data set"; ?????????????. TextTranslation["?????????????"] "Japan classics-type datasets". Following the first link (grey box) on the original Japanese page one gets to t[his page][7], from the bottom of which (look for a download symbol and the text "(TAR+GZ 24.57 MB)" the dataset can be downloaded. This is the google translation of the reference: > Japanese Classic Statement Character Data Set" (Kokugaku Kenkyu Other > Collection / CODH Processing) which in the original is "???????????????????????CODH??". The glyphs come from "bamboo letters" from 15 classic texts; the dataset contains more than 20000 glyphs of the 10 most frequent symbols. These letters were annotated by the "National Institute of Letters of Japan" (google). The dataset was also used in a [data/machine learning challenge][8]. After extracting the file to a folder on the desktop I can import the images with: characterImages = Import["/Users/thiel/Desktop/pmjt_sample_20161116/train_test_file_list.h5", "Data"]; Here is a sample of the glyphs: Grid[Partition[ImageAdjust[Image[#/256.]] & /@ characterImages[[3, 1 ;; 24]], 6], Frame -> All] ![enter image description here][9] Similar to the MNIST dataset, they consist of 28 times 28 pixels. They give the grey level as integers, which is why I divide by 256. This shows the pixelation of the glyphs: Image[characterImages[[3, 12]]/256.] // ImageAdjust ![enter image description here][10] The dataset conveniently contains a training and a test set. There are characterImages[[3]] // Length 19909 glyphs in the training set. All of these are annotated: Flatten[characterImages[[4]]] ![enter image description here][11] We can convert the training data to images like so: Monitor[trainingset = Table[ImageAdjust[Image[characterImages[[3, k]]/256.]] -> Flatten[characterImages[[4]]][[k]], {k, 1, 19909}];, k] The test set contains characterImages[[1]] // Length 3514 glyphs. The annotation of which are found in characterImages[[2]] // Flatten // Short ![enter image description here][12] This is how we prepare the test set data for our machine learning: Monitor[testset = Table[ImageAdjust[Image[characterImages[[1, k]]/256.]] -> Flatten[characterImages[[2]]][[k]], {k, 1, 3514}];, k] What are the 10 most frequent symbols? ---------- In one of the annotation files we find the character codes of the symbols so that we can map the classification to actual symbols. These are the glyphs we will consider: Rasterize /@ (Style[ FromCharacterCode[ ToExpression["16^^" <> StringJoin[Characters[#]]]], 100] & /@ {"3057", "306B", "306E", "3066", "308A", "3092", "304B", "304F", "304D", "3082"}) ![enter image description here][13] We can also make an image collage of them: ImageCollage[ Image /@ (Rasterize /@ (Style[ FromCharacterCode[ ToExpression["16^^" <> StringJoin[Characters[#]]]], 100] & /@ {"3057", "306B", "306E", "3066", "308A", "3092", "304B", "304F", "304D", "3082"})), Method -> "Rows"] ![enter image description here][14] I though it would be nice if they were in a more calligraphic form. So I tried to used ImageRestyle to try and make them more calligraphic. I think that the attempt was not very successful, but I want to show it anyway. First I download a couple of calligraphic symbols: caligraphy = "Thumbnail" /. Normal[Normal[WebImageSearch["japanese calligraphy"]]] I liked the 5th one, which I will use for the style transfer: calisymbols = ImageRestyle[#, caligraphy[[5]], PerformanceGoal -> "Quality"] & /@ (Rasterize /@ (Style[ FromCharacterCode[ ToExpression["16^^" <> StringJoin[Characters[#]]]], 100] & /@ {"3057", "306B", "306E", "3066", "308A", "3092", "304B", "304F", "304D", "3082"})) ![enter image description here][15] We can also remove the background to get an alternative representation: ![enter image description here][16] Machine Learning (Classify) ---------- Our first approach to classify the glyphs will be via Classify. The classification is very fast (only a couple of seconds on my machine) and quite accurate: standardcl = Classify[trainingset, ValidationSet -> testset, PerformanceGoal -> "Quality"] We can first calculate the ClassifierMeasurements cmstandard = ClassifierMeasurements[standardcl, testset] and the accuracy: cmstandard["Accuracy"] With 92.5% this is quite accurate and also much faster than the competitor product. The confusion plot shows that it is a very reasonable accuracy - in particular when considering that native speakers have problems identifying the glyphs. cmstandard["ConfusionMatrixPlot"] ![enter image description here][17] Here are some glyphs that the Classifier identified as class "0": cmstandard["Examples" -> {0, 0}][[1 ;; 20]] ![enter image description here][18] They look quite diverse, but so does the original set: RandomSample[Select[trainingset, #[[2]] == 0 &], 20] ![enter image description here][19] It is quite impressive that the Classify function manages to achieve this level of accuracy. We can, however, do better! Machine Learning (NetTrain) ---------- We can improve the accuracy by training our own custom made network, which in this case will be the standard net that is often used for the MNIST dataset in Mathematica. It turns out that we do not even need a GPU for the training. lenet = NetChain[{ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2], ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2], FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]}, "Output" -> NetDecoder[{"Class", Range[0, 9]}], "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]] ![enter image description here][20] This we can then train: lenet = NetTrain[lenet, trainingset, ValidationSet -> testset, MaxTrainingRounds -> 20]; ![enter image description here][21] The training is done in about 4:30 minutes, but it appears that 20 training rounds are not necessary. Here are some results when applying the classifier: imgs = Keys@RandomSample[testset, 60]; Thread[imgs -> lenet[imgs]] ![enter image description here][22] Let's see how well it does: cm = ClassifierMeasurements[lenet, testset] cm["ConfusionMatrixPlot"] ![enter image description here][23] This corresponds to an accuracy of cm["Accuracy"] 95.6%, which is really impressive given the dataset. The following table illustrates the difficulty: Grid[Table[ Prepend[Select[trainingset, #[[2]] == k &][[1 ;; 10]][[All, 1]], ImageResize[calisymbols[[k + 1]], 28]], {k, 0, 9}], Frame -> All, Background -> {{Red}, None}] ![enter image description here][24] The first (red) column is derived from the glyph based on the character code - using our "calligraphy-style". The remaining columns show items from the training set, i.e. glyphs that were manually classified to belong to the given group. Conclusion ---------- I do not speak/read Japanese and have no more than Wikipedia knowledge about the Edo period. I still find it quite amazing that using the Wolfram Language's high level Classify function and one of the pre-implemented networks, we can achieve quite a remarkable classification. I am sure that the network can be improved. I would also like to see a more complete dataset with more than 10 different symbols. [at0]: http://community.wolfram.com/web/vitaliyk [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.24.03.png&userId=48754 [2]: http://community.wolfram.com/groups/-/m/t/852277 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.30.18.png&userId=48754 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.33.24.png&userId=48754 [5]: http://codh.rois.ac.jp/char-shape/ [6]: https://translate.google.com/translate?hl=en&sl=auto&tl=en&u=http://codh.rois.ac.jp/char-shape/ [7]: http://codh.rois.ac.jp/char-shape/book/ [8]: https://translate.google.com/translate?hl=en&sl=ja&tl=en&u=http://codh.rois.ac.jp/seminar/japanese-character-challenge-20170210/&sandbox=1 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.56.19.png&userId=48754 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.57.57.png&userId=48754 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-13at23.59.36.png&userId=48754 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.01.19.png&userId=48754 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.04.22.png&userId=48754 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.05.04.png&userId=48754 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.07.44.png&userId=48754 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.08.53.png&userId=48754 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.15.09.png&userId=48754 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.17.07.png&userId=48754 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.18.25.png&userId=48754 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.21.43.png&userId=48754 [21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.22.50.png&userId=48754 [22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.27.23.png&userId=48754 [23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.28.27.png&userId=48754 [24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-11-14at00.40.41.png&userId=48754 Marco Thiel 2017-11-14T00:49:30Z https://community.wolfram.com/groups/-/m/t/2864162 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/04e935d4-d928-4868-adb8-ae6f84fae1f8 Anton Antonov 2023-04-02T19:01:27Z https://community.wolfram.com/groups/-/m/t/1369571 I recently came across a video that intended to show how neural networks interpret images of (not so beautiful) things if they have only been trained on beautiful things. It is quite a nice question, I think. [Here is a website describing the technique][1], and [here is a video that illustrates the idea][2]. In this post I will show you, how to generate similar effects easily with the Wolfram Language: ![enter image description here][3] and in video format: ![enter image description here][4] On the right you see the "interpretation" of a neural network that has been shown lots of photos of flowers, when it actually looks at a rubbish dump with a couple of birds sitting on the rubbish. Devising a plan ---------- We will need a training dataset and should hope to find a network in the [Wolfram Neural Net Repository][5] that more or less does what we want to do. If you have watched some of the [excellent training videos on neural nets][6] offered by Wolfram you will have noticed that the general suggestion is not to develop your own neural networks from scratch, but rather use what is already there and perhaps combine it, or adapt it so that you can achieve what you want. This is also very well described in this recent [blog-post by experts on the topic][7]. I am usually happy if I can use the work of others and do not have to re-invent the wheel. If you read the posts describing how to build a network that have only seen beautiful things, you will find that they used a variation of the pix2pix network and an implementation in tensorflow (a "conditional adversarial network"). If you go through the extensive list of networks that are offered on the Wolfram Neural Net Repository you will see that there are Pix2pix resources, e.g. ResourceObject["Pix2pix Photo-To-Street-Map Translation"] or net=ResourceObject["Pix2pix Street-Map-To-Photo Translation"] I will use the latter resource object, but that does not actually matter. Next, we will need to build a training set. Scraping data for the training set ---------- The next thing we need is a solid training set. My first attempt was to use ServiceConnect with the google search to obtain lots of images of flowers. googleCS = ServiceConnect["GoogleCustomSearch"] imgs = ServiceExecute["GoogleCustomSearch", "Search", {"Query" -> "Flowers", MaxItems -> 1000, "SearchType" -> "Image"}]; It turns out that the max of results returned is only 100, which is not enough for our purpose. I tried to fix this by using imgs2 = ServiceExecute["GoogleCustomSearch", "Search", {"Query" -> "Flowers", MaxItems -> 1000, "StartIndex" -> 101, "SearchType" -> "Image"}]; but that did not work. So WebImageSearch is the way to go. It does cost ServiceCredits, but the costs are relatively limited. Let's download information on 1000 images of flowers: imgswebsearch = WebImageSearch["Flowers", MaxItems -> 1000]; Export["~/Desktop/imglinks.mx", imgswebsearch] A WebImageSearch of up to 10 results costs 3 ServiceCredits. So this should be 300 credits. 500 credits can be bought for $3, and 5000 for $25 (+VAT). This would mean the generation of our training set comes in at maximally $1.8, which is manageable - particularly if we consider the price of the eGPU that we will use later on.... Just in case we export the result, because we paid for it and might have to recover it later if we suffer a kernel crash or something. Alright. Now we have a dataset that looks more or less like this: ![enter image description here][8] Great. That contains the "ImageHyperlink" which we will now use to download all the images: rawimgs = Import /@ ("ImageHyperlink" /. Normal[imgswebsearch]); Export["~/Desktop/rawimgs.mx", rawimgs] Again, we export the result (better safe than sorry!). Let's make the images conform: imagesconform = ConformImages[Select[rawimgs, ImageQ]]; By using Select[...,ImageQ] we make sure that we use only images; and not error messages of the cases where it didn't work. Generating a training set ---------- In the original posts they suggest that they used edges, i.e. EdgeDetect to generate partial information of the images, and then linked that to the full image like so: rules = ImageAdjust[EdgeDetect[ImageAdjust[#]]] -> # & /@ imagesconform; It turns out that my results with that were less than impressive so I went for a more time consuming approach that gave better results. I used Monitor[rulesnew = Table[Colorize[ClusteringComponents[rules[[i, 2]], 7]] -> rules[[i, 2]], {i, 1, Length[rules]}];, I] i.e. ClusteringComponents to generate a trainingset. Partial information on the images now looked like this: ![enter image description here][9] rather than ![enter image description here][10] when we use EdgeDetect. Our training data set now links the image with partial (CluseteringComponents) information via a rule to the original image. Basically, we give partial information of the world and train the network to see flowers. Just in case we export the data set like so: Export["~/Desktop/rulesnew.mx", rulesnew] Training the network ---------- If you want to train on the EdgeDetect version you can use: retrainednet = NetTrain[net, rules, TargetDevice -> "GPU", TrainingProgressReporting -> "Panel", TimeGoal -> Quantity[120, "Minutes"]] otherwise you can use retrainednet2 = NetTrain[net, rulesnew, TargetDevice -> "GPU", TrainingProgressReporting -> "Panel", TimeGoal -> Quantity[120, "Minutes"]] Note that I use a GPU and considerable training time (2h). On a CPU this would take quite a while. Here are typical results of the EdgeDetect network: retrainednet[EdgeDetect[CurrentImage[], 0.7]] ![enter image description here][11] and the ClusteringComponents one: ![enter image description here][12] We should not forget to export the network: Export["~/Desktop/teachnwnicethings2.wlnet", retrainednet2] More examples ---------- Let's look at the ClusteringComponents network a bit closer. We apply GraphicsRow[{ImageResize[#, {256, 256}], beautifulnet[Colorize[ClusteringComponents[#, 7]]]}] & to different images to obtain: ![enter image description here][13] Application to videos ---------- Suppose that I have the frames of a recorded movie stored in the variable movie1. Then load our network into the function beautifulnet beautifulnet = Import["/Users/thiel/Desktop/teachnwnicethings2.wlnet"] Then the following will generate frames for an animation: animation1 = GraphicsRow[{ImageResize[#, {256, 256}], beautifulnet[Colorize[ClusteringComponents[#, 7]]]}] & /@ movie1; We can animate this like so: ListAnimate[animation1] ![enter image description here][14] Conclusion ---------- These are only very preliminary results. But we see the workflow from scraping data, via generating a training set, choosing a network and training it. I think that having more images and more training, perhaps a small change of the net might give us much better results. The video is quite bad, because we should use a better object than "four cables in a hand". It is also a bit debatable whether it is ok to say that this is how a network that only has seen beautiful things interprets the world, but I couldn't resit the hype. Sorry for that! This is certainly in the realm of "recreational use of the Wolfram Language", but the network does appear to make the world more colourful and provides a very special interpretation of the world. I hope that people in this forum who are better than me at this ([@Sebastian Bodenstein][at0] , [@Matteo Salvarezza][at1] , [@Meghan Rieu-Werden][at2] , [@Vitaliy Kaurov][at3] ?) can improve on the results. Cheers, Marco [at0]: http://community.wolfram.com/web/sebastianb [at1]: http://community.wolfram.com/web/matteosalvarezza [at2]: http://community.wolfram.com/web/meghanr [at3]: http://community.wolfram.com/web/vitaliyk [1]: http://www.memo.tv/learning-to-see-you-are-what-you-see/ [2]: https://vimeo.com/260612034 [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-07-0600.33.06.png&userId=48754 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1838animated.gif&userId=48754 [5]: https://resources.wolframcloud.com/NeuralNetRepository/ [6]: http://www.wolfram.com/broadcast/s?sx=neural%20networks&c=105&v=1522 [7]: http://blog.wolfram.com/2018/06/14/launching-the-wolfram-neural-net-repository/ [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-07-0523.04.32.png&userId=48754 [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-07-0523.13.54.png&userId=48754 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-07-0523.15.48.png&userId=48754 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-07-0523.21.05.png&userId=48754 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2018-07-0523.22.34.png&userId=48754 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=resulttable.png&userId=48754 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1838animated.gif&userId=48754 Marco Thiel 2018-07-05T23:14:51Z https://community.wolfram.com/groups/-/m/t/1851854 ![enter image description here][1] ![enter image description here][2] [The Wolfram Neural Net Repository][3] has published 95 NN models for various TaskTypes by 2019. This circular diagram visualizes the information contained in them. **view of circluar diagram** - An arc indicates one NN model. The arcs are ordered clockwise from the 12 o'clock position in the order of the year the model was released. - The outermost circumference is the TaskType. In addition, the name of NN models is displayed by Tooltip. - The inner circumference is the usage status of the Layer function of each NN model. White indicates that the Layer function is not used, and the darker the color, the more frequently the Layer function is used. In addition, the specific usage count is displayed by Tooltip. - The innermost circumference is the total number of parameters. The higher the height of the bar graph, the bigger the total number of parameters. In addition, the specific number is displayed by Tooltip. **TaskType** There are 10 TaskTypes. The legend of TaskTypes is at the top left of the figure. The trends are as follows. - Top 3 is Classification: 26, Feature Extraction: 18, Image Processing: 18, accounting for 65% of the total 95 models - Classification and Feature Extraction are released stably every year - Language Modeling, Audio Analysis, Object Detection are recent trends **LayerTypeCounts** In 95 NN models, 36 Layer functions are used. The trends are as follows. - The 4 most used Layer functions are BatchNormalizationLayer, ConvolutionLayer, ElementwiseLayer, LinearLayer. - "YOLO V2 Trained on MS-COCO Data" and "CapsNet Trained on MNIST Data" use the most types of Layer functions, and they use 14 types of Layer functions. - Some models use only one Layer function(EmbeddingLayer), and their TaskType are Feature Extraction(e.g. GloVe). - It is difficult to understand from the diagram, but those whose count exceeds 1000 are "Enhanced Super-Resolution GAN Trained on DIV2K, Flickr2K and OST Data" and "Unguided Volumetric Regression Net for 3D Face Reconstruction". The count of BERT that are producing impressive results in language processing is 862. On the other hand, the only NN model "Sketch-RNN Trained on QuickDraw Data" which TaskType is Data Generation makes very interesting inferences even though it uses only 3 Layer functions. **the total number of parameters** The total number of parameters is from the minimum 0.1 M of "Vanilla CNN for Facial Landmark Regression" to the maximum 659 M of "GloVe 300-Dimensional Word Vectors Trained on Common Crawl 840B". There are 23 NN models with a total number of parameters exceeding 100M, which is about 1/4 of all. Feature extraction: 8, classification: 7, semantic segmentation: 6, language modeling: 2. Get the data ============== I will explain how to make this circluar diagram. Create a list consisting of the following elements for all the NN models published in The Wolfram Neural Net Repository. - name - release year - TaskType - Total Number of Parameter - list of number of Layer functions used in it (* Get all the NN models published in the Wolfram Neural Net Repository?*) modelnames = NetModel[]; models = NetModel /@ modelnames; (* {name, release year, TaskType,?Total Number of Parameter} *) list = {#, NetModel[#, "SourceMetadata"]["Date"], NetModel[#, "TaskType"], NetInformation[NetModel[#], "ArraysTotalElementCount"]} & /@ modelnames; Correct missing data. (* correct release year from the description of The Wolfram Neural Net Repository *) list[[First@ First@Position[modelnames, "ResNet-101 Trained on YFCC100m Geotagged Data"], 2]] = DateObject[{2017}]; list[[First@First@Position[modelnames, "Yahoo Open NSFW Model V1"], 2]] = DateObject[{2016}]; (* correct TaskType from the description of The Wolfram Neural Net Repository *) list[[First@First@Position[modelnames, "LeNet"], 3]] = {"Classification"}; (* limited to one TaskType *) list[[First@ First@Position[modelnames, "GPT-2 Transformer Trained on WebText Data"], 3]] = {"Language Modeling"}; list[[First@ First@Position[modelnames, "GPT Transformer Trained on BookCorpus Data"], 3]] = {"Language Modeling"}; list[[First@ First@Position[modelnames, "Self-Normalizing Net for Numeric Data"], 3]] = {"Classification"}; list[[First@ First@Position[modelnames, "Wolfram AudioIdentify V1 Trained on AudioSet Data"], 3]] = {"Audio Analysis"}; Get a list of number of Layer functions used in NN models (* Get all the Layer functions used in NN models *) ltc = NetInformation[#, "LayerTypeCounts"] & /@ models; apis = Keys[ltc] // Flatten // DeleteDuplicates // Sort; (* List of number of Layer functions used in each NN model *) rule = Map[# :> 0 &, apis]; apiscnt = (apis /. # /. rule) &@ltc; (* {name of NN model, Year, TaskType,?Total Number of Parameter, list of number of Layer functions } *) list2 = Append[list[[#]], apiscnt[[#]]] & /@ Range[Length@models]; Sort the list by release year and TaskType. (*Sort by year*) listSortDate = Sort[list2, #1[[2]] < #2[[2]] &]; (* Sort by TaskType *) gbyear = GroupBy[listSortDate, #[[2]][[1, 1]] &]; sorttype = Sort /@ (#[[All, {3, 1, 2, 4, 5}]] & /@ gbyear); reorder = (#[[All, {2, 3, 1, 4, 5}]]) & /@ sorttype; listSortDateTask = Flatten[Values[reorder], 1]; Create circluar diagram ============== Set parameters independent of TaskType. (* Background color *) backcolor = White; (* Number of Layer functions *) an = Length@listSortDateTask[[1, -1]]; (* List of TaskTypes *) tasktypes = listSortDateTask[[All, 3]] // DeleteDuplicates // Sort; (* Number of TaskTypes *) tn = Length@tasktypes; (* inner circle radius of circular diagram *) R = 10; (* Display range of circular diagram *) c = 1.5 Pi; (* bar chart ratio of Total Number of Parameter *) psrate = R/(2 Max[ listSortDateTask[[All, 4]] /. Indeterminate -> 0]);(* Max Total Number of Parameter of all NN models *) psmax = Max[listSortDateTask[[All, 4]] /. Indeterminate -> 0]; (* colors of Layer functions *) SeedRandom[1]; col = RandomColor[an]; (* colors of TaskTypes from 12 colors of crayon *) colt = {RGBColor[1, 1, 0], RGBColor[1, 0.5, 0], RGBColor[0.6, 0.4, 0.2], RGBColor[1, 0, 0], RGBColor[1, 0.5, 0.5], RGBColor[0.5, 0, 0.5], RGBColor[Rational[157, 255], Rational[4, 5], Rational[224, 255]], RGBColor[Rational[181, 255], 1, Rational[4, 51]], RGBColor[0, 1, 0], RGBColor[0, 0, 1]}; (* get color of Layer functions. White if a function is not used *) Col[n_, cnt_] := If[cnt[[n]] == 0, backcolor, col[[n]]] (* get opacity of Layer functions depend on number of Layer function *) Opa[n_, cnt_] := Which[cnt[[n]] == 0, 1, cnt[[n]] < 10, .1, cnt[[n]] < 100, .5, True, 1] Define functions. (* cell for a layer function of a model *) cell[r_, {ang1_, ang2_}, col_, opa_, apiname_, apicount_] := { (* front *) Opacity[opa], col, If[col === backcolor, Disk[{0, 0}, r, {ang1, ang2}], Tooltip[Disk[{0, 0}, r, {ang1, ang2}], ToString[apiname] <> " : " <> ToString[apicount] ]], (* background *) EdgeForm[{Thickness[0], backcolor}], Opacity[1], backcolor, Disk[{0, 0}, r - 1, {ang1, ang2}] } (* arc for a model *) Arc[list_, modelno_, mn_] := Module[{theta, apicnt, coltt}, theta = {Pi/2 - (modelno - 1) c/mn, Pi/2 - modelno c/mn}; (* list of api count *) apicnt = list[[modelno, -1]]; (* color of TaskType *) coltt = colt[[First@First@Position[tasktypes, list[[modelno, 3]]]]]; Join[ (* TaskType *) {EdgeForm[{Thickness[0], backcolor}], coltt, Tooltip[Disk[{0, 0}, an + R + 1.5, theta], list[[modelno, 1]]]}, (* background *) {EdgeForm[{Thickness[0], backcolor}], backcolor, Disk[{0, 0}, an + R + .5, theta]}, (* cells of all Layer functions of NN model *) cell[an + R + 1 - #, theta, Col[#, apicnt], Opa[#, apicnt], apis[[#]], apicnt[[#]]] & /@ Range[an] ] ] (* Total Number of Parameter *) tnp[list_, modelno_, mn_] := Module[{theta, psize}, theta = {Pi/2 - (modelno - 1) c/mn, Pi/2 - modelno c/mn}; psize = list[[modelno, 4]];(* Total Number of Parameter *) If[psize === Indeterminate, psize = 0;]; {Gray, Thickness[.0025], Rotate[Tooltip[ Line[{{R/2 - 1, 0}, {R/2 - 1 + (R/2) Sqrt[psize/psmax], 0}}], psize], Mean@theta, {0, 0}]} ] (* circular diagram *) circulardiagram[tt_: 0] := Module[{listsel, yearnames, yearacc, yearmv, mn}, (* Parameters dependent of TaskType *) listsel = If[tt === 0, listSortDateTask, Select[listSortDateTask, #[[3]] === tt &]]; mn = Length@listsel ;(* Number of NN Models *) (* year information *) yearnames = (gb = GroupBy[listsel, #[[2]][[1, 1]] &]) // Keys; yearacc = Prepend[(Length /@ gb // Values // Accumulate), 0]; yearmv = MovingAverage[yearacc, 2]; Graphics[Join[ (* NN model *) Arc[listsel, #, mn] & /@ Range[mn], (* index: name & TaskType *) {Text[ Style["Model name & TaskType ", Gray, Bold, 6], {0, an + R + 1}, Right]}, (* index: Layer function name *) Text[Style[ToString[apis[[#]]] <> "", Gray, Bold, 6], {-1.5, an + R + 1 - .5 - #}, Right] & /@ Range[an], (* index: Layer function color *) {EdgeForm[{Thickness[0], col[[#]]}], FaceForm[col[[#]]], Disk[{-1, an + R + 1 - .5 - #}, .3]} & /@ Range[an], (* index: Total Number of Parameters *) {Text[ Style["Total Number of Parameters ", Gray, Bold, 6], {0, R/2 + 2}, Right]}, (* border line *) {LightGray, Thickness[.001], Rotate[Line[{{R, 0}, {an + R + 1, 0}}], Pi/2 - # c/mn, {0, 0}]} & /@ Range[mn],(* release year border line *) {LightGray, Thickness[.003], Rotate[Line[{{R, 0}, {an + R + 2, 0}}], Pi/2 - # c/mn, {0, 0}]} & /@ yearacc, {EdgeForm[{Thickness[.002], LightGray}], FaceForm[None], Disk[{0, 0}, R, {-Pi, Pi/2}]}, {EdgeForm[{Thickness[.001], LightGray}], FaceForm[None], Disk[{0, 0}, R + #, {-Pi, Pi/2}]} & /@ Range[an],(* inner circle edge *) {EdgeForm[{Thickness[.002], LightGray}], FaceForm[None], Disk[{0, 0}, R/2 - 1, {-Pi, Pi/2}]}, {EdgeForm[{Thickness[0], backcolor}], FaceForm[backcolor], Disk[{0, 0}, R/2 - 1 - .1]}, (* release year *) Text[Style[ToString[yearnames[[#]]], Gray, Bold, 11], (an + R + 3) {Cos[tmp = Pi/2 - yearmv[[#]] c/mn], Sin[tmp]}, Center, {Sin[tmp], -Cos[tmp]}] & /@ Range[Length@yearnames], (* Total Number of Parameter *) tnp[listsel, #, mn] & /@ Range[mn] ], (* usage guide of TaskTypes *) Epilog -> If[tt === 0, Inset[Grid[ Prepend[Transpose[{colt, Style[#, Bold, Gray, 8] & /@ tasktypes[[All, 1]]}], {Style[ "Task Type", Bold, Gray, 10], SpanFromLeft}], Alignment -> Left], {-an - R + 8, an + R - 9}], Inset[Null]], ImageSize -> If[tt === 0, 700, 350], Background -> backcolor, PlotLabel -> If[tt === 0, None, Style[First@tt, Gray, Bold, 13]] ] ] Create the circular diagram at the top of this post. circulardiagram[] ![enter image description here][4] Create circular diagram for each TaskType. Grid[Partition[circulardiagram /@ tasktypes, UpTo[3]], Spacings -> {2, 2}] ![enter image description here][5] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=cd202301_300.png&userId=11733 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=f01.jpg&userId=1013863 [3]: https://resources.wolframcloud.com/NeuralNetRepository/ [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=f01.jpg&userId=1013863 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=f02.jpg&userId=1013863 Kotaro Okazaki 2020-01-03T23:32:02Z https://community.wolfram.com/groups/-/m/t/2328597 > **GitHub Repository:** https://github.com/alecGraves/Howl &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/da6d3b76-7bbf-4a55-9fcd-058e7f1e17bf Alec Graves 2021-07-27T14:09:12Z https://community.wolfram.com/groups/-/m/t/562203 *WOLFRAM MATERIALS for the ARTICLE:* > William J Turkel, Digital Research Methods with Mathematica. > https://williamjturkel.net/digital-research-methods-with-mathematica/ > [Full book in notebook][1] > [GitHub Repository][2] This summer (2020) I revised the second edition of my open content, open access and open source textbook *Digital Research Methods with Mathematica*. It is freely available as a Mathematica notebook which can be accessed with either the Mathematica software or with Wolfram's free Wolfram Player. The content remains mostly the same as the previous version with a few minor edits. New in this edition are more than 100 short screencasts which I created to support online learning during the COVID-19 pandemic. https://williamjturkel.net/digital-research-methods-with-mathematica [![enter image description here][3]](https://williamjturkel.net/digital-research-methods-with-mathematica) The book focuses on learning to read code to the point where one can modify it to solve related research problems. Here are the topics that are covered. 1. Introduction to Mathematica. Interacting with notebooks. 2. Reading Code. Word frequency, word clouds and stopwords. 3. Computable Knowledge. Entities, tables, timelines and maps. 4. Text Content. Mathematica notebooks and expressions, strings and natural language processing. 5. Data Structures. Lists, associations and datasets. 6. Reusing Code. Defining and developing functions, keyword in context (KWIC). 7. Networks. Metadata, matrices and social network analysis. 8. Indexing and Searching. Pattern matching, topic classification and term distribution. 9. Geospatial Analysis. Geographic information: raster, vector and attribute data. 10. Images. Computer vision, face detection, feature extraction and image mining. 11. Page Images. Optical character recognition (OCR), figure extraction and classification. 12. Crawling. Browser automation, batch downloading, web archives and WARC files. 13. Linked Open Data. Resource description framework (RDF), SPARQL queries and endpoints, JSON-LD. 14. Markup Languages. Scraping and parsing, XML, really simple syndication (RSS) and text encoding initiative (TEI). 15. Studying Societies. Computational social science, search data, social media and social networks. 16. Extracting Keywords. Information retrieval, term frequency-inverse document frequency (TF-IDF) and rapid automatic keyword extraction (RAKE). 17. Word and Document Vectors. Feature extraction, dimension reduction, word embeddings and global vectors. 18. References, web services, bibliographic linked open data and citation networks. 19. Natural Language. Multilingual analysis, computational linguistics and sentiment analysis. 20. Web Services. Entity networks, publication search, dashboards, manipulating JSON. 21. Databases. Parts, selections and transformations, computations and querying, relations. 22. Measuring Images. Photogrammetry, georectification, handwriting and facial 3D reconstruction. 23. Machine Learning. Unsupervised clustering, classify, predict and transfer learning. ## SAMPLE PAGES: Lesson 21.3. Measuring Images: Handwriting &[Wolfram Notebook][4] [1]: https://github.com/williamjturkel/Digital-Research-Methods-2nd-Ed/releases/download/v2.01/turkel-digital-research-methods-with-mathematica-2nd-ed-rev-01-2020.nb [2]: https://github.com/williamjturkel/Digital-Research-Methods-2nd-Ed [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=digital-research-methods-2-rev-01-cover-2020.jpg&userId=90045 [4]: https://www.wolframcloud.com/obj/5d6eb35b-79bf-4d18-979f-9833d7f25878 William Turkel 2015-09-08T19:25:28Z https://community.wolfram.com/groups/-/m/t/1341081 For my research project I had to encounter a thorny problem. But before I tell about the problem I would like to briefly mention something about my research project. Basically I am using embryonic stem cells that self-organize to form spheroids (balls of cells) to study gastrulation events. In order to not bog down the readers with technical jargon, “gastrulation” is a process where the stem cells start to form the different layers; each layer then goes onto form the various tissues/organs, in the process unraveling the developmental plan of the entire organism. I am using experimental techniques and quantitative principles from biophysics and engineering to understand some aspects of this crucial process Now coming back to the problem at hand, the gastruloids (image below) are quite rough in their appearance and not as beautiful as one would like them to be (only a mother can love such an image). Any means of quantifying these gastruloids requires me to initially segment them. When you see a time-lapse images of gastruloids it becomes apparent that they shed a lot of cells (for reasons I do not know yet). This adds considerable noise to the system; oftentimes to the point that – as a human – my eyes are fooled and run into the difficulty of finding the right contours for the spheroids. Here comes the disclosure: classical means/operations in image-processing (gradients and edge detection, filtering, morphological operations etc.. ) prove utterly futile for image segmentation in my case. ![enter image description here][1] (A gastruloid – virtually a ball of cells with many shed around the periphery) So what can you do to address the problem where even the best image processing tool in existence – the human eyes – fails. This is precisely where you take help of neural networks. Neural networks are selling like hotcakes during the recent years and added life and hope to the once dead area of artificial intelligence. Again to avoid underlying technical details, neural networks is a paradigm utilized by the computer to mimic the working of a human brain by taking into account the complex interactions between the cells – but only digitally. There are many flavours of neural networks out there, each one geared towards performing a specific task. With advancements made in the area of deep learning/artificial intelligence, the neural nets have started to surpass humans in tasks that humans have been known to be best for i.e. classification tasks. A few recent examples that come to mind include Google’s AlphaGo beating the former World Go champion and an AI diagnosing skin cancer with an unprecedented accuracy. I utilized one such flavour of neural networks (a deep convolutional network – termed as UNET) to solve my longstanding problem. I constructed the network in Wolfram-Language with external help from Alexey Golyshev. UNET is a deep convolutional network that has a series of convolutional and pooling operations in the contraction phase of the net (wherein the features are extracted) and a sequence of deconvolution & convolution operations in the expansion phase which then yields an output from the network. This output can be subjected to a threshold to ultimately generate a binarized mask (the image segmentation). ![enter image description here][2] The architecture of UNET as provided by the author: https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/ (* ::Package:: *) BeginPackage["UNETSegmentation`"] (* ::Section:: *) (*Creating UNet*) conv[n_]:=NetChain[ { ConvolutionLayer[n,3,"PaddingSize"->{1,1}], Ramp, BatchNormalizationLayer[], ConvolutionLayer[n,3,"PaddingSize"->{1,1}], Ramp, BatchNormalizationLayer[] } ]; pool := PoolingLayer[{2,2},2]; dec[n_]:=NetGraph[ { "deconv" -> DeconvolutionLayer[n,{2,2},"Stride"->{2,2}], "cat" -> CatenateLayer[], "conv" -> conv[n] }, { NetPort["Input1"]->"cat", NetPort["Input2"]->"deconv"->"cat"->"conv" } ]; nodeGraphMXNET[net_,opt: ("MXNetNodeGraph"|"MXNetNodeGraphPlot")]:= net~NetInformation~opt; UNET := NetGraph[ <| "enc_1"-> conv[64], "enc_2"-> {pool,conv[128]}, "enc_3"-> {pool,conv[256]}, "enc_4"-> {pool,conv[512]}, "enc_5"-> {pool,conv[1024]}, "dec_1"-> dec[512], "dec_2"-> dec[256], "dec_3"-> dec[128], "dec_4"-> dec[64], "map"->{ConvolutionLayer[1,{1,1}],LogisticSigmoid} |>, { NetPort["Input"]->"enc_1"->"enc_2"->"enc_3"->"enc_4"->"enc_5", {"enc_4","enc_5"}->"dec_1", {"enc_3","dec_1"}->"dec_2", {"enc_2","dec_2"}->"dec_3", {"enc_1","dec_3"}->"dec_4", "dec_4"->"map"}, "Input"->NetEncoder[{"Image",{160,160},ColorSpace->"Grayscale"}] ] (* ::Section:: *) (*DataPrep*) dataPrep[dirImage_,dirMask_]:=Module[{X, masks,imgfilenames, maskfilenames,ordering, fNames,func}, func[dir_] := (SetDirectory[dir]; fNames = FileNames[]; ordering = Flatten@StringCases[fNames,x_~~p:DigitCharacter.. :> ToExpression@p]; Part[fNames,Ordering@ordering]); imgfilenames = func@dirImage; X = ImageResize[Import[dirImage<>"\\"<>#],{160,160}]&/@imgfilenames; maskfilenames = func@dirMask; masks = Import[dirMask<>"\\"<>#]&/@maskfilenames; {X, NetEncoder[{"Image",{160,160},ColorSpace->"Grayscale"}]/@masks} ] (* ::Section:: *) (*Training UNet*) trainNetwithValidation[net_,dataset_,labeldataset_,validationset_,labelvalidationset_, batchsize_: 8, maxtrainRounds_: 100]:=Module[{}, SetDirectory[NotebookDirectory[]]; NetTrain[net, dataset->labeldataset,All, ValidationSet -> Thread[validationset-> labelvalidationset], BatchSize->batchsize,MaxTrainingRounds->maxtrainRounds, TargetDevice->"GPU", TrainingProgressCheckpointing->{"Directory","results","Interval"->Quantity[5,"Rounds"]}] ]; trainNet[net_,dataset_,labeldataset_, batchsize_:8, maxtrainRounds_: 10]:=Module[{}, SetDirectory[NotebookDirectory[]]; NetTrain[net, dataset->labeldataset,All,BatchSize->batchsize,MaxTrainingRounds->maxtrainRounds, TargetDevice->"GPU", TrainingProgressCheckpointing->{"Directory","results","Interval"-> Quantity[5,"Rounds"]}] ]; (* ::Section:: *) (*Measure Accuracy*) measureModelAccuracy[net_,data_,groundTruth_]:= Module[{acc}, acc =Table[{i, 1.0 - HammingDistance[N@Round@Flatten@net[data[[i]],TargetDevice->"GPU"], Flatten@groundTruth[[i]]]/(160*160)},{i,Length@data} ]; {Mean@Part[acc,All,2],TableForm@acc} ]; (* ::Section:: *) (*Miscellaneous*) saveNeuralNet[net_]:= Module[{dir = NotebookDirectory[]}, Export[dir<>"unet.wlnet",net]]/; Head[net]=== NetGraph; saveInputs[data_,labels_,opt:("data"|"validation")]:=Module[{}, SetDirectory[NotebookDirectory[]]; Switch[opt,"data", Export["X.mx",data];Export["Y.mx",labels], "validation", Export["Xval.mx",data];Export["Yval.mx",labels] ] ] EndPackage[]; The above code can also be found in the repository @ [Wolfram-MXNET GITHUB][3] I trained my network over my laptop GPU (Nvidia GTX 1050) by feeding an augmented data (a set of 300 images constructed from a small dataset) . The training was done in under 3 minutes !. The accuracy (computed as the Hamming Distance between two vectors) of the generated binary masks with respect to the ground truth (unseen data) for a set of 90 images was 98.55 %. And with this a task that previously required me to painstakingly trace the contour of the gastruloids manually can now be performed in a matter of milliseconds. All the saved time and perspiration to be utilized somewhere else? ![enter image description here][4] Below is the results obtained by applying our trained net on one input: ![enter image description here][5] The interesting aspect for me regarding the network was that despite my gastruloids being highly dynamic (changing shape over time) I never had to explicity state it to the network. All the necessary features were learned from the limited number of images that I trained my network with. This is the beauty of the neural network. ![enter image description here][6] Finally the output of the net as applied on a number of unseen images: ![enter image description here][7] Note: I have a python MXNET version of UNET @ [python mxnet GITHUB][8] The wolfram version of UNET however seems to outperform the python version even though it also utilizes MXNET at the back-end for implementing neural networks. It should not come as a surprise because my guess is that the people at Wolfram Research may have done internal optimizations on top of the library [1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=gastruloid.png&userId=942204 [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=u-net-architecture-initial-authors-implementation.png&userId=942204 [3]: https://github.com/alihashmiii/UNet-Segmentation-Wolfram [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img1.png&userId=942204 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img2.png&userId=942204 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img3.png&userId=942204 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=img4.png&userId=942204 [8]: https://github.com/alihashmiii/blobsegmentation Ali Hashmi 2018-05-17T15:06:45Z https://community.wolfram.com/groups/-/m/t/1646303 # UNET [](https://zenodo.org/badge/latestdoi/137186334) [](https://github.com/dwyl/esta/issues) ---------- ![Automated 3D muscle segmentation using UNET / RESNET using DIXON MRI data][17] A package to generate and train a UNET deep convolutional network for 2D and 3D image segmentation. Some code was based on [work][1] by [@Ali Hashmi][at0], which was also dicussed in [this post][2] The full version of the toolbox can be found on my [github page][3]. * [Information](#information) * [Install toolbox](#install-toolbox) * [Using the toolbox](#using-the-toolbox) * [Functionality](#functionality) * [Visualization](#visualization) * [Example](#example) ## Information UNET is developed for [Mathematica](https://www.wolfram.com/mathematica/). It contains the following toolboxes: - UnetCore - UnetSupport Documentation of all functions and their options is fully integrated in the Mathematica documentation. The toolbox always works within the latest version of Mathematica and does not support any backward compatibility. All code and documentation is maintained and uploaded to github using [Workbench](https://www.wolfram.com/workbench/). ## Install toolbox Install the toolbox in the Mathematica UserBaseDirectory > Applications. FileNameJoin[{$UserBaseDirectory, "Applications"}] ## Using the toolbox The toolbox can be loaded by using <<UNET` The notbook ``UNET.nb`` shows examples of how to use the toolbox on artificially generated 2D data. There are also examples how to visualize the layer of your trained network and how to visualize the training itself. ## Functionality The network supports multi channel inputs and multi class segmentation. * UNET generates a UNET convolutional network. * 2D UNET ![UNET 2D][4] * 3D UNET ![UNET 3D][5] * Loss Layers: Training the data is done using three loss layers: a SoftDiceLossLayer, BrierLossLayer and a CrossEntropyLossLayer. ![SoftDiceLossLayer, BrierLossLayer and a CrossEntropyLossLayer][6] * Convolution Blocks: The toobox contains five different convolution blocks that build up the network: [UNET][7], UResNet, [RestNet][8], UDenseNet, [DensNet][9]. ![Convolution blocks][10] * SplitTrainData splits the data and labels into training, validation and test data. ![split train Data][11] * TrainUNET trains the network. ![Train UNET][12] ## Visualization * Visualize the network and results. * Visualize the features of the layers. ![Visualize layer features][13] * Visualize the results. ![Visualize the results][14] * Animate the training process. ![UNET 2D animation][15] ![UNET 3D animation][16] ## Example * Example: 3D segmentation of lower legg muscles using MRI data. ![Automated 3D muscle segmentation using UNET / RESNET using DIXON MRI data][17] [1]: https://github.com/alihashmiii/UNet-Segmentation-Wolfram [2]: https://community.wolfram.com/groups/-/m/t/1341081?p_p_auth=w8PIeeiA [3]: https://github.com/mfroeling [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=UNET2D.PNG&userId=1332602 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=UNET3D.PNG&userId=1332602 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Loss.PNG&userId=1332602 [7]: https://arxiv.org/abs/1505.04597 [8]: https://arxiv.org/abs/1512.03385 [9]: https://arxiv.org/abs/1608.06993 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=convblocks.PNG&userId=1332602 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Split.PNG&userId=1332602 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Train.PNG&userId=1332602 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Visualize1.PNG&userId=1332602 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Visualize2.PNG&userId=1332602 [15]: https://community.wolfram.com//c/portal/getImageAttachment?filename=amin0-v2.gif&userId=1332602 [16]: https://community.wolfram.com//c/portal/getImageAttachment?filename=amin4-v2.gif&userId=1332602 [17]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Muscle_Segmentation.jpg&userId=1332602 [at0]: https://community.wolfram.com/web/alihashmi87 Martijn Froeling 2019-04-03T20:01:26Z https://community.wolfram.com/groups/-/m/t/1843550 *MODERATOR NOTE: a submission to computations art contest, see more:* https://wolfr.am/CompArt-22 ---------- Showing Abs and Arg fields of Mandelbrot set under 200 iterations ![Mandelbrot_wallpaper.png][1] --- Showing Abs field of Mandelbrot set under 9 iterations ![9_iteration_abs.png][2] --- Showing Re vs. Im complex mapping of Mandelbrot set under 200 iterations ![ReIm_Complex_map.png][3] --- Mandelbrot Set on Neural Network =========================================== [The MXNet-based neural network framework](http://reference.wolfram.com/language/guide/NeuralNetworks.html) has been introduced to Wolfram Language since version 11. It is mostly designed for deep learning. However if we think about it, neural network is merely another fancy way to implement a certain program. As more and more functions are supported, it is now possible for us to "compile" general numerical program to neural network, which essentially gives us an interface to GPGPU parallel ability without the hassle of low-level coding. Inspired by [Brian](https://community.wolfram.com/web/tl2854)'s [recent post on MathCompile](https://community.wolfram.com/groups/-/m/t/1830592), we demonstrate in this post how to implement a neural network to compute the Mandelbrot set. ## Utilities As usual here is the code dump of some helper functions. Readers can evaluate then safely skip this section. ### code dump of helper functions ``` ClearAll[pipe, branch] pipe = RightComposition; branch = Through@*{##} &; ``` ``` Needs["NeuralNetworks`"] ``` ``` Clear[netInputPortSort] netInputPortSort[inPortLst_List] := Function[origNet, netInputPortSort[origNet, inPortLst]] netInputPortSort[origNet_NetGraph, inPortLst_List] := Block[{NetGraph = Inactive[NetGraph], fullnet, net}, fullnet = origNet ; net = fullnet[[1]] ; net["Inputs"] = AssociationThread[Rule[inPortLst, inPortLst /. net["Inputs"]]] ; ReplacePart[fullnet, 1 -> net] ] // Activate ``` --- # Basic concept The center of the Mandelbrot set computation is a simple iteration: $$\left\{ \begin{align} z_n &= c+z_{n-1}^2 \\ z_0 &= 0 \end{align} \right.$$ Under simple iteration it leads to the Mandelbrot set. ``` Block[{region, c, faclets} , (* random-points as different c for iteration: *) region = Disk[{-.5, 0}, 2] // DiscretizeGraphics // DiscretizeRegion[#, MaxCellMeasure -> .0001] & ; faclets = MeshCells[region, 2][[;; , 1]] ; c = MeshCoordinates[region].{1, I} ;(* The iteration taking place simultaneously for all c, nested 21 steps: *) Nest[Function[z, z^2 + c], 0, 21] // (* Visualizing the result: *) pipe[ pipe[(* Regularizing the norm for a clear visualization: *) Abs, HistogramTransform, Rescale, (1 - #)^2 &] , Append[ReIm[c]\[Transpose], #]\[Transpose] & , Graphics3D[ GraphicsComplex[#, {EdgeForm[], Polygon@faclets}] , Boxed -> False, SphericalRegion -> True, RotationAction -> "Fit" , ViewPoint -> {1, -2.4, 2.4}, ViewVertical -> {0, 0, 1} ] & ] ] ``` ![3D mesh of Mandelbrot set][4] Using the built-in [`MandelbrotSetPlot`](http://reference.wolfram.com/language/ref/MandelbrotSetPlot.html) function, it's straightforward to plot the traditional Mandelbrot set visualization: ``` MandelbrotSetPlot[{-2.5 - 2 I, 1.5 + 2 I}] ``` ![MandelbrotSetPlot][5] With some test we can notice that, for fixed `MaxIterations`, the relationship between computing time of `MandelbrotSetPlot` and `ImageResolution` follows a power law. ``` builtInTimeTest = {#, AbsoluteTiming[MandelbrotSetPlot[{-2.5 - 2 I, 1.5 + 2 I}, MaxIterations -> 100, ImageResolution -> #];][[1]]} & /@ Range[101, 4001, 100] NonlinearModelFit[builtInTimeTest, a r^k, {a, k}, r][r] // LogLogPlot[#, {r, 400, 5000}, PlotStyle -> Directive[Red, AbsoluteThickness[2]], Frame -> True, FrameTicks -> All] & // Show[{# , ListLogLogPlot[builtInTimeTest, PlotMarkers -> "OpenMarkers", PlotRange -> All] }, FrameLabel -> {"ImageResolution", "AbsoluteTiming"}] & ``` ![time benchmark of MandelbrotSetPlot][6] This is a scenario where parallel computation might make a difference. --- # Mandelbrot set on neural network -- A naïve attempt ## The implementation ### Iteration core net Neural network (short for **_NN_**) doesn't support complex number (yet). Aside from that, the iteration formula is simple enough for [`ThreadingLayer`](http://reference.wolfram.com/language/ref/ThreadingLayer.html): ``` iterNet = Module[{c = cX + cY I, z = x + y I}, z^2 + c // pipe[ ReIm, ComplexExpand, Echo[#, "{Re[\!\(\*SuperscriptBox[\(z\), \(2\)]\)+c], Im[\!\(\*SuperscriptBox[\(z\), \(2\)]\)+c]}:"] & , Map@pipe[ Inactive[Function][{cX, cY, x, y}, #] & , Activate , ThreadingLayer ] , Inactive[NetGraph][# , { (NetPort /@ StringSplit["cX,cY,x,y", ","]) -> # & /@ {1, 2} , 1 -> NetPort["x"], 2 -> NetPort["y"] } ] & , Activate ] ] ``` ![iterNet][7] It's very easy to perform the iteration with the help of [`Nest`](http://reference.wolfram.com/language/ref/Nest.html) / [`NestList`](http://reference.wolfram.com/language/ref/NestList.html). ``` Block[{region, c, xyInit, result} , (* random-points as different c for iteration: *) region = Disk[{0, 0}, .5] // DiscretizeGraphics // DiscretizeRegion[#, MaxCellMeasure -> .01] & ; c = MeshCoordinates[region].{1, I} // ReIm ; c = AssociationThread[StringSplit["cX,cY", ","], c\[Transpose]] ; xyInit = AssociationThread[StringSplit["x,y", ","], 0*Values[c]] ; result = NestList[ pipe[ Map[Clip[#, {-2, 2}] &] , Join[c, #] & , iterNet ] , xyInit, 20 ] ; result // pipe[ Values, Rest, Map@Transpose , Transpose , MapThread[{#2, BSplineCurve@#1} &, {#, # // Length // Range // N // Rescale // Map[ColorData["Rainbow"]]}] & , Graphics[#, Frame -> True, FrameTicks -> All, PlotRange -> {GoldenRatio {-1, 1}, {-1, 1}}, PlotRangeClipping -> True] & ] ] ``` ![Nest on iterNet][8] ### Nest net Now we have the core net representing the iteration formula, we need to mimic the functionality of `Nest[f, expr, n]` fully on a neural network. The go-to function is of course [`NetNestOperator`](https://reference.wolfram.com/language/ref/NetNestOperator.html). However in our case, the same $c$ is used repeatedly for all iterations, so we'd like to pass it directly to each iteration. Thus, we do it with the following customized `NetNestPartialOperator` function: ``` ClearAll[NetNestPartialOperator] NetNestPartialOperator[net_, staticPorts_List, nestPort_, iterNum_Integer] := Module[{pIdx, netIdx, staticpath, nestpath}, netIdx = Range[iterNum] ; pIdx = AssociationThread[staticPorts, staticPorts // Length // Range // Map[ToString]] ; staticpath = Function[p, NetPort["static" <> pIdx[p]] -> (NetPort[#, p] & /@ netIdx)] /@ staticPorts ; nestpath = Thread[Flatten[{NetPort["nest"], netIdx // Most}] -> (NetPort[#, nestPort] & /@ netIdx)] ; Inactive[NetGraph][ConstantArray[net, iterNum], {staticpath, nestpath} // Flatten] // Activate // netInputPortSort[Flatten[{"nest", "static" <> # & /@ Values[pIdx]}]] ] ``` Basically our `NetNestPartialOperator` takes a [`net_NetGraph`](https://reference.wolfram.com/language/ref/NetGraph.html) as the 1st argument. One of `net`'s input ports, i.e. `nestPort`, is going to be nested across iterations; other input ports (i.e. `staticPorts`) will be feed in constant/non-nested values. Here is a simple example realizing a 5 steps nest iteration `Nest[Function[c,a+b+c],c0,5]`: ``` Module[{coreNet, nestNet} , coreNet = NetGraph[{ThreadingLayer[#1+#2+#3&]}, {{NetPort["a"],NetPort["b"],NetPort["c"]}->1},"a"->"Real","b"->"Real","c"->"Real"] ; nestNet = NetNestPartialOperator[coreNet, {"a", "b"}, "c", 5] ; { {"coreNet", "nestNet"} , Quiet[ NetInformation[#, "MXNetNodeGraph"] // Graph[#, VertexSize -> 0.2, VertexLabels -> "Name"] & ] & /@ {coreNet, nestNet} }\[Transpose] // Grid[#, Background -> {{GrayLevel[.9], None}, None}, Frame -> All, FrameStyle -> Black] & ] ``` ![NetNestPartialOperator example][9] ### The Mandelbrot net In order to construct a net suitable for `NetNestPartialOperator`, we merge the real and imaginary ports in `iterNet` into a single "complex" port. ``` zNet = NetGraph[{ReplicateLayer[1], ReplicateLayer[1], CatenateLayer[]}, {NetPort["1"] -> 1, NetPort["2"] -> 2, {1, 2} -> 3}]; nestcoreNet = NetGraph[ {PartLayer[1], PartLayer[2], iterNet, zNet} , { (NetPort[#] -> NetPort[3, #]) & /@ StringSplit["cX,cY", ","] , NetPort["z"] -> {1, 2} , {{1, 2}, StringSplit["x,y", ","]} // MapThread[(#1 -> NetPort[3, #2]) &] , {StringSplit["x,y", ","], StringSplit["1,2", ","]} // MapThread[(NetPort[3, #1] -> NetPort[4, #2]) &] } ] ``` ![nestcoreNet][10] Thus a full net computing the Mandelbrot set is constructed as follows: ``` ClearAll[mandelbrot] mandelbrot[iterNum_Integer?Positive] := NetGraph[{ zNet , ElementwiseLayer[0 # &] , NetNestPartialOperator[nestcoreNet, StringSplit["cX,cY", ","], "z", iterNum] }, { {NetPort["cX"], NetPort["cY"]} -> 1 -> 2 , {2, NetPort["cX"], NetPort["cY"]} -> 3 } ] ``` ## Results We showcase our `mandelbrot` net with 9 iteration steps over the complex region ${-2-1.2I,1+1.2I}$, with a image resolution $401 \times 501$. The network is executed on a GPU (NVIDIA GeForce GTX 1050 Ti) with [`"Real64"`](https://reference.wolfram.com/language/ref/NetGraph.html#1010560149) precision. ``` mandelbrotF = mandelbrot[9]; ``` ``` result = Module[{region = {-2 - 1.2 I, 1 + 1.2 I}, resol = 501, aspr}, region = region // ReIm // Transpose ; aspr = region.{-1, 1} // Reverse // Apply[Divide] ; resol = resol {1, aspr} // Round ; {resol, region} // pipe[ MapThread[Rescale[Range[#1] // N // Rescale, {0, 1}, #2] &] , Tuples , AssociationThread[{"cX", "cY"}, #\[Transpose]] & , AbsoluteTiming[mandelbrotF[#, WorkingPrecision -> "Real64", TargetDevice -> "GPU"]] & , pipe[ branch[pipe[First, Quantity[#, "Seconds"] &, Echo[#, "Computing time on NN:"] &], Last] , Last] , Developer`ToPackedArray , ArrayReshape[#, {2, Sequence @@ resol}] & , {1, I}.# & , Transpose, Reverse ] ]; (* Computing time on NN: 3.08505 s *) ``` ``` result // Dimensions (* {401, 501} *) ``` ``` result // pipe[ branch[ pipe[ Abs , pipe[ branch[Flatten /* HistogramTransform, Dimensions /* Last] , Apply@Partition ] , Rescale, (1 - #)^5 & ] , pipe[Arg, Sin, Rescale] ] , Map @ pipe[ Image[#, ImageSize -> Length[#]] & , Colorize[#, ColorFunction -> "DarkColorFractalGradient"] & ] ] ``` ![naive result][11] Note the highlighted [periodic bulbs and Mandelbrot dendritic islands](https://dhushara.com/DarkHeart/DarkHeart.htm) in the left image, and the pattern approximately following the [external rays](https://en.wikipedia.org/wiki/External_ray) of the set in the right image. As a comparison, here is the result from the same region, iteration steps and resolution using the built-in `MandelbrotSetPlot` function. ``` AbsoluteTiming[ MandelbrotSetPlot[{-2 - 1.2 I, 1 + 1.2 I}, MaxIterations -> 9, ImageResolution -> 501] ] // pipe[ branch[pipe[First, Quantity[#, "Seconds"] &, Echo[#, "Computing time:"] &], Last], Last , #[[1, 1]] &, Reverse, Image[#, ImageSize -> Dimensions[#][[1]]] & ] (* Computing time: 0.187943 s *) ``` ![MandelbrotSetPlotOver(-2-1.2I,1+1.2I)][12] Clearly at this resolution scale, `MandelbrotSetPlot` is much faster than our neural-net function. But at a much larger resolution our function will eventually win. ``` myMandelbrotSetPlotTiming[resolution_Integer] := Module[ {region = {-2 - 1.2 I, 1 + 1.2 I}, aspr, resol} , region = region // ReIm // Transpose ; aspr = region.{-1, 1} // Reverse // Apply[Divide] ; resol = resolution {1, aspr} // Round ; {resol, region} // pipe[ MapThread[Rescale[Range[#1] // N // Rescale, {0, 1}, #2] &] , Tuples , AssociationThread[{"cX", "cY"}, #\[Transpose]] & , AbsoluteTiming[mandelbrotF[#, WorkingPrecision -> "Real64", TargetDevice -> "GPU"];][[1]] & , {resolution, #} & ] ] myTimeTest = myMandelbrotSetPlotTiming /@ Range[101, 4001, 100]; NonlinearModelFit[myTimeTest, a r^k, {a, k}, r][r] // LogLogPlot[#, {r, 400, 5000}, PlotStyle -> Directive[Purple, AbsoluteThickness[2]], Frame -> True, FrameTicks -> All] & // Show[ { # , ListLogLogPlot[myTimeTest, PlotMarkers -> Graphics[{FaceForm[White], EdgeForm[{Orange, AbsoluteThickness[1]}], Polygon[CirclePoints[4]]}, ImageSize -> 7], PlotRange -> All] } , FrameLabel -> {"ImageResolution", "AbsoluteTiming"} ] & ``` ![time benchmark of NN based mandelbrot][13] Comparing with previous benchmark result of `MandelbrotSetPlot`, we can see our NN-based function has an advantage when the image resolution is large enough. ![time bench comparison][14] --- # Mandelbrot set on neural network -- Avoid overflow Careful readers will find that for larger iteration steps and/or a larger computing region, our naïvely implemented `mandelbrot` will lead to overflow. That is of course due to the $c$ values leading to divergency. One simple way to avoid this overflow is to constrain the nested $z$ to a bounded region at every iteration step. ## The implementation ### A constrained iteration core net The simplest way to constrain the nested $z$ is to clip it into a rectangle region. ``` iterNet = Module[{c = cX + cY I, z = x + y I}, z^2 + c // pipe[ ReIm, ComplexExpand, Echo[#, "{Re[\!\(\*SuperscriptBox[\(z\), \(2\)]\)+c], Im[\!\(\*SuperscriptBox[\(z\), \(2\)]\)+c]}:"] & , Map@pipe[ (* The constraint: -> *) Block[{escR = 3}, escR Clip[#/escR]] & , Inactive[Function][{cX, cY, x, y}, #] & , Activate , ThreadingLayer ] , Inactive[NetGraph][# , { (NetPort /@ StringSplit["cX,cY,x,y", ","]) -> # & /@ {1, 2} , 1 -> NetPort["x"], 2 -> NetPort["y"] } ] & , Activate ] ] ``` ### The Mandelbrot net Re-evaluating the rest of the code leads to our simply constrained `mandelbrotCons` function. ``` nestcoreNet = NetGraph[ {PartLayer[1], PartLayer[2], iterNet, zNet} , { (NetPort[#] -> NetPort[3, #]) & /@ StringSplit["cX,cY", ","] , NetPort["z"] -> {1, 2} , {{1, 2}, StringSplit["x,y", ","]} // MapThread[(#1 -> NetPort[3, #2]) &] , {StringSplit["x,y", ","], StringSplit["1,2", ","]} // MapThread[(NetPort[3, #1] -> NetPort[4, #2]) &] } ] ``` ``` ClearAll[mandelbrotCons] mandelbrotCons[iterNum_Integer?Positive] := NetGraph[{ zNet , ElementwiseLayer[0 # &] , NetNestPartialOperator[nestcoreNet, StringSplit["cX,cY", ","], "z", iterNum] }, { {NetPort["cX"], NetPort["cY"]} -> 1 -> 2 , {2, NetPort["cX"], NetPort["cY"]} -> 3 } ] ``` ## Results Now we can go for much larger iteration steps on a larger region. ``` mandelbrotConsF = mandelbrotCons[200]; ``` ``` result = Module[{region = {-3 - 2 I, 1 + 2 I}, resol = 1001, aspr}, region = region // ReIm // Transpose ; aspr = region.{-1, 1} // Reverse // Apply[Divide] ; resol = resol {1, aspr} // Round ; {resol, region} // pipe[ MapThread[Rescale[Range[#1] // N // Rescale, {0, 1}, #2] &] , Tuples , AssociationThread[{"cX", "cY"}, #\[Transpose]] & , AbsoluteTiming[mandelbrotConsF[#, WorkingPrecision -> "Real64", TargetDevice -> "GPU"]] & , pipe[ branch[pipe[First, Quantity[#, "Seconds"] &, Echo[#, "Computing time on NN:"] &], Last] , Last] , Developer`ToPackedArray , ArrayReshape[#, {2, Sequence @@ resol}] & , {1, I}.# & , Transpose, Reverse ] ]; (* Computing time on NN: 6.29684 s *) ``` ``` result // Dimensions (* {1001, 1001} *) ``` ``` result // pipe[ branch[ pipe[ Abs , pipe[ branch[Flatten /* HistogramTransform, Dimensions /* Last] , Apply@Partition ] , Rescale, (1 - #)^5 & ] , pipe[Arg, Sin, Rescale] ] , Map @ pipe[ Image[#, ImageSize -> Length[#]] & , Colorize[#, ColorFunction -> "DarkColorFractalGradient"] & ] ] ``` ![constrained result][15] Or thresholding at 2 to get the classical Mandelbrot set. ``` result // Abs // UnitStep[2 - #] & // Image ``` ![classical Mandelbrot 200][16] ### Zoom-in For zoomed-in regions, this clipped version gives interesting results. ``` result = Module[{region = {-0.65 + 0.47 I, -0.4 + 0.72 I}, resol = 1001, aspr}, region = region // ReIm // Transpose ; aspr = region.{-1, 1} // Reverse // Apply[Divide] ; resol = resol {1, aspr} // Round ; {resol, region} // pipe[ MapThread[Rescale[Range[#1] // N // Rescale, {0, 1}, #2] &] , Tuples , AssociationThread[{"cX", "cY"}, #\[Transpose]] & , AbsoluteTiming[mandelbrotConsF[#, WorkingPrecision -> "Real64", TargetDevice -> "GPU"]] & , pipe[ branch[pipe[First, Quantity[#, "Seconds"] &, Echo[#, "Computing time on NN:"] &], Last] , Last] , Developer`ToPackedArray , ArrayReshape[#, {2, Sequence @@ resol}] & , {1, I}.# & , Transpose, Reverse ] ]; (* Computing time on NN: 6.06289 s *) ``` ``` result // pipe[ branch[ pipe[ Abs , pipe[branch[Flatten /* HistogramTransform, Dimensions /* Last], Apply@Partition] , Rescale, (1 - #)^5 &, Sin[\[Pi]/2 #] &, Rescale ] , pipe[Arg, Sin[#/2] &, Rescale] ] , Map@pipe[ Image[#, ImageSize -> Round[Length[#]/2]] & , Colorize[#, ColorFunction -> "StarryNightColors"] & ] ] ``` ![zoom-inAt(-0.65+0.47I,-0.4+0.72I)][17] ### Complex map We can also stylize the result any way we want, say, showing the complex mapping like the [`ComplexPlot`](https://reference.wolfram.com/language/ref/ComplexPlot.html). Abs vs. Arg: ``` result // pipe[ branch[ pipe[ Arg , branch[ pipe[Sin[#/2] &, Rescale] , pipe[ Sin[50 #] &, ArcSin, Rescale, #^.5 & ] ] ] , pipe[ Abs , Rescale, Log[10^-3 + #] &, Rescale , branch[ pipe[Sin[200 #] &, ArcSin, Rescale, #^.5 &] , pipe[#^5 &, Rescale[#, {0, 1}, {1, .3}] &] ] ] ] , Apply@Function[{arg2, abs2}, {arg2[[1]], arg2[[2]] abs2[[1]], abs2[[2]]}] , Image[#, ColorSpace -> "HSB", Interleaving -> False] & ] ``` ![Complex map: Abs vs Arg][18] Or Re vs. Im: ``` result // pipe[ branch[ pipe[ ReIm, Transpose[#, {2, 3, 1}] & , Map@pipe[ Cos[100 2 \[Pi] #] &, Rescale, #^.2 & ] ] , pipe[ Abs , Rescale, Log[10^-3 + #] &, Rescale , branch[ pipe[Sin[\[Pi]/2 #] &, Rescale] , pipe[#^5 &, Rescale[#, {0, 1}, {1, .5}] &] ] ] ] , Apply@Function[{reim, abs2}, {abs2[[1]], Times @@ reim, abs2[[2]]}] , Image[#, ColorSpace -> "HSB", Interleaving -> False] & ] ``` ![Complex map: Re vs Im][19] ### A different region (The detailed code in this section is omitted in the post, but included in the attached notebook.) Computing over a different region (`-0.0452407411 + 0.9868162204352258 I + 2.7 10^-5 {-1 - I, 1 + I}`) and fiddling with color palettes from `ColorData["ThemeGradients"]` gives us this wallpaper-like result (the [`ColorFunction`](https://reference.wolfram.com/language/ref/Colorize.html#777264821) used here is `"M10DefaultDensityGradient"`). ![zoom-in 2][20] Also the corresponding complex map plots: ![complex maps: Abs vs Arg & Re vs Im][21] --- # References - [Exploding the Dark Heart of Chaos](https://dhushara.com/DarkHeart/DarkHeart.htm) - [Wikipedia: External ray](https://en.wikipedia.org/wiki/External_ray) - [MathWorld: Mandelbrot Set](http://mathworld.wolfram.com/MandelbrotSet.html) - [Wikipedia: Mandelbrot set](https://en.wikipedia.org/wiki/Mandelbrot_set) - [The Mandelbrot Set Browser](http://www.cuug.ab.ca/dewara/mandelbrot/Mandelbrowser.html) [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Mandelbrot_wallpaper.png&userId=93201 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=9_iteration_abs.png&userId=93201 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ReIm_Complex_map.png&userId=93201 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=3D_mesh.png&userId=93201 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=MandelbrotSetPlot.png&userId=93201 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=time_bench_MandelbrotSetPlot.png&userId=93201 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=iterNet.png&userId=93201 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=iterNet_Nest.png&userId=93201 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=NetNestPartialOperator_example.png&userId=93201 [10]: https://community.wolfram.com//c/portal/getImageAttachment?filename=nestcoreNet.png&userId=93201 [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=naive_result.png&userId=93201 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=MandelbrotSetPlot2.png&userId=93201 [13]: https://community.wolfram.com//c/portal/getImageAttachment?filename=time_bench_NN.png&userId=93201 [14]: https://community.wolfram.com//c/portal/getImageAttachment?filename=time_bench_comparison.png&userId=93201 [15]: https://community.wolfram.com//c/portal/getImageAttachment?filename=constrained_result.png&userId=93201 [16]: https://community.wolfram.com//c/portal/getImageAttachment?filename=classical_Mandelbrot_200.png&userId=93201 [17]: https://community.wolfram.com//c/portal/getImageAttachment?filename=zoom_in_1.png&userId=93201 [18]: https://community.wolfram.com//c/portal/getImageAttachment?filename=AbsArg_Complex_map.png&userId=93201 [19]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ReIm_Complex_map.png&userId=93201 [20]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Mandelbrot_wallpaper.png&userId=93201 [21]: https://community.wolfram.com//c/portal/getImageAttachment?filename=complex_maps_2.png&userId=93201 Silvia Hao 2019-12-19T07:59:59Z https://community.wolfram.com/groups/-/m/t/3495066 ![Visual and statistical analysis of LLMs via problem complexity][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=HeroImage-WSS25.bmp&userId=3492577 [2]: https://www.wolframcloud.com/obj/2bc91532-706a-4dbe-ab99-4446339eba44 Rakesh Vijay Kumar 2025-07-08T21:23:18Z https://community.wolfram.com/groups/-/m/t/2931334 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=rtapis-5.png&userId=20103 [2]: https://www.wolframcloud.com/obj/730d5d38-e03f-49fc-8b91-6de321867da6 Michael Trott 2023-06-05T15:18:09Z https://community.wolfram.com/groups/-/m/t/2137315 ## Introduction In this article we will create a simple driving simulation to train a classifier to autonomously drive a virtual car. Our simulation will consist of a 3D scene composed of a windy road and a single car controller either by the user or the computer. The car will have a camera positioned in front of it to provide an image from its point of view (POV). This point of view image will be used as the input to our classifier to decide how to control the car. During each step of the simulation the car will be allowed to move forward or turn in place left or right. ![enter image description here][1] ## Building the Simulation ### Simulation Graphics We will begin by creating the graphics functions to assemble and display the simulation. First lets create a texture for the ground. For simplicity we will rasterize a built-in curve containing both left and right turns for the road. track = Rasterize[ With[{road = HilbertCurve[2, DataRange -> {{0.25, 1.75}, {0.25, 1.75}}] /. Line -> BSplineCurve}, Graphics[{ CapForm["Round"], Gray, AbsoluteThickness[36], road, Yellow, Dashing[0.033], AbsoluteThickness[2], road }, PlotRange -> {{0, 2}, {0, 2}}, Background -> RGBColor["#e8e1ba"] ] ] ] ![enter image description here][2] Next we will define a function to assemble the 3D scene encompassing the ground and car. We will be rendering this scene with two different cameras to provide an overview of the car on the road as well as the car's point of view image. CarSimScene[sim_] := Module[{ground, car}, ground = { Texture[track], EdgeForm[None], Polygon[ {{0, 0, 0}, {100, 0, 0}, {100, 100, 0}, {0, 100, 0}}, VertexTextureCoordinates -> {{0, 0}, {1, 0}, {1, 1}, {0, 1}} ] }; car = With[{dimensions = {2, 1, 0}}, {Red, Rotate[ Cuboid[sim["Position"] - dimensions , sim["Position"] + dimensions + {0, 0, 1}], sim["Rotation"], {0, 0, 1}, sim["Position"] ] } ]; {ground, car} ] Lastly we need a function to position a camera in the 3D scene at the location of the car to record the point of view image. This image will only be used when classifying, so we rasterize at a relatively low resolution. After some experimentation I chose to use ImageResolution instead of RasterSize because it was over 2x faster to generate the final image. The ViewVector coordinates were determined experimentally to match the typical view from inside a car. CarPOV[sim_] := Module[{ rotation = sim["Rotation"], pos = sim["Position"] }, Rasterize[ Graphics3D[ CarSimScene[sim], ImageSize -> Small, ViewVector -> { sim["Position"] + {2.1 Cos[sim["Rotation"]], 2.1 Sin[sim["Rotation"]], 1.5}, sim["Position"] + {5 Cos[sim["Rotation"]], 5 Sin[sim["Rotation"]], -0.25} }, ViewAngle -> 1.5, Boxed -> False, Background -> LightBlue, ViewRange -> {0, 0.1} ], ImageResolution -> 8 ] ] We can now put all of our graphics functions together into a single overarching function that displays the entire car simulation including the 3D scene and an inset from the car's point of view. DisplayCarSim[sim_] := Module[{}, Overlay[{ Graphics3D[ CarSimScene[sim], PlotRange -> {{0, 100}, {0, 100}, Automatic}, ImageSize -> Large, Background -> LightBlue, Boxed -> False ], Labeled[ Framed[sim["POV"], FrameMargins -> None, FrameStyle -> Gray, ImageSize -> Small], "Car POV", LabelStyle -> Gray] }] ] ### Simulation Internals With our simulation graphics complete we will proceed to building the functions that step the simulation forward. Our simulation state will be contained in a single association containing properties for the car's position, rotation, and current POV image, along with the history of all previous POV images and their associated actions. NewCarSim initializes our data model for a new simulation. We allow a parameter to define which direction on the track the car starts from. We will generate training data in the "Forward" direction and then test the trained self-driving car in the opposite direction. NewCarSim[direction_] := Module[{sim}, sim = If[ SymbolName[direction] == "Forward", <|"Position" -> {10, 10, 0}, "Rotation" -> 0|>, <|"Position" -> {90, 14, 0}, "Rotation" -> Pi|> ]; sim["History"] = {}; sim["POV"] = CarPOV[sim]; sim ] Advancing the simulation is easy, StepCarSim takes a simulation state and action, one of "Up", "Left", or "Right", and moves the car accordingly. Every time the simulation is stepped forward the current car POV image is saved along with the provided action in our history list. At the end of a simulation step a new car POV image is generated. StepCarSim[sim_, action_] := Module[{}, AppendTo[sim["History"], sim["POV"] -> action]; Switch[ action, "Left", sim["Rotation"] += 0.05, "Right", sim["Rotation"] -= 0.05, "Up", sim["Position"] += {Cos[sim["Rotation"]], Sin[sim["Rotation"]], 0} ]; sim["POV"] = CarPOV[sim]; sim ]; SetAttributes[StepCarSim, HoldFirst]; Our simulation is nearly complete, now it is time to build the interactive interface which allows a person to drive the car and generate training data. ## Generating Example Training Data sim = NewCarSim[Forward]; EventHandler[ Style[Dynamic[DisplayCarSim[sim]], Selectable -> False], { "LeftArrowKeyDown" :> StepCarSim[sim, "Left"], "RightArrowKeyDown" :> StepCarSim[sim, "Right"], "UpArrowKeyDown" :> StepCarSim[sim, "Up"] } ] ![enter image description here][3] When I manually drove the car down the length of the entire track, the simulation generated nearly 600 training examples. This data is available in the Supplemental Materials section below if you would prefer not to create your own. Lets take a look at a few of the training examples: TableForm@RandomSample[sim["History"], 3] ![enter image description here][4] ## Training the Self-Driving Car Having generated our example data, we are now ready to train a classifier to drive the car! selfDrivingCar = Classify[sim["History"], Method -> "LogisticRegression"] ![enter image description here][5] I chose to specify logistic regression because it provided the best accuracy for a relatively small evaluation time. We can show information about our classifier and see accuracy above 70%. By inspecting the learning curve and accuracy curve it looks like we are approaching a limit to our learning, but could probably still benefit from a few more training examples. Still, with our modest amount of example data, the autonomous car performs well. Information[selfDrivingCar] ![enter image description here][6] To preview our self-driving car we will step the simulation one hundred times using our new classifier to drive the car. This only gets us a little ways around the track, but shows the classifier is working. DynamicModule[{selfDrivingSim = NewCarSim[Backward]}, Print[Dynamic[DisplayCarSim[selfDrivingSim]]]; Do[StepCarSim[selfDrivingSim, selfDrivingCar[selfDrivingSim["POV"]]], 100]; ] ![enter image description here][7] We could increase the number of steps to see the car drive to the end, but due to the relatively slow rendering performance I prefer to pre-render every 5th or 10th frame and assemble those in a Manipulate to easily pan back and forth through time: selfDrivingSim = NewCarSim[Backward]; frames = Table[ With[{img = Rasterize[DisplayCarSim[selfDrivingSim]]}, Do[StepCarSim[selfDrivingSim, selfDrivingCar[selfDrivingSim["POV"]]], 5]; img ], 120 ]; Manipulate[frames[[t]], {t, 1, Length[frames] - 1, 1}] ![enter image description here][8] So, how well does our self-driving car work? Quite well as a matter of fact. It can easily make it down the entire track, and stays on the right side of the road too! But what happens if we start the car on the wrong side of the road? Does it get confused? Let's see: DynamicModule[{selfDrivingSim = NewCarSim[Backward]}, selfDrivingSim["Position"] -= {0, 4, 0}; Print[Dynamic[DisplayCarSim[selfDrivingSim]]]; Do[StepCarSim[selfDrivingSim, selfDrivingCar[selfDrivingSim["POV"]]], 100]; ] ![enter image description here][9] When starting on the wrong side of the road the self-driving car immediately corrects and steers back to the right side! Our car works wonderfully, especially considering the modest training set and limited manual optimization of Classify. ## Conclusion In conclusion we can use Mathematica to create and render a 3D simulation while also interfacing with its machine learning tools to build a complete end to end experimental system. I hope this example provides some insight into the power of Mathematica's machine learning and graphics capabilities. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ezgif-6-c4f303fe5f33.gif&userId=11733 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-12-13at2.19.25PM.png&userId=2137254 [3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-12-13at2.22.36PM.png&userId=2137254 [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=10103ScreenShot2020-12-13at2.23.23PM.png&userId=2137254 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=9171ScreenShot2020-12-13at2.24.29PM.png&userId=2137254 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=5376ScreenShot2020-12-13at2.25.22PM.png&userId=2137254 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-12-13at2.26.30PM.png&userId=2137254 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2020-12-13at2.27.13PM.png&userId=2137254 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=2628correcting-self-driving-car.gif&userId=2137254 Dan Bridges 2020-12-13T22:43:34Z https://community.wolfram.com/groups/-/m/t/2469295 *WOLFRAM MATERIALS for the BOOK:* > Etienne Bernard, *Introduction to Machine Learning*. > Wolfram Media: https://www.wolfram-media.com/products/introduction-to-machine-learning.html > Amazon: http://amzn.to/3oky6AC > ISBN: 978-1579550486 > Code-only notebooks: https://wolfr.am/iml [![enter image description here][1]][2] ## Introduction Hi everyone! For those who don't known me, I worked at Wolfram Research for 8 years (went pretty fast!) and led the development of the machine learning tools that are now part of the Wolfram Language (Classify, Predict, NetTrain, ...). It is now time for me to go on other adventures, but I couldn't leave "just like this", and decided to write a book called "Introduction to Machine Learning” http://amzn.to/3oky6AC to share my understanding of this field after these years of tool design and development. Writing this book has been quite of a journey, with some ups and downs, but it was overall pretty fun to play with the tools we built and figuring out the best ways to introduce machine learning. This book is intended for a general audience and has three goals: 1. Explain what machine learning is. 2. Teach how to practice machine learning. 3. Give an understanding of how machine learning works. It is written in a computational essay style (alternating text and simple computations - basically a long WL notebook), with lots of illustrations and examples. The code snippets are used to show how to practice machine learning, to illustrate concepts, and to complement - or even replace - mathematical formulations. It was quite nice to see how much math could be removed thanks to code (although, I often made the decision to explain concepts both with formulas and code snippets, to have the best of both worlds). Anyway, I encourage you to have a look at it, I included chapter 3 as a notebook in this post, I hope that you enjoy it or at least find it useful You can also check out the code-only notebooks https://wolfr.am/iml which contain both code snippets and illustrations code. **PS:** I am starting a weekly *#MLConcept* post series on Twitter [@etiennebcp][3] to highlight the most important topics and start conversations. You are welcome to join in! ## Sample chapter &[Wolfram Notebook][4] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=8296book_cover.jpg&userId=20103 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=8296book_cover.jpg&userId=20103 [3]: https://twitter.com/etiennebcp [4]: https://www.wolframcloud.com/obj/f0adf08b-7ebf-48a3-942e-e2473b561a97 Etienne Bernard 2022-02-11T19:33:27Z https://community.wolfram.com/groups/-/m/t/1093926 ![enter image description here][16] # Introduction Back in [Wolfram Summer School 2016][1] I worked on the project "Image Transformation with Neural Networks: Real-Time Style Transfer and Super-Resolution", which got later [published on Wolfram Community][2]. At the time I had to use the MXNetLink package, but now all the needed functionality is built-in, so here is a top-level implementation of artistic style transfer with Wolfram Language. This is a slightly simplified version of the original method, as it uses a single VGG layer to extract the style features, but a full implementation is of course possible with minor modifications to the code. You can also find this example in the docs: [NetTrain][3] >> Applications >> Computer Vision >> Style Transfer # Code Create a new image with the content of a given image and in the style of another given image. This implementation follows the method described in Gatys et al., *A Neural Algorithm of Artistic Style*. An example content and style image: ![enter image description here][4] To create the image which is a mix of both of these images, start by obtaining a pre-trained image classification network: vggNet = NetModel["VGG-16 Trained on ImageNet Competition Data"]; Take a subnet that will be used as a feature extractor for the style and content images: featureNet = Take[vggNet, {1, "relu4_1"}] ![enter image description here][5] There are three loss functions used. The first loss ensures that the "content" is similar in the synthesized image and the content image: contentLoss = NetGraph[{MeanSquaredLossLayer[]}, {1 -> NetPort["LossContent"]}] ![enter image description here][6] The second loss ensures that the "style" is similar in the synthesized image and the style image. Style similarity is defined as the mean-squared difference between the Gram matrices of the input and target: gramMatrix = NetGraph[{FlattenLayer[-1], TransposeLayer[1 -> 2], DotLayer[]}, {1 -> 3, 1 -> 2 -> 3}]; styleLoss = NetGraph[{gramMatrix, gramMatrix, MeanSquaredLossLayer[]}, {NetPort["Input"] -> 1, NetPort["Target"] -> 2, {1, 2} -> 3, 3 -> NetPort["LossStyle"]}] ![enter image description here][7] The third loss ensures that the magnitude of intensity changes across adjacent pixels in the synthesized image is small. This helps the synthesized image look more natural: l2Loss = NetGraph[{ThreadingLayer[(#1 - #2)^2 &], SummationLayer[]}, {{NetPort["Input"], NetPort["Target"]} -> 1 -> 2}]; tvLoss = NetGraph[<| "dx1" -> PaddingLayer[{{0, 0}, {1, 0}, {0, 0}}, "Padding" -> "Fixed" ], "dx2" -> PaddingLayer[{{0, 0}, {0, 1}, {0, 0}}, "Padding" -> "Fixed"], "dy1" -> PaddingLayer[{{0, 0}, {0, 0}, {1, 0}}, "Padding" -> "Fixed" ], "dy2" -> PaddingLayer[{{0, 0}, {0, 0}, {0, 1}}, "Padding" -> "Fixed"], "lossx" -> l2Loss, "lossy" -> l2Loss, "tot" -> TotalLayer[]|>, {{"dx1", "dx2"} -> "lossx", {"dy1", "dy2"} -> "lossy", {"lossx", "lossy"} -> "tot" -> NetPort["LossTV"]}] ![enter image description here][8] Define a function that creates the final training net for any content and style image. This function also creates a random initial image: createTransferNet[net_, content_Image, styleFeatSize_] := Module[{dims = Prepend[3]@Reverse@ImageDimensions[content]}, NetGraph[<| "Image" -> ConstantArrayLayer["Array" -> RandomReal[{-0.1, 0.1}, dims]], "imageFeat" -> NetReplacePart[net, "Input" -> dims], "content" -> contentLoss, "style" -> styleLoss, "tv" -> tvLoss|>, {"Image" -> "imageFeat", {"imageFeat", NetPort["ContentFeature"]} -> "content", {"imageFeat", NetPort["StyleFeature"]} -> "style", "Image" -> "tv"}, "StyleFeature" -> styleFeatSize ] ] Define a [NetDecoder][9] for visualizing the predicted image: meanIm = NetExtract[featureNet, "Input"][["MeanImage"]] > {0.48502, 0.457957, 0.407604} decoder = NetDecoder[{"Image", "MeanImage" -> meanIm}] ![enter image description here][10] The training data consists of features extracted from the content and style images. Define a feature extraction function: extractFeatures[img_] := NetReplacePart[featureNet, "Input" ->NetEncoder[{"Image", ImageDimensions[img], "MeanImage" ->meanIm}]][img]; Create a training set consisting of a single example of a content and style feature: trainingdata = <| "ContentFeature" -> {extractFeatures[contentImg]}, "StyleFeature" -> {extractFeatures[styleImg]} |> Create the training net whose input dimensions correspond to the content and style image dimensions: net = createTransferNet[featureNet, contentImg, Dimensions@First@trainingdata["StyleFeature"]]; When training, the three losses are weighted differently to set the relative importance of the content and style. These values might need to be changed with different content and style images. Create a loss specification that defines the final loss as a combination of the three losses: perPixel = 1/(3*Apply[Times, ImageDimensions[contentImg]]); lossSpec = {"LossContent" -> Scaled[6.*10^-5], "LossStyle" -> Scaled[0.5*10^-14], "LossTV" -> Scaled[20.*perPixel]}; Optimize the image using [NetTrain][11]. [LearningRateMultipliers][12] are used to freeze all parameters in the net except for the [ConstantArrayLayer][13]. The training is best done on a GPU, as it will take up to an hour to get good results with CPU training. The training can be stopped at any time via Evaluation -> Abort Evaluation: trainedNet = NetTrain[net, trainingdata, lossSpec, LearningRateMultipliers -> {"Image" -> 1, _ -> None}, TrainingProgressReporting -> Function[decoder[#Weights[{"Image", "Array"}]]], MaxTrainingRounds -> 300, BatchSize -> 1, Method -> {"ADAM", "InitialLearningRate" -> 0.05}, TargetDevice -> "GPU" ] ![enter image description here][14] Extract the final image from the [ConstantArrayLayer][15] of the trained net: decoder[NetExtract[trainedNet, {"Image", "Array"}]] ![enter image description here][16] [1]: https://education.wolfram.com/summer/school/alumni/2016/salvarezza/ [2]: http://community.wolfram.com/groups/-/m/t/885941 [3]: http://reference.wolfram.com/language/ref/NetTrain.html [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=I_432.png&userId=95400 [5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_179.png&userId=95400 [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_180.png&userId=95400 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_181.png&userId=95400 [8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_182.png&userId=95400 [9]: http://reference.wolfram.com/language/ref/NetDecoder.html [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_184.png&userId=95400 [11]: http://reference.wolfram.com/language/ref/NetTrain.html [12]: http://reference.wolfram.com/language/ref/LearningRateMultipliers.html [13]: http://reference.wolfram.com/language/ref/ConstantArrayLayer.html [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=O_185.png&userId=95400 [15]: http://reference.wolfram.com/language/ref/ConstantArrayLayer.html [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=I_466.png&userId=95400 Matteo Salvarezza 2017-05-15T10:33:59Z