Community RSS Feed
https://community.wolfram.com
RSS Feed for Wolfram Community showing any discussions in tag Data Science sorted by activeHow to calculate probability density using observations?
https://community.wolfram.com/groups/-/m/t/2334559
Hi All
I am currently doing Master in data science. I came across the function PDF probability density function which is used to find cumulative probability(range) of a continuous random variable.
The PDF probability density function is plotted against probability density in y axis and Random variable in x axis.
I am not able to understand how to convert an experiments observation of continuous random variable into probability density function.
Kindly help me understand with a small example.
Thank you.naveen mani2021-08-05T08:06:36Z[WSS21] Implementing the statistical problem solving process
https://community.wolfram.com/groups/-/m/t/2312324
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/dc7c2396-7b19-4634-a06c-79c78ddf1a6cKimberly Gardner2021-07-13T17:16:48Z[WSC19] Automated spot the difference
https://community.wolfram.com/groups/-/m/t/1732399
Description
-----------
I created a [website][1] that automatically makes a spot-the-difference challenge from various images. When given an image, I first find the specific items within the picture using ImageContents. I then remove or edit each of those contents and put the edited subimage back onto the original image, or in the case of removal, use InPaint to cover the gap.
When the user arrives at the site, they choose a difficulty level, whether they would like one image or all of the images in that difficulty range, and an image. They can click to the very right in order to reveal the correct solution.
General Image Functions
-----------------------
The heart of the functionality in my project is ImageContents, a function which uses machine learning to find and identify things in a given image. Though ImageContents uses the same idea as ImageIdentify, it can find items in subimages and not only the entire image. For example, lets take the ImageContents of this elephant image. It finds each elephant separately.
![enter image description here][2]
One of the first problems I ran into was that Inpaint will sometimes use parts of the original image which contain other objects: for example, Inpaint might successfully cover the large elephant, but catch the small elephant in its mask, covering the large elephant with an unsettling floating baby elephant head.
![enter image description here][3]
In order to solve this, I created my first function: `mask` which solves the issue of not being able to auto-generate masks based solely on their bounding boxes.
boundingboxes[image_]:=Quiet[Check[
List @@@ Flatten@
Values@Normal@ImageContents[image, All, "BoundingBox"], {}]]
The next function I created is called `expunge`, which utilizes `mask` to Inpaint the image contents.
mask[{{col1_, row1_}, {col2_, row2_}}, {nrows_, ncols_},
previous_: None] :=
Table[If[nrows - row2 < row <= nrows - row1 + 1 &&
col1 < col <= col2, 1,
If[previous === None, 0, previous[[row, col]]]], {row,
nrows}, {col, ncols}]
The next function I made is called ImageMapAt. ImageTrim trims the contents out of the image and applies the given function to the trimmed contents. ImageMapAt uses ImageCompose to shove the edited contents back into the original image, one at a time. Also, ImageMapAt uses the parameters {Left,Top} to align the edited contents correctly when pasting them back into the image, so it doesn't return outputs like this:
![enter image description here][4]
## Image Difference Functions ##
Once all my general functions were working, my next step was to create functions to be applied to the images to automate the differences. These functions were based on built-in functions in the Wolfram Language, but specified to visible yet not too obvious variations. These include `blurred`, which blurs an image by 2 units, `darker` which darkens dominant colors, `aquarecolor` which replaces a dominant color with part of the gradient Aquamarine, and various others.
## Displaying Image Differences ##
Once I had created differences, my next step was to display them. The idea is simple, merely subtract the original image from the one with differences. The difficult part was finding a way to display them nicely as the default had the differences placed on a black background making them hard to see and the ColorNegated version of it was too bright, so I used ImageAdjust. Here are some of the results.
![enter image description here][5]
## Image Distance Sorting ##
Now that I had created and displayed the differences, I sorted the images into the order of larger changes to smaller ones using ImageDistance, because the larger the distance is, the bigger the change. For example, compare these two distances. Wouldnt you agree the first one is roughly ten times easier than the second?
![enter image description here][6]
![enter image description here][7]
## Website Code ##
I then created LibraryAdd, a function that is used to add images into my curated image library. Lastly, I created my website. The basic idea is that next to the original (left) and the new image (right), there will be a blank/white image that I refer to as "blank." The user clicks on the "empty space" once they think they know the difference, and then the correct difference is shown. If it is clicked again, the image returns to its original blank state. The code is shown below.
CloudDeploy[
FormFunction[{"difficulty" -> {"easy" -> {0, 1/4},
"medium" -> {1/4, 1/2}, "hard" -> {1/2, 3/4},
"impossible" -> {3/4, 1}}, "selection" -> {"random", "all"},
"image" -> (CloudImport[First@# <> "/1.png"] -> First[#] & /@
Select[CloudObjects[
"gallery"], ! StringContainsQ[First@#, ".png"] &])},
With[{total = Length[CloudObjects[#image <> "/"]]/2,
original = #image <> "/1"},
Grid[With[{orderly =
Table[With[{new = #image <> "/" <> ToString[i]},
"<img height=\"200px\" src=\"" <> If[#2, blank, #] <> "\"" <>
If[#2,
" onclick=\"javascript:this.src=(this.src=='" <> # <>
"' ? '" <> blank <> "' : '" <> # <> "');\"", ""] <>
">" & @@@ {original <> ".png" -> False,
new <> ".png" -> False, new <> "a.png" -> True}],
{i, TakeQuantile[Range[2, total], #difficulty]}]},
If[#["selection"] === "random", {RandomChoice[orderly]},
orderly]]]] &,
AppearanceRules -> <|"Title" -> "Spot the Difference",
"Description" ->
"Choose a difficulty, whether you would like to see the \
solutions or not, and an image, then hit submit to try to spot the \
difference between the original image on the left and the edited one \
on the right. <i><a \
href=\"mailto:stella@maymin.com\">stella@maymin.com</a></i>"|>], \
"spot.me", Permissions -> "Public"]
## Future Work ##
In the future, as an extension of my website, I would like to find a way to create more of a game, perhaps by using dynamics. Another extension may be to create a timer counting in seconds how long it took the user to click the correct difference, or a counter of how many incorrect clicks on the Locator there were. Using those high scores, I think it might be possible to create a Leaderboard across all users. This would create a competition between users all over the world, so they can try to beat each others high scores in the race to the top. Lastly, a simpler extension is to create the option of allowing the user to upload their own images, which would be saved in my master library. This would let users play with images others have uploaded.
## Acknowledgements ##
I would like to thank my mentor, Rory Fougler, for helping me with my code every step of the way. It really made a huge difference. I would also like to thank Chip Hurst for providing various insights and helpful functions throughout my project. Lastly, I would like to thank Mads Bahrami, Kyle Keane, and Anna Musser for making this entire experience truly unforgettable. Overall, I am so grateful for my knowledge and capabilities now, thanks to my stay here at the Wolfram High School Summer Camp.
[1]: https://www.wolframcloud.com/obj/stella/spot.me
[2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ImageContentselephants.png&userId=1720407
[3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=OopsPost.png&userId=1720407
[4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=oopsididitagainpost.png&userId=1720407
[5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=orderlydiffs.png&userId=1720407
[6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ImageDistance01.PNG&userId=1720407
[7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ImageDistance03.PNG&userId=1720407Stella Maymin2019-07-12T01:28:57ZPerform a regression analysis with a moderating variable?
https://community.wolfram.com/groups/-/m/t/2334397
Hi all! I'm quite new to Mathematica (and coming from SPSS) and now I feel a bit lost. I would like to run a regression analysis, with one moderator variable. My variables are qualitative (1-7 likert scale). If this is helpful, I also attached a sample file. If you could provide some guidance, or some initial steps to take, I would really appreciate it. Thanks a lotMaya Novak2021-08-04T18:18:30ZSimulating a real-time ticker
https://community.wolfram.com/groups/-/m/t/2084630
Hello all,
It seems that my previous inquiry was removed by the moderators as they wanted to see me invest time and effort in my problem. I have invested time and effort in the more "algorithmic" parts of my project, but now I need a bit of help in controlling the charting of stock data.
Here is my situation. For the past 20 years, as an Electrical Engineer with an interest in the recognition of stock trading patterns, I have worked with Technical Analysis. I have worked for startups and written software that can recognize trendlines, technical patterns, and Elliott Waves. I have done all of my previous work in C# and absolutely love the topic. It gives me something to do besides my day job as a university professor in the Department of Computer Science.
I have been enamoured with Mathematica since the mid 2000's. However, for some reason, I have found it difficult to learn, compared to all the previous (traditional and not so traditional) languages I have mastered (Basic, Forth, C, C++, C#). One of my problems is that I was always impatient to get the work done and I could do that with something like C++ with my favorite, C#. It could take me a couple of 100's of lines, but the process was rather simple.
But with Mathematica, it is a different mental approach. What I love with Mathematica, is that I can take those 100's of C# lines and compress them into 10 lines (or less) of Mathematica. The problem is that, while I can write those 100's of C# lines in an hour or so, it takes me a week for me to get my head around all the functions, tables, lists, etc,,, I use in Mathematica.
The final result in Mathematica is like a work of art. It is beautiful, it speaks to my mind. The work in C# produces a beautiful application, but I don't find the program itself that fascinating. It is a bit like pointillism. The pixels (the programs or lines of code), in themselves are not that interesting, but the overall effect (the application) can be quite stunning.
In Mathematica, you don't have pixels, but swaths of paint of different colors (List, Map, BlockMap, etc...) and combinations of a few swaths (a single line of Mathematica can be simply mesmerizing). However, it can take hours to design that swath or understand someone else's swath.
I thus decided to delve into Mathematica and I wanted an interesting project to do so. I chose to rewrite my Technical Analysis Suite in Mathematica. My C# version, displays stock data, indicators, patterns (like channels, triangles, pennants, flags, etc...) and now can analyze all the potential Elliott wave patterns present in a stock move from one time (minute, day, week or month) to another.
I plan on taking a large number of stocks over a certain period and generate all of the possible Elliott wave patterns that were created for each stock during that period. Then I will feed to patterns (and their subsequent result) into an AI algorithm to create an Algorithm that can then be used to determine if a particular stock should be bought or sold.
I have never been convinced that a single individual (Ralph Nelson Elliott) could, before the advent of computers, figure out how markets work. But, I figure that his theory of Elliott Waves could be tested.
The problem, from what I understand, is that, if you put five Elliotticians (as they are called) into a room and show them a stock chart, you will get 5 different interpretations, all of them acceptable!
My view is to take away the human element, find any and all potential waves, then feed them into an AI machine and see if our powerful modern-day algorithms can figure out which patterns have strong predictive powers.
As such, I have put in a lot of work in converting my C# code to Mathematica. I can't tell you how amazed I am in the power of Mathematica, taking thousands of lines of C# code and reducing them to about a dozen lines of Mathematica code!
Given a stock over a period of time, my code will determine ALL the potential Elliott Wave patterns that occurred for that stock during that period of time. At some point that information could have been used to make trading decisions.
However, my interest is in deciding, once and for all for me, whether or not the different patterns mentioned in the literature (things like the bat, the butterfly, the crab, the shark, etc... from "The Harmonic Trader" do occur and do have predictive power. I have found that they do occur, and often. As for the predictive power, I see anecdotal evidence, but would like to take the decisive step of simply feeding mathematically provable patterns (meaning that the patterns I feed are ALL the possible wave patterns in the chart) into an AI program and generating an algorithm (e.g. Neural Network) that could then tell us what its predictive power is.
We could also see if the Support/Resistance prices generated by the Elliott Waves are also validated.
In order to do so, I wanted to have the ability to display the stock on a chart. I have used Mathematica[s InteractiveTradingChart which offers a great way to display stock data. My problem is that InteractiveTradingChart shows the chart in one short. What I want to do is to show the chart incrementally. What I mean by that is that I want to show the first bar, and then, after a short period of time, show the next bar, and so on. I might want to show each new bar after a delay of 5 seconds (or any other time period) in order to simulate a ticker. Also, my algorithm works in an incremental fashion. You feed it a new bar and it figures out what new information or new Elliott Wave may have emerged from the new bar. All I've been able to do so far is ask the chart to redraw itself, with the new bar, But I do this by reloading the FinancialData (which takes a while) after having updated the last day of interest. This is not satisfying. I would like to read all the data in one short (reading weekly bars for instance, from Jan 1, 2020 to the current day) and then feeding the bars one at a time to the chart once every 5 seconds.
I don't know how to programatically control the display of the chart. There is a little window on the bottom which can be slid left or right or stretched to show more or less of the stock. One approach I an thinking of is to start the window on the first day of the stock data I have and every 5 seconds, change the change the right edge of the window to the next day. In this way, it would look like the chart is updating itself every 5 seconds.
Can anyone help with my problems?Henrick Jeanty2020-09-27T18:04:52ZClassifying stocks by their price volatility
https://community.wolfram.com/groups/-/m/t/2333494
![Percent Change over Time of Tesla, Amazon, and Ford][1]
&[Wolfram Notebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=StockVolatility.png&userId=2317066
[2]: https://www.wolframcloud.com/obj/0435cfb6-cd1d-495e-8419-76323ffdd81eArshaan Sayed2021-08-03T21:23:02Z[WSG21] Daily Study Group: Building and Applying Epidemic Models
https://community.wolfram.com/groups/-/m/t/2251812
A new study group for Building and Applying Epidemic Models with the Wolfram Language begins Monday, May 10, 2021!
Making progress in an online course can be daunting when you have to study all alone. Join a cohort of fellow Wolfram Language users for a two-week study group in which you will start in week one with the basics of implementing compartment-based epidemiological models in the Wolfram Language. In the second week, you will cover multi-group models–taking into account vital demographics (i.e. birth, death) and age groups–in addition to introducing control measures and ending with stochastic models.
**Sign up here**: https://wolfr.am/UZfPoLAqJeremy Stratton-Smith2021-04-23T20:17:17ZTraining a recurrent neural network (RNN) to generate piano music
https://community.wolfram.com/groups/-/m/t/2328597
> **GitHub Repository:** https://github.com/alecGraves/Howl
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/da6d3b76-7bbf-4a55-9fcd-058e7f1e17bfAlec Graves2021-07-27T14:09:12ZBrain haemorrhage diagnosis: using LeNet based deep learning model
https://community.wolfram.com/groups/-/m/t/2273879
Introduction
------------
Brain haemorrhage is a type of stroke. It's caused by an artery in the brain bursting and causing confined bleeding in the enclosing tissues. This bleeding kills brain cells.
The Greek root for blood is hemo. Haemorrhage means "blood bursting forth." Brain haemorrhages are also called cerebral haemorrhages, intracranial haemorrhages, or intracerebral haemorrhages.
Cerebral haemorrhage deems for about 13% of all strokes in the United States. It is the next foremost cause of stroke. (The principal cause of stroke is a blood clot – thrombus – in an artery in the brain, which blocks blood flow and cuts off required oxygen and nutrients to the brain.)
Importing Dataset
------------
I implied a data set of haemorrhage and non-haemorrhage brain from Kaggle. Each class placed in its corresponding variable.
infected=FileNames["*.png","E:\\COURSES\\Wolfram\\BrainTumorImagesDataset\\training_set\\hemmorhage_data"];
uninfected=FileNames["*.png","E:\\COURSES\\Wolfram\\BrainTumorImagesDataset\\training_set\\non_hemmorhage_data"];
Constructing File Objects for Images
------------
I wanted to match each brain image with a value of either true for Hemorrhage or false for no Hemorrhage. To improve the efficiency of the importation, I formulated separate file objects for each of the image variables. Each variable contained 70 images for parasitized and uninfected cells.
infectedIMG = File /@ infected;
uninfectedIMG = File /@ uninfected;
Then I created a record of 70 true and false values which would be used to be matched up with their respective images. I set these lists in variables and made another variable to connect the list of true and false values along with another variable that connected the infected and uninfected file objects.
Length[infectedIMG]
70
infectedvalues=Table[True,Length[infected]];Length[uninfected]
70
uninfectedvalues=Table[False,Length[uninfected]];
Finally, using the AssociationThread function, I associated the images with their values and divided the data into two groups, 75% for training and 25% for validation.
data=RandomSample[AssociationThread[infectedIMG->infectedvalues]];
traininglength=Length[data]*.75
52.5
trainingdata=data[[1;;52]];validationdata=data[[53;;]];
Creating the Neural Network
------------
I then started to work on the Neural Network, which used MNIST image classification. The network's goal is to classify uninfected and infected using true and false to describe whether the patient suffers Brain Haemorrhage or not. I built a NetChain function that had multiple layers. One striking layer is the Resize layer which changes the image dimensions of each image to 135 by 135. This changes the images to comply with the sensitivity of the neural network to the size of images. Further layers include the convolution layer, ramp, and pooling layer, which all work to narrow down pieces and create categories to classify each image to associate them.
dims={135,135}
{135,135}
lenet=NetChain[{ResizeLayer[dims],ConvolutionLayer[20,5],Ramp,(*Takesoutthethenotusefulfeatures*)PoolingLayer[2,2],(*Downsamples*)ConvolutionLayer[50,5],Ramp,(*Takesoutthethenotusefulfeatures*)PoolingLayer[2,2],(*Downsamples*)FlattenLayer[],500,(*Makesfeaturesintofeaturevector"*)Ramp,2,(*Takesoutthethenotusefulfeatures-Trueorfalse*)SoftmaxLayer[]},(*Turnsthevectorintoprobabilities*)"Output"NetDecoder[{"Class",{True,False}}],(*Tensorintotrueorfalse*)"Input"NetEncoder["Image"](*Turnsimageintonumbers*)]
Training the Neural Networks with NetTrain
------------
I trained the neural nets with 10 training rounds.
results=NetTrain[lenet,Normal[trainingdata],All,ValidationSet->Normal[validationdata],MaxTrainingRounds->10,TargetDevice->"CPU"]
![NetTrain Result][1]
Training the Neural Network with Augmented Layers
------------
Next I implemented an ImageAugmentationLayer, which randomly crops images to create new data sets to improve my neural network.
augment = ImageAugmentationLayer[{135, 135}, "Input" -> NetEncoder[{"Image", {139, 139}}], "Output" -> NetDecoder["Image"]]
I made the images 139 by 139 and allowed the augmentation layer to crop the images by 4 pixels at random within the constraints of the dimensions of 135 by 135.
dims2 = {139, 139}
lenet2 = NetChain[{ResizeLayer[dims2],
ImageAugmentationLayer[{135, 135}], ConvolutionLayer[20, 5], Ramp,
PoolingLayer[2, 2], ConvolutionLayer[50, 5], Ramp,
PoolingLayer[2, 2], FlattenLayer[], 500, Ramp, 2, SoftmaxLayer[]},
"Output" -> NetDecoder[{"Class", {True, False}}],
"Input" -> NetEncoder["Image"]]
I trained this data using the neural net, with only 7 layers on CPU.
results2 =
NetTrain[lenet2, Normal[trainingdata], All,
ValidationSet -> Normal[validationdata], MaxTrainingRounds -> 7]
![enter image description here][2]
Creating a Testing Set for Data
------------
![enter image description here][3]
Data Visualization
------------
Lastly, I made a ConfusionMatrixPlot using the Classifier Measurements function which compares the neural networks predicted class against the actual class result.
![enter image description here][4]
Conclusion
------------
I built a neural network that strongly diagnosed Brain Haemorrhage with an accuracy of about 99.000000078%. Furthermore, as displayed in the ConfusionMatrix, there were 18 examples of the neural network prediction matching with the actual results for true and 18 examples of the neural network and actual matching for true.
Future Improvements
------------
To additionally enhance this project, I could implement more augmented datasets to further train and enhance the neural net. Moreover, I could use various images from different datasets to prevent overfitting and improve efficiency. Lastly, I could execute a function that pinpoints Brain Haemorrhage by finding the edges of the Haemorrhage area and sensing the infected cells through the function, image distribution and colour detection.
&[Wolfram Notebook][5]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=1.png&userId=2253909
[2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=2.png&userId=2253909
[3]: https://community.wolfram.com//c/portal/getImageAttachment?filename=3.png&userId=2253909
[4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=4.png&userId=2253909
[5]: https://www.wolframcloud.com/obj/f4d8f183-800b-4e6c-993f-6ee5d572e29aAman Dewangan2021-05-22T17:23:02ZEpidemic simulation with a polygon container
https://community.wolfram.com/groups/-/m/t/1901002
*MODERATOR NOTE: coronavirus resources & updates:* https://wolfr.am/coronavirus
----------
![enter image description here][1]
&[WolframNotebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=3687ezgif.com-optimize.gif&userId=11733
[2]: https://www.wolframcloud.com/obj/c443430b-2f0e-461a-ad95-801016802147Francisco Rodríguez2020-03-18T01:05:06Z