Community RSS Feed
https://community.wolfram.com
RSS Feed for Wolfram Community showing any discussions in tag Staff Picks sorted by activeSqueezing Pi from a Menger Sponge
https://community.wolfram.com/groups/-/m/t/822984
In the [Menger sponge](http://mathworld.wolfram.com/MengerSponge.html), a step divides a cube into 27 cubes, then the center and six touching cubes are removed. Here's what it looks like after three steps.
Graphics3D[
Cuboid /@ ((FromDigits[#, 3] & /@ Transpose[#]) & /@
Select[Partition[#, 3] & /@ Tuples[Range[0, 2], {9}],
Max[Count[#, 1] & /@ #] < 2 &]), Boxed -> False]
![enter image description here][1]
You can also take slices, as shown in the Wolfram Demonstration [Menger Sponge Slices](http://demonstrations.wolfram.com/MengerSpongeSlices/).
![enter image description here][2]
A string rewriting system gives a related fractal known as the [Sierpiński Carpet](http://mathworld.wolfram.com/SierpinskiCarpet.html). It is a face of the Menger sponge. Here are two different pieces of code to generate it.
Row[{ArrayPlot[
Table[If[
Not[MemberQ[
Transpose[(IntegerDigits[#, 3, 5] & /@ {a, b})], {1, 1}]], 1,
0], {a, 0, 3^5 - 1}, {b, 0, 3^5 - 1}], PixelConstrained -> True,
ImageSize -> 250], Spacer[20],
ArrayPlot[
Nest[ArrayFlatten[# /. {0 -> {{0, 0, 0}, {0, 0, 0}, {0, 0, 0}},
1 -> {{1, 1, 1}, {1, 0, 1}, {1, 1, 1}}}, 2] &, {{1}}, 5],
Frame -> False, PixelConstrained -> True, ImageSize -> 250]}]
![enter image description here][3]
Taking a horizontal line through the center gives the [Cantor set](http://mathworld.wolfram.com/CantorSet.html) (also known as the middle-thirds set), an infinite closed perfect set of zero measure.
![enter image description here][4]
You might recognize the ternary (base 3) numbers in each set of code. For the 1D Cantor set, remove any ternary number with a 1. For the 2D Sierpinski carpet, if the *n*th ternary digits are both 1, remove that pair of numbers. For the 3D Menger sponge, if two or three of *n*th ternary digits are1, remove that triplet of numbers. All of these objects tend to a set of measure zero, whether the measure is length, area, or volume.
So those are ways to get measure 0 with a fractal. How about making Pi with a fractal? These two infinite products look promising for 1D.
![enter image description here][5]
Transforming the fractions into fractals requires adding a segment when the fraction is greater than 1. Here's what the fractals look like after a few steps, followed by a check to make sure the segment lengths are adding up correctly.
![enter image description here][6]
I shall call the first of those the happy Pi fractal. To be more bold about it, here are those formulas again.
![enter image description here][7]
Plotting the values given by these fractals gives something that looks like the [Minkowski ? function](http://mathworld.wolfram.com/MinkowskisQuestionMarkFunction.html), which is based on the [Farey sequence](http://mathworld.wolfram.com/FareySequence.html) of irreducible fractions. Build circles of radii twice the denominator squared above each fraction to obtain the [Ford circles](http://demonstrations.wolfram.com/FordCircles/), a fractal of tangent circles.
![enter image description here][8]
An irreducible fraction a/b in the Farey sequence has gcd(a,b)=1 and a less than b. It happens that the probability of random integers being relatively prime with *a* less than *b* is 3/Pi^2. We can use that to approximate the length of F1422, all the irreducible fractions with denominator of 1422 or less. Out of a 1422*1422 grid of pixels, about 3/Pi^2percent of them should be black.
![enter image description here][9]
By squaring 1422 and multiplying we should be able to get a reasonable approximation for the size of F1422. Then we can check to find the actual size.
![enter image description here][10]
Fairly accurate, but enough with 1D. For a 2D fractal for Pi, we can use the Wallis product again.
![enter image description here][11]
The corresponding fractal is the [Wallis sieve](http://demonstrations.wolfram.com/WallisSievePiApproximation/).
Start with a 2*2 square and divide it into 4 squares.
Divide each new subsquare by 3, and remove the middle square (8/9).
Divide each new subsquare by 5, and remove the middle square (24/25).
Divide each new subsquare by 7, and remove the middle square (48/49).
Divide each new subsquare by 9, and remove the middle square (80/81).
And so on. Here's what it looks like after a few steps.
![enter image description here][12]
As mentioned earlier, this fractal has an area equal to Pi.
![enter image description here][13]
Is there a Menger sponge equivalent of the Wallis sieve? At each step, we'd divide the remaining cubes into the next largest odd cube, then take away the cubes extending out from the center.
![enter image description here][14]
That's the same volume as a sphere of radius 1. What would one of these fractals look like? The code is similar: getting the digits 1, 2, or 3 in bases 3, 5, or 7 more than once causes a triplet to get tossed out.
wallissponge = Select[Tuples[Select[Tuples[Range[0, 6], {3}], Sign[{3, 5, 7} - #] == {1, 1, 1} &], {3}],
Max[MapIndexed[Count[#1 - #2[[1]], 0] &, Transpose[#]]] < 2 &];
Graphics3D[{EdgeForm[None], Cuboid /@ ((5 7 #[[1]] + 7 #[[2]] + #[[3]] & /@ # ) & /@ wallissponge)}, Boxed -> False, ImageSize -> 600]
![enter image description here][15]
Use eight of these to get a fractal object with the same volume as the unit sphere.
We've taken some fractals of measure zero and tweaked them a bit to get Pi. If anyone can break this fractal into pieces and make a sphere, please let me know.
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng1.gif&userId=21530
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng2.gif&userId=21530
[3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng3.gif&userId=21530
[4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng4.gif&userId=21530
[5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng5.gif&userId=21530
[6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng6.gif&userId=21530
[7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng7.gif&userId=21530
[8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng8.gif&userId=21530
[9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng9.gif&userId=21530
[10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng15.gif&userId=21530
[11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng10.gif&userId=21530
[12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng11.gif&userId=21530
[13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng12.gif&userId=21530
[14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng13.gif&userId=21530
[15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=meng14.gif&userId=21530Ed Pegg2016-03-14T14:42:30ZThe Most Convenient Keyboard Shortcuts Ever
https://community.wolfram.com/groups/-/m/t/1855244
Here are two simple but extremely convenient keyboard shortcuts that I've used for years.
- This post is a favor to all my friends who type slowly (like me)
- These shortcuts simply create a pair of brackets containing a template placeholder as the current new selection
- Essentially, it's just a stripped-down version of the bulky (Command + Shift + K + Up/Down Arrow + Enter) completion
**Favorite Shortcut #1:**
Autocomplete single brackets by pressing `CommandKey + [`
![enter image description here][1]
**Favorite Shortcut #2:** Autocomplete part brackets by pressing `CommandKey + ]`
![enter image description here][2]
## How to set them up ##
On MacOS, first make a backup of the following file, in case you break it:
FileNameJoin[{$InstallationDirectory, "SystemFiles/FrontEnd/TextResources/Macintosh/KeyEventTranslations.tr"}];
On my machine this path resolves to: *"/Applications/Mathematica.app/Contents/SystemFiles/FrontEnd/TextResources/Macintosh/KeyEventTranslations.tr"*
Then, preferably in a non-Mathematica editor (e.g. Sublime), open the above file and copy-and-paste these following code into the list inside `EventTranslations`. For the first shortcut:
Item[KeyEvent["[",Modifiers->{Command}],FrontEndExecute[{FrontEnd`NotebookWrite[FrontEnd`InputNotebook[],"[\[SelectionPlaceholder]]"],FrontEndToken["MovePreviousPlaceHolder"]}]]
And for the second (of course making sure they are separated by a comma):
Item[KeyEvent["]",Modifiers->{Command}],FrontEndExecute[{FrontEnd`NotebookWrite[FrontEnd`InputNotebook[],"\[LeftDoubleBracket]\[SelectionPlaceholder]\[RightDoubleBracket]"],FrontEndToken["MovePreviousPlaceHolder"],FrontEndToken["MovePreviousPlaceHolder"]}]]
Finally, restart Mathematica, and presto, they should both be working. And to remove the shortcuts, simply remove those items or replace the .tr file with the backup.
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=own.gif&userId=900170
[2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=part.gif&userId=900170Mike Sollami2020-01-08T20:42:08Z[GIF] Rotation Redux (Inverse Cayley transform of rotating grid of circles)
https://community.wolfram.com/groups/-/m/t/1857532
![Inverse Cayley transform of rotating grid of circles][1]
**Rotation Redux**
A [recent post by Dave Whyte (a.k.a. @beesandbombs)][2] on Twitter reminded me of my old GIF [_Rotation_][3], which is almost 5 years old (and dates back to before I was posting on the Wolfram Community). I figured it would be fun to do a new twist on the same basic animation.
The basic idea is fairly simple: start with a rotating grid of circles:
![Rotating grid of circles][4]
Apply the map $z \mapsto 1/z$ to the circles to invert them inside the unit circle:
![Rotating grid of circles, inverted inside unit circle][5]
(Notice that the inside circles are rotating the other way; not surprising since $1/z$ is proportional to $\bar{z}$.)
Now just apply the inverse of the Cayley transform which maps the upper half-plane to the interior of the unit circle (and the lower half-plane to the exterior) and you get the animation from the top of the post.
InverseCayley[z_] := I (1 + z)/(1 - z);
Here's the rest of the code:
With[{r = 12., cols = RGBColor /@ {"#381460", "#f4f4f4"}},
Manipulate[
Graphics[
{cols[[1]], Thickness[.003], Line[{{-8, 0}, {8, 0}}],
Table[
Polygon /@
Transpose[
Table[
{ReIm[InverseCayley[E^(I t) + #]],
ReIm[InverseCayley[1/(E^(I t) + #)]]}
&[E^(I θ)*(a + I b)], {t, 0., 2 π,
If[Norm[a + I b] < 8, 2 π/400, 2 π/20]}
]
],
{a, -r, r, 2}, {b, DeleteCases[Range[-r, r, 2], If[a == 0., 0., 100]]}],
If[Abs[θ - π/2] < .01,
{Polygon[{{-8, 2.1}, {8, 2.1}, {8, 10}, {-8, 10}}],
Polygon[{{-8, -2.1}, {8, -2.1}, {8, -10}, {-8, -10}}]}]
},
PlotRange -> 4.5, Axes -> False, ImageSize -> {540, 540},
Background -> cols[[-1]]],
{θ, π/4, 3 π/4}]
]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=HRot12c.gif&userId=610054
[2]: https://twitter.com/beesandbombs/status/1215997000791011328
[3]: https://shonkwiler.org/featured/rotation
[4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ER2c.gif&userId=610054
[5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ER3c.gif&userId=610054Clayton Shonkwiler2020-01-13T00:43:19ZHow-To-Guide: External GPU on OSX - how to use CUDA on your Mac
https://community.wolfram.com/groups/-/m/t/1085633
The neural network and machine learning framework has become one of the key features of the latest releases of the Wolfram Language. Training neural networks can be very time consuming on a standard CPU. Luckily the Wolfram Language offers an incredible easy way to use a GPU to train networks - and do lots of other cool stuff. The problem with this was/is that most current Macs do not have an NVIDIA graphics card, which is necessary to access this framework within the Wolfram Language. Therefore, Wolfram Inc. had decided to drop support for GPUs on Macs. There is however a way to use GPUs on Macs. For example you can use an [external GPU like the one offered by Bizon][1].
![enter image description here][2]
Apart from the BizonBox there a couple of cables and a power supply. You can buy/configure different versions of the BizonBox: there is a range of different graphics cards available and you can buy a the BizonBox 2s which basically connects via Thunderbolt and the BizonBox 3 which connects to USB-C.
Luckily, Wolfram have decided to reintroduce support for GPUs in Mathematica 11.1.1 - see [the discussion here][3].
I have a variety of these BizonBoxes (both 2s and 3) and a range of Macs. I thought it would be a good idea to post a how-to. The essence of what I will be describing in this post should work for most Macs. I ran Sierra on all of them. Here is the recipe to get the thing to work:
Installation of the BizonBox, the required drivers, and compilers
-----------------------------------------------------------------
0. I will assume that you have Sierra installed and that Xcode is running. One of the really important steps if you want to use compilers is to ***downgrade*** the command line tools to version 7.3 You will have to log into your Apple Developer account and download the Command Line Tools version 7.3. Install the tools and run the terminal command (not in Mathematica!):
sudo xcode-select --switch /Library/Developer/CommandLineTools
1. Reboot your Mac into safe mode, i.e. hold CMD+R while rebooting.
2. Open a terminal (under item Utilities at the top of the screen).
3. Enter
csrutil disable
4. Shut the computer down.
5. Connect your BizonBox to the mains and to either the thunderbolt or USB-C port of your Mac.
6. Restart your Mac.
7. Click on the Apple symbol in the top left. Then "About this Mac" and "System Report". In the Thunderbolt section you should see something like this:
![enter image description here][4]
8. In the documentation of the BizonBox you will find a link to a program called bizonboxmac.zip. Download that file and unzip it.
9. Open the folder and click on "bizonbox.prefPane" to install. (If prompted to, do update!)
10. You should see this window:
![enter image description here][6]
11. Click on Activate. Type in password if required to do so. It should give something like this:
![enter image description here][7]
Then restart.
12. Install the CUDA Toolkit: [https://developer.nvidia.com/cuda-downloads][8]. You'll have to click through some questions for the download.
![enter image description here][9]
what you download should be something like cuda_8.0.61_mac.dmg and it should be more or less 1.44 GB worth.
13. Install the toolkit with all its elements.
![enter image description here][10]
14. Restart your computer.
First tests
-----------
Now you should be good to go. Open Mathematica 11.1.1. Execute
Needs["CUDALink`"]
Needs["CCompilerDriver`"]
CUDAResourcesInstall[]
Then try:
CUDAResourcesInformation[]
which should look somewhat like this:
![enter image description here][11]
Then you should check
SystemInformation[]
Head to Links and then CUDA.This should look similar to this:
![enter image description here][12]
So far so good. Next is the really crucial thing:
CUDAQ[]
should give TRUE. If that's what you see you are good to go. Be more daring and try
CUDAImageConvolve[ExampleData[{"TestImage","Lena"}], N[BoxMatrix[1]/9]] // AbsoluteTiming
![enter image description here][13]
You might notice that the non-GPU version of this command runs faster:
ImageConvolve[ExampleData[{"TestImage","Lena"}], N[BoxMatrix[1]/9]] // AbsoluteTiming
runs in something like 0.0824 seconds, but that's ok.
Benchmarking (training neural networks)
---------------------------------------
Let's do some Benchmarking. Download some example data:
obj = ResourceObject["CIFAR-10"];
trainingData = ResourceData[obj, "TrainingData"];
RandomSample[trainingData, 5]
You can check whether it worked:
RandomSample[trainingData, 5]
should give something like this:
![enter image description here][14]
These are the classes of the 50000 images:
classes = Union@Values[trainingData]
![enter image description here][15]
Let's build a network
module = NetChain[{ConvolutionLayer[100, {3, 3}],
BatchNormalizationLayer[], ElementwiseLayer[Ramp],
PoolingLayer[{3, 3}, "PaddingSize" -> 1]}]
net = NetChain[{module, module, module, module, FlattenLayer[], 500,
Ramp, 10, SoftmaxLayer[]},
"Input" -> NetEncoder[{"Image", {32, 32}}],
"Output" -> NetDecoder[{"Class", classes}]]
When you train the network:
{time, trained} = AbsoluteTiming@NetTrain[net, trainingData, Automatic, "TargetDevice" -> "GPU"];
you should see something like this:
![enter image description here][16]
So the thing started 45 secs ago and it supposed to finish in 2m54s. In fact, it finished after 3m30s. If we run the same on the CPU we get:
![enter image description here][17]
The estimate kept changing a bit, but it settled down at about 18h20m.That is slower by a factor of about 315, which is quite substantial.
Use of compiler
---------------
Up to now we have not needed the actual compiler. Let's try this, too. Let's grow a Mandelbulb:
width = 4*640;
height = 4*480;
iconfig = {width, height, 1, 0, 1, 6};
config = {0.001, 0.0, 0.0, 0.0, 8.0, 15.0, 10.0, 5.0};
camera = {{2.0, 2.0, 2.0}, {0.0, 0.0, 0.0}};
AppendTo[camera, Normalize[camera[[2]] - camera[[1]]]];
AppendTo[camera,
0.75*Normalize[Cross[camera[[3]], {0.0, 1.0, 0.0}]]];
AppendTo[camera, 0.75*Normalize[Cross[camera[[4]], camera[[3]]]]];
config = Join[{config, Flatten[camera]}];
pixelsMem = CUDAMemoryAllocate["Float", {height, width, 3}]
srcf = FileNameJoin[{$CUDALinkPath, "SupportFiles", "mandelbulb.cu"}]
Now this should work:
mandelbulb =
CUDAFunctionLoad[File[srcf], "MandelbulbGPU", {{"Float", _, "Output"}, {"Float", _, "Input"}, {"Integer32", _, "Input"}, "Integer32", "Float", "Float"}, {16}, "UnmangleCode" -> False, "CompileOptions" -> "--Wno-deprecated-gpu-targets ", "ShellOutputFunction" -> Print]
Under certain circumstances you might want to specify the location of the compiler like so:
mandelbulb =
CUDAFunctionLoad[File[srcf], "MandelbulbGPU", {{"Float", _, "Output"}, {"Float", _, "Input"}, {"Integer32", _, "Input"}, "Integer32", "Float",
"Float"}, {16}, "UnmangleCode" -> False, "CompileOptions" -> "--Wno-deprecated-gpu-targets ", "ShellOutputFunction" -> Print,
"CompilerInstallation" -> "/Developer/NVIDIA/CUDA-8.0/bin/"]
This should give:
![enter image description here][18]
Now
mandelbulb[pixelsMem, Flatten[config], iconfig, 0, 0.0, 0.0, {width*height*3}];
pixels = CUDAMemoryGet[pixelsMem];
Image[pixels]
gives
![enter image description here][19]
So it appears that all is working fine.
Problems
--------
I did come up with some problems though. There is quite a number of CUDA functions:
Names["CUDALink`*"]
![enter image description here][20]
Many work just fine.
res = RandomReal[1, 5000];
ListLinePlot[res]
![enter image description here][21]
ListLinePlot[First@CUDAImageConvolve[{res}, {GaussianMatrix[{{10}, 10}]}]]
![enter image description here][22]
The thing is that some don't and I am not sure why (I have a hypothesis though). Here are some functions that do **not** appear to work:
CUDAColorNegate
CUDAClamp
CUDAFold
CUDAVolumetricRender
CUDAFluidDynamics
and some more. I would be very grateful if someone could check these on OSX (and perhaps Windows?). I am not sure if the this is due to some particularity of my systems or something that could be flagged up to Wolfram Inc for checking.
When I wanted to try that systematically I wanted to use the function
WolframLanguageData
to look for the first example in the documentation of the CUDA functions, but it appears that no CUDA function is in the WolframLanguageData. I think tit would be great to have them there, too, and am not sure why they wouldn't be there.
In spite of these problems I hope that this post will help some Mac users to get CUDA going. It is a great framework and simple to use in the Wolfram Language. With the BizonBox and Mathematica 11.1.1 Mac users are no longer excluded from accessing this feature.
Cheers,
Marco
PS: Note, that there is anecdotal evidence that one can even use the BizonBox under Windows running in a virtual box under OSX. I don't have Windows, but I'd like to hear if anyone get this running.
[1]: https://bizon-tech.com
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.09.10.png&userId=48754
[3]: http://community.wolfram.com/groups/-/m/t/902394
[4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot1.png&userId=48754
[5]: http://bizon-tech.com/bizonboxmac.zip
[6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2.png&userId=48754
[7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot3.png&userId=48754
[8]: https://developer.nvidia.com/cuda-downloads
[9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot6.png&userId=48754
[10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot7.png&userId=48754
[11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.38.22.png&userId=48754
[12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-06at18.46.46.png&userId=48754
[13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.49.15.png&userId=48754
[14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.52.43.png&userId=48754
[15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.53.30.png&userId=48754
[16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at20.37.20.png&userId=48754
[17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at20.39.02.png&userId=48754
[18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at23.04.38.png&userId=48754
[19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at21.50.42.png&userId=48754
[20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at23.10.05.png&userId=48754
[21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at23.15.59.png&userId=48754
[22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at23.16.36.png&userId=48754Marco Thiel2017-05-07T22:21:42ZUsing SPARQL to construct a timeline of the Supreme Court Chief Justices
https://community.wolfram.com/groups/-/m/t/1857773
The intent of this post is to show those inexpert in SPARQL but curious about capabilities how much can be done with it within the Wolfram Language. My specific quest is to generate a timeline of chief justices of the United States Supreme Court but, in the course of doing so, to share some learning about SPARQL and Wikidata.
The main insight is to exploit the many examples of SPARQL code at the Wikidata query site, find one that is relevant, and then use ImportString[#,"SPARQLQuery"]& to generate symbolic SPARQL within the Wolfram Language. Once that's done, we can use core Wolfram Language tools to modify the code for the specific task at hand and then run it against the Wikidata "endpoint." Once the results come back from Wikidata, some basic Wolfram Language code generates a nice timeline with relative ease.
The notebook finishes with a little section on how to run Python from within a Wolfram notebook and get the same kind of results. Again, one doesn't need to know much about Python to do this (trust me!). Wikidata actually itself provides the basic code. And, as version 12.1 and beyond permit the Wolfram ecosystem to expand its capabilities with other languages, it may be possible to use Wikidata's fluency in multiple computer languages (Ruby, JavaScript, etc.) to diversify the set of tools with which to attack Wikidata while remaining in the comfortable and powerful Wolfram system.
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/wolfram-community/Published/Chief%20Justice%20Timeline%20from%20Wikidata.nbSeth Chandler2020-01-13T16:50:45Z[Notebook] How to make a Christmas card on the cloud
https://community.wolfram.com/groups/-/m/t/1849339
[![enter image description here][1]](https://www.wolframcloud.com/env/silviah/Published/xmas.html)
**POSTCARD:** https://www.wolframcloud.com/env/silviah/Published/xmas.html
&[embedded notebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Xmas_postcard.jpg&userId=20103
[2]: https://www.wolframcloud.com/obj/wolfram-community/Published/How_to_make_a_Christmas_card_on_the_cloud.nbSilvia Hao2019-12-29T14:07:04ZFinding all structural breaks in time series
https://community.wolfram.com/groups/-/m/t/1749226
## Introduction
In this document we show how to find the so called "structural breaks",
[[Wk1](https://en.wikipedia.org/wiki/Structural_break)],
in a given time series.
The algorithm is based in on a systematic application of Chow Test,
[[Wk2](https://en.wikipedia.org/wiki/Chow_test)],
combined with an algorithm for local extrema finding in noisy time series,
[[AA1](https://mathematicaforprediction.wordpress.com/2015/09/27/finding-local-extrema-in-noisy-data-using-quantile-regression/)].
The algorithm implementation is based on the packages
["MonadicQuantileRegression.m"](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m),
[[AAp1](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m)],
and ["MonadicStructuralBreaksFinder.m"](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicStructuralBreaksFinder.m),
[[AAp2](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicStructuralBreaksFinder.m)].
The package [[AAp1](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m)]
provides the software monad QRMon that allows rapid and concise specification of
[Quantile Regression](https://en.wikipedia.org/wiki/Quantile_regression) workflows.
The package
[[AAp2](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicStructuralBreaksFinder.m)]
extends QRMon with functionalities related to structural breaks finding.
### What is a structural break?
It looks like at least one type of "structural breaks" are defined through regression models, [[Wk1](https://en.wikipedia.org/wiki/Structural_break)]. Roughly speaking a structural break point of time series is a regressor point that splits the time series in such way that the obtained two parts have very different regression parameters.
One way to test such a point is to use Chow test, [[Wk2](https://en.wikipedia.org/wiki/Chow_test)]. From [[Wk2](https://en.wikipedia.org/wiki/Chow_test)] we have the definition:
The Chow test, proposed by econometrician Gregory Chow in 1960, is a test of whether the true coefficients in two linear regressions on different data sets are equal. In econometrics, it is most commonly used in time series analysis to test for the presence of a structural break at a period which can be assumed to be known a priori (for instance, a major historical event such as a war).
### Example
Here is an example of the described algorithm application to the data from [[Wk2](https://en.wikipedia.org/wiki/Chow_test#/media/File:Chowtest4.svg)].
QRMonUnit[data]⟹QRMonPlotStructuralBreakSplits[ImageSize -> Small];
![IntroductionsExample](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Introductions-example.png)
## Load packages
Here we load the packages [AAp1] and [AAp2].
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicQuantileRegression.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicStructuralBreaksFinder.m"]
## Data used
In this section we assign the data used in this document.
### Illustration data from Wikipedia
Here is the data used in the Wikipedia article "Chow test", [[Wk2](https://en.wikipedia.org/wiki/Chow_test#/media/File:Chowtest4.svg)].
data = {{0.08, 0.34}, {0.16, 0.55}, {0.24, 0.54}, {0.32, 0.77}, {0.4,
0.77}, {0.48, 1.2}, {0.56, 0.57}, {0.64, 1.3}, {0.72, 1.}, {0.8,
1.3}, {0.88, 1.2}, {0.96, 0.88}, {1., 1.2}, {1.1, 1.3}, {1.2,
1.3}, {1.3, 1.4}, {1.4, 1.5}, {1.4, 1.5}, {1.5, 1.5}, {1.6,
1.6}, {1.7, 1.1}, {1.8, 0.98}, {1.8, 1.1}, {1.9, 1.4}, {2.,
1.3}, {2.1, 1.5}, {2.2, 1.3}, {2.2, 1.3}, {2.3, 1.2}, {2.4,
1.1}, {2.5, 1.1}, {2.6, 1.2}, {2.6, 1.4}, {2.7, 1.3}, {2.8,
1.6}, {2.9, 1.5}, {3., 1.4}, {3., 1.8}, {3.1, 1.4}, {3.2,
1.4}, {3.3, 1.4}, {3.4, 2.}, {3.4, 2.}, {3.5, 1.5}, {3.6,
1.8}, {3.7, 2.1}, {3.8, 1.6}, {3.8, 1.8}, {3.9, 1.9}, {4., 2.1}};
ListPlot[data]
![DataUsedWk2](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Data-used-Wk2.png)
### S&P 500 Index
Here we get the time series corresponding to [S&P 500 Index](https://en.wikipedia.org/wiki/S%26P_500_Index).
tsSP500 = FinancialData[Entity["Financial", "^SPX"], {{2015, 1, 1}, Date[]}]
DateListPlot[tsSP500, ImageSize -> Medium]
![DataUsedSP500](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Data-used-SP500.png)
## Application of Chow Test
The Chow Test statistic is implemented in [AAp1].
In this document we rely on the relative comparison of the Chow Test statistic values: the larger the value of the Chow test statistic,
the more likely we have a structural break.
Here is how we can apply the Chow Test with a QRMon pipeline to the [Wk2] data given above.
chowStats =
QRMonUnit[data]⟹
QRMonChowTestStatistic[Range[1, 3, 0.05], {1, x}]⟹
QRMonTakeValue;
We see that the regressor points $\text{$\$$Failed}$ and $1.7$ have the largest Chow Test statistic values.
Block[{chPoint = TakeLargestBy[chowStats, Part[#, 2]& , 1]},
ListPlot[{chowStats, chPoint}, Filling -> Axis, PlotLabel -> Row[{"Point with largest Chow Test statistic:",
Spacer[8], chPoint}]]]
![ApplicationOfChowTestchowStats](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Application-of-Chow-Test-chowStats.png)
The first argument of QRMonChowTestStatistic is a list of regressor points or Automatic.
The second argument is a list of functions to be used for the regressions.
Here is an example of an automatic values call.
chowStats2 = QRMonUnit[data]⟹QRMonChowTestStatistic⟹QRMonTakeValue;
ListPlot[chowStats2, GridLines -> {
Part[
Part[chowStats2, All, 1],
OutlierIdentifiers`OutlierPosition[
Part[chowStats2, All, 2], OutlierIdentifiers`SPLUSQuartileIdentifierParameters]], None}, GridLinesStyle -> Directive[{Orange, Dashed}], Filling -> Axis]
![ApplicationOfChowTestchowStats2](https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Application-of-Chow-Test-chowStats2.png)
For the set of values displayed above we can apply simple 1D outlier identification methods,
[[AAp3](https://github.com/antononcube/MathematicaForPrediction/blob/master/OutlierIdentifiers.m)],
to automatically find the structural break point.
chowStats2[[All, 1]][[OutlierPosition[chowStats2[[All, 2]], SPLUSQuartileIdentifierParameters]]]
(* {1.7} *)
OutlierPosition[chowStats2[[All, 2]], SPLUSQuartileIdentifierParameters]
(* {20} *)
We cannot use that approach for finding all structural breaks in the general time series cases though as exemplified with the following code using the time series S&P 500 Index.
chowStats3 = QRMonUnit[tsSP500]⟹QRMonChowTestStatistic⟹QRMonTakeValue;
DateListPlot[chowStats3, Joined -> False, Filling -> Axis]
![ApplicationOfChowTestSP500](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Application-of-Chow-Test-SP500.png)
OutlierPosition[chowStats3[[All, 2]], SPLUSQuartileIdentifierParameters]
(* {} *)
OutlierPosition[chowStats3[[All, 2]], HampelIdentifierParameters]
(* {} *)
In the rest of the document we provide an algorithm that works for general time series.
## Finding all structural break points
Consider the problem of finding of **all** structural breaks in a given time series.
That can be done (reasonably well) with the following procedure.
1. Chose functions for testing for structural breaks (usually linear.)
2. Apply Chow Test over dense enough set of regressor points.
3. Make a time series of the obtained Chow Test statistics.
4. Find the local maxima of the Chow Test statistics time series.
5. Determine the most significant break point.
6. Plot the splits corresponding to the found structural breaks.
QRMon has a function, QRMonFindLocalExtrema, for finding local extrema; see [AAp1, AA1].
For the goal of finding all structural breaks, that semi-symbolic algorithm is the crucial part in the steps above.
## Computation
### Chose fitting functions
fitFuncs = {1, x};
### Find Chow test statistics local maxima
The computation below combines steps 2,3, and 4.
qrObj =
QRMonUnit[tsSP500]⟹
QRMonFindChowTestLocalMaxima["Knots" -> 20,
"NearestWithOutliers" -> True,
"NumberOfProximityPoints" -> 5, "EchoPlots" -> True,
"DateListPlot" -> True,
ImageSize -> Medium]⟹
QRMonEchoValue;
![ComputationLocalMaxima](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Computation-local-maxima.png)
### Find most significant structural break point
splitPoint = TakeLargestBy[qrObj⟹QRMonTakeValue, #[[2]] &, 1][[1, 1]]
### Plot structural breaks splits and corresponding fittings
Here we just make the plots without showing them.
sbPlots =
QRMonUnit[tsSP500]⟹
QRMonPlotStructuralBreakSplits[(qrObj⟹ QRMonTakeValue)[[All, 1]],
"LeftPartColor" -> Gray, "DateListPlot" -> True,
"Echo" -> False,
ImageSize -> Medium]⟹
QRMonTakeValue;
The function QRMonPlotStructuralBreakSplits returns an association that has as keys paired split points and Chow Test statistics; the plots are association's values.
Here we tabulate the plots with plots with most significant breaks shown first.
Multicolumn[
KeyValueMap[
Show[#2, PlotLabel ->
Grid[{{"Point:", #1[[1]]}, {"Chow Test statistic:", #1[[2]]}}, Alignment -> Left]] &, KeySortBy[sbPlots, -#[[2]] &]], 2]
![ComputationStructuralBreaksPlots](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Computation-structural-breaks-plots.png)
## Future plans
We can further apply the algorithm explained above to identifying time series states or components.
The structural break points are used as knots in appropriate Quantile Regression fitting. Here is an example.
The plan is to develop such an identifier of time series states in the near future.
(And present it at [WTC-2019](https://www.wolfram.com/events/technology-conference/2019/).)
![FuturePlansTimeSeriesStates](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Future-plans-time-series-states.png)
## References
### Articles
\[Wk1\] Wikipedia entry, [Structural breaks](https://en.wikipedia.org/wiki/Structural_break).
\[Wk2\] Wikipedia entry, [Chow test](https://en.wikipedia.org/wiki/Chow_test).
\[AA1\] Anton Antonov, ["Finding local extrema in noisy data using Quantile Regression"](https://mathematicaforprediction.wordpress.com/2015/09/27/finding-local-extrema-in-noisy-data-using-quantile-regression/), (2019), [MathematicaForPrediction at WordPress](https://mathematicaforprediction.wordpress.com/2015/09/27/finding-local-extrema-in-noisy-data-using-quantile-regression/).
\[AA2\] Anton Antonov, ["A monad for Quantile Regression workflows"](https://github.com/antononcube/MathematicaForPrediction/blob/master/MarkdownDocuments/A-monad-for-Quantile-Regression-workflows.md), (2018), [MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction/).
### Packages
\[AAp1\] Anton Antonov, [Monadic Quantile Regression Mathematica package](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m), (2018), [MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction/).
\[AAp2\] Anton Antonov, [Monadic Structural Breaks Finder Mathematica package](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicStructuralBreaksFinder.m), (2019), [MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction/).
\[AAp3\] Anton Antonov, [Implementation of one dimensional outlier identifying algorithms in Mathematica](https://github.com/antononcube/MathematicaForPrediction/blob/master/OutlierIdentifiers.m), (2013), [MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction/).
### Videos
\[AAv1\] Anton Antonov, Structural Breaks with QRMon, (2019), YouTube.Anton Antonov2019-07-31T19:26:28Z[Notebook] Summable Areas
https://community.wolfram.com/groups/-/m/t/1849304
##### *This is an interesting geometry problem posted by Dr. Shan Zun, ex-IMO head coach, PRC*
----------
![illustration][1]
&[embedded notebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=illustration.png&userId=23928
[2]: https://www.wolframcloud.com/obj/wolfram-community/Published/EqualSumArea_Publish.nbShenghui Yang2019-12-28T21:53:45Z