Community RSS Feed
http://community.wolfram.com
RSS Feed for Wolfram Community showing any discussions in tag Staff Picks sorted by activeVisual Interface to the Wolfram Language
http://community.wolfram.com/groups/-/m/t/1124967
Hi everyone.
I'm working on a visual interface to the Wolfram Language called [visX][1], and I'd like to ask what you all think of it.
Wolfram Language code can often be thought of as a set of blocks, each of which takes some inputs, does something, and produces an output. VisX lets you write WL code exactly this way - you draw a digram, connecting blocks with links. For example, say you want to count how many times each digit (0 to 9) occurs in the first 30 digits of Pi. With text-based WL code, you'd write
digits = RealDigits[N[Pi, 30]][[1]]
Count[digits, #] & /@ Range[0, 9]
In visX, you'd draw this:
![digits of Pi][2]
I guess it's pretty self-explanatory. In addition to using built-in WL blocks, you can write your own, like the CountInList block. Normally, blocks just transform inputs to output, but the CountInList block is mapped over its input which is indicated by the little brackets on the outside of its connection ports. (That's basically visual syntactic sugar for "/@" or Map.) The 4 in the upper-right corner indicates that results inside this block are showing results from the 4-th time through the map. The block with "digits" in it sets a variable, which is then referenced in the CountInList block.
You define blocks (which are basically functions) by just making an empty rectangle and dragging contents in then wiring them together, then you can use copies of the block wherever you want. A change in any copy of the block will be reflected in all other copies. There's no real difference between a defining a block and using it. Recursion can be specified by just including a copy of the block inside itself. Blocks can call other blocks in the same manner.
Just like regular WL code, visX blocks can be nested deeply, but with the visual interface, it's easy to zoom in and out. At any point, the UI will show you the right amount of detail for each block - sometimes no detail at all, sometimes its name and labels on its inputs, sometimes its actual contents (which can then be edited or further zoomed...).
visX is stand-alone software that runs locally on your machine, evaluates the diagram using your local Mathematica kernel, and receives the results and puts them back in the diagram. You can load data files using Import as usual.
One of the problems that I've seen with visual languages in the past is that while simple things are easy to do, the code quickly gets too complex to manage and the visual interface starts to get in the way. With the Wolfram Language in theory everything is an expression, and this can lead you to write functional-style programs which are easily thought of as a diagram, but that's not always the most natural way to express a computation. Sometimes you just need a little for loop. Consider calculating Fibonacci numbers. Start the sequence with 1, 1, ... then each element of the sequence is the sum of the previous two. Yes, you can write a recursive algorithm to do this, but most people just want to write a little for loop. In visX, you can do this (calculates the 6th Fibonacci number):
![embedded code][4]
I've tried to let you use blocks-and-links when that's the most natural thing (which is usually), and text-based code when that's better. Of course, you can mix them together however you want.
A second problem I've found with visual programming languages is that it can actually be much slower to use then writing out text, because you have to laboriously drag and drop every single block. Even simple algebraic expressions like
Sin[x]^2 + Cos[x]^2
2x^2 + 4x*y + 8y^2
would involve a lot of blocks because of all the Plus, Times, and Power blocks, as well as all the constants and symbols. With visX, you can enter Wolfram Language code snippets like those, and it will parse them and transform them into blocks which you can then insert into your diagram all at once and edit at will. This makes it much faster to get your idea onto the screen so that you can start evaluating it and developing it. I'm also working on the ability to take a visX block and give you back the Wolfram Language code that it represents.
The examples given here are simple, but of course you can use this interface for putting together a complex piece of code as well. I find it especially handy when building up a calculation with lots of intermediate results along the way, or to rapidly prototype an algorithm where I want to be able to easily switch the data flows around.
Does this project seem useful to anyone? I'd like to get some feedback - what do you think of it? Would you use it? For what?
If there's interest, I could do a small-scale alpha test in about a month from now.
More info at [visx.io][5].
-Nicholas Hoff
*edited to clarify block definitions and recursion*
[1]: http://visx.io
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=pi_digits_without_chrome.png&userId=1124239
[3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=pi_digits.png&userId=1124239
[4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=embeded_wl.png&userId=1124239
[5]: http://visx.ioNicholas Hoff2017-06-20T10:02:52ZAbout package initialization
http://community.wolfram.com/groups/-/m/t/982404
I would like to share some thoughts about package initialization. As always, I post these ideas in the hope to stimulate discussion between package developers, hopefully get some feedback and learn something. It is more efficient if we learn from each other than if each of us has to go through the same stumbling blocks alone.
## Introduction
Some packages only contain definitions. Complex packages typically contain some sort of initialization—code that evaluates when the package loads. There are many different reasons why this may be necessary:
- Load external dependencies such as LibraryLink functions, installable MathLink programs, etc.
- Load and process configuration files
- Load data files (precomputed data, caches, etc.)
- Verify and set up some external environment, e.g. what if the package needs to call external processes using RunProcess, or even modify the system `PATH` or `LD_LIBRARY_PATH` to make them work? (MaTeX needs something similar)
## Things to keep in mind when the package has initialization code
Initialization code can affect loading performance. A polished, high-quality package loads without delay. This is especially important on slow platforms such as the Raspberry Pi computer. The currently released version of my IGraph/M package takes 30-60 seconds to load on the Raspberry Pi, which is just unacceptable. The development version fixes this.
Initialization code can cause serious problems when the package is loaded on kernel startup, not to mention slowing down kernel start. There are certain fundamental functions which [do not work during kernel initialization][1], such as `Throw` and `Catch`. These are dependencies to many other common functions. In the end, even very bad things can happen: if `Import` is used during kernel initialization, not only will it fail once, it will also stop working for the rest of the session.
How might a package get loaded during kernel startup? A user may load it in their kernel `init.m` file. Or they may place it in their `$UserBaseDirectory/Autoload` directory. Or if the package is distributed as a paclet, [Mathematica will offer a GUI setting to load it on startup][2]. According to my experience, Murphy's laws apply everywhere: if you give your users even the slightest chance to mess up something, one of them certainly will.
Of course, these are minor issues. You do not need to solve them to write a useful package. But if you are aiming for creating a high-quality package that will be used my many people, it is good to keep these things in mind.
## Strategies for robust initialization
This is the section I would like some feedback on. My goal is to make initialization *fast* and *robust* so it is safe to run during kernel startup.
### Prefer simple functions to complex ones
Try to use functions which run fast and are safe during initialization.
For reading and writing configuration files, prefer `Get` instead of `Import[..., "Package"]` and `Put` instead of `Export[..., "Package"]`.
For reading text files, prefer `ReadString` instead of `Import[..., "String"]`.
Generally, use low-level IO functions, such as `Read`, `ReadList`, etc., and do not forget to `Close` the stream.
`ReadString` is new in version 10.0, and triggers loading ProcessLink (which contains functions like `RunProcess`). This means that we cannot use it in v9.0, and also its first use is slow because it has to load soe dependencies. A faster and more compatible alternative is
readString[file_] :=
Module[{stream}
stream = OpenRead[file];
res = Read[stream, Record, RecordSeparators -> {}];
Close[stream]
res
]
### Delay initialization when possible
This is the most effective and most general tool when dealing with initialization. The usual idiom to define a symbol `sym` is
sym := sym = computeValue[];
Then `sym` will be computed the first time it is *used* and not when the package is *loaded*. `sym` could be a variable holding persistent configuration that is read from a file, precomputed data, or even a LibraryLink function:
libFun := libFun = LibraryFunctionLoad[...];
This idiom is extremely convenient because it can be applied to existing code with a minimal change.
### Use scheduled tasks to delay-load
I learnt this trick from [@WReach on StackExchange](http://mathematica.stackexchange.com/a/132225/12) and [@Ilian Gachevski][at0].
Use a scheduled task to delay initialization until kernel startup has finished:
RunScheduledTask[
(* perform some complex initialization *)
; RemoveScheduledTask @ $ScheduledTask
, {0}
]
This is really just a hack and it is not fully reliable (perhaps there's a risk of race conditions). I needed to use a delay of 1 second instead of 0 seconds to make it work reliably.
I do not recommend using this in published packages, but the technique is very useful for personal packages. I use it to set up [function argument completions](http://mathematica.stackexchange.com/a/129910/12) in a personal package that I always auto-load.
### If your users ask about reliably auto-loading your package, suggest DeclarePackage
This is just an idea: if you need to load a package at kernel startup (but not actually use it in kernel init files), [then use `DeclarePackage`][3] instead of `Needs`
<br><br>
Any comments, suggestion, are most welcome!
[at0]: http://community.wolfram.com/web/iliang
[1]: http://mathematica.stackexchange.com/q/17164/12
[2]: http://mathematica.stackexchange.com/a/132065/12
[3]: http://reference.wolfram.com/language/tutorial/AutomaticLoadingOfPackages.htmlSzabolcs Horvát2016-12-16T11:47:52Z[Mathematica-vs-R] Text analysis of Trump tweets
http://community.wolfram.com/groups/-/m/t/967299
## Introduction
This post is to proclaim the [MathematicaVsR at GitHub](https://github.com/antononcube/MathematicaVsR) project ["Text analysis of Trump tweets"](https://github.com/antononcube/MathematicaVsR/tree/master/Projects/TextAnalysisOfTrumpTweets) in which we compare Mathematica and R over text analyses of Twitter messages made by Donald Trump (and his staff) before the USA president elections in 2016.
This project follows and extends the exposition and analysis of the R-based blog post ["Text analysis of Trump's tweets confirms he writes only the (angrier) Android half"](http://varianceexplained.org/r/trump-tweets/) by David Robinson at [VarianceExplained.org](http://varianceexplained.org); see [1].
The blog post \[[1](http://varianceexplained.org/r/trump-tweets/)\] links to several sources that claim that during the election campaign Donald Trump tweeted from his Android phone and his campaign staff tweeted from an iPhone. The blog post [1] examines this hypothesis in a quantitative way (using various R packages.)
The hypothesis in question is well summarized with the tweet:
> Every non-hyperbolic tweet is from iPhone (his staff).
> Every hyperbolic tweet is from Android (from him). [pic.twitter.com/GWr6D8h5ed](pic.twitter.com/GWr6D8h5ed)
> -- Todd Vaziri (@tvaziri) August 6, 2016
This conjecture is fairly well supported by the following [mosaic plots](https://mathematicaforprediction.wordpress.com/2014/03/17/mosaic-plots-for-data-visualization/), \[[2](https://mathematicaforprediction.wordpress.com/2014/03/17/mosaic-plots-for-data-visualization/)\]:
[![TextAnalysisOfTrumpTweets-iPhone-MosaicPlot-Sentiment-Device](http://i.imgur.com/eKjxlTvm.png)](http://i.imgur.com/eKjxlTv.png) [![TextAnalysisOfTrumpTweets-iPhone-MosaicPlot-Device-Weekday-Sentiment](http://i.imgur.com/RMfuNNtm.png)](http://i.imgur.com/RMfuNNt.png)
We can see the that Twitter messages from iPhone are much more likely to be neutral, and the ones from Android are much more polarized. As
Christian Rudder (one of the founders of [OkCupid](https://www.okcupid.com), a dating website) explains in the chapter "Death by a Thousand Mehs" of the book ["Dataclysm"](http://dataclysm.org), \[[3](http://dataclysm.org)\], having a polarizing image (online persona) is as a very good strategy to engage online audience:
> [...] And the effect isn't small-being highly polarizing will in fact get you about 70 percent more messages. That means variance allows you to effectively jump several "leagues" up in the dating pecking order - [...]
(The mosaic plots above were made for the Mathematica-part of this project. Mosaic plots and weekday tags are not used in [1].)
### Links
- The Mathematica part: [PDF file](https://github.com/antononcube/MathematicaVsR/blob/master/Projects/TextAnalysisOfTrumpTweets/Mathematica/Text-analysis-of-Trump-tweets.pdf), [Markdown file](https://github.com/antononcube/MathematicaVsR/blob/master/Projects/TextAnalysisOfTrumpTweets/Mathematica/Text-analysis-of-Trump-tweets.md).
- The R part consists of :
- the blog post \[[1](http://varianceexplained.org/r/trump-tweets/)\], and
- the R-notebook given as [Markdown](https://github.com/antononcube/MathematicaVsR/blob/master/Projects/TextAnalysisOfTrumpTweets/R/TextAnalysisOfTrumpTweets.Rmd) and [HTML](https://cdn.rawgit.com/antononcube/MathematicaVsR/master/Projects/TextAnalysisOfTrumpTweets/R/TextAnalysisOfTrumpTweets.nb.html).
## Concrete steps
The Mathematica-part of this project does not follow closely the blog post [1]. After the ingestion of the data provided in [1], the Mathematica-part applies alternative algorithms to support and extend the analysis in [1].
The sections in the [R-part notebook](https://github.com/antononcube/MathematicaVsR/blob/master/Projects/TextAnalysisOfTrumpTweets/R/TextAnalysisOfTrumpTweets.Rmd) correspond to some -- not all -- of the sections in the Mathematica-part.
The following list of steps is for the Mathematica-part.
1. **Data ingestion**
- The blog post [1] shows how to do in R the ingestion of Twitter data of Donald Trump messages.
- That can be done in Mathematica too using the built-in function `ServiceConnect`,
but that is not necessary since [1] provides a link to the ingested data used [1]:
load(url("http://varianceexplained.org/files/trump_tweets_df.rda"))
- Which leads to the ingesting of an R data frame in the Mathematica-part using RLink.
2. **Adding tags**
- We have to extract device tags for the messages -- each message is associated with one of the tags "Android", "iPad", or "iPhone".
- Using the message time-stamps each message is associated with time tags corresponding to the creation time month, hour, weekday, etc.
- Here is summary of the data at this stage:
![enter image description here][1]
3. **Time series and time related distributions**
- We can make several types of time series plots for general insight and to support the main conjecture.
- Here is a Mathematica made plot for the same statistic computed in [1] that shows differences in tweet posting behavior:
![enter image description here][2]
- Here are distributions plots of tweets per weekday:
![enter image description here][3]
4. **Classification into sentiments and Facebook topics**
- Using the built-in classifiers of Mathematica each tweet message is associated with a sentiment tag and a Facebook topic tag.
- In [1] the results of this step are derived in several stages.
- Here is a mosaic plot for conditional probabilities of devices, topics, and sentiments:
![enter image description here][4]
5. **Device-word association rules**
- Using [Association rule learning](https://en.wikipedia.org/wiki/Association_rule_learning) device tags are associated with words in the tweets.
- In the Mathematica-part these associations rules are not needed for the sentiment analysis (because of the built-in classifiers.)
- The association rule mining is done mostly to support and extend the text analysis in [1] and, of course, for comparison purposes.
- Here is an example of derived association rules together with their most important measures:
![enter image description here][5]
In [1] the sentiments are derived from computed device-word associations, so in [1] the order of steps is 1-2-3-5-4. In Mathematica we do not need the steps 3 and 5 in order to get the sentiments in the 4th step.
## Comparison
Using Mathematica for sentiment analysis is much more direct because of the built-in classifiers.
The R-based blog post [1] uses heavily the "pipeline" operator `%>%` which is kind of a recent addition to R (and it is both fashionable and convenient to use it.) In Mathematica the related operators are `Postfix` (`//`), `Prefix` (`@`), `Infix` (`~~`), `Composition` (`@*`), and `RightComposition` (`/*`).
Making the time series plots with the R package "ggplot2" requires making special data frames. I am inclined to think that the Mathematica plotting of time series is more direct, but for this task the data wrangling codes in Mathematica and R are fairly comparable.
Generally speaking, the R package ["arules"](https://cran.r-project.org/web/packages/arules/index.html) -- used in this project for Associations rule learning -- is somewhat awkward to use:
- it is data frame centric, does not work directly with lists of lists, and
- requires the use of factors.
The Apriori implementation in ["arules"](https://cran.r-project.org/web/packages/arules/index.html) is much faster than the one in ["AprioriAlgorithm.m"](https://github.com/antononcube/MathematicaForPrediction/blob/master/AprioriAlgorithm.m) -- "arules" uses a more efficient algorithm [implemented in C](http://www.borgelt.net/fpm.html).
## References
\[1\] David Robinson, ["Text analysis of Trump's tweets confirms he writes only the (angrier) Android half"](http://varianceexplained.org/r/trump-tweets/), (2016), [VarianceExplained.org](http://varianceexplained.org).
\[2\] Anton Antonov, ["Mosaic plots for data visualization"](https://mathematicaforprediction.wordpress.com/2014/03/17/mosaic-plots-for-data-visualization/), (2014), [MathematicaForPrediction at WordPress](https://mathematicaforprediction.wordpress.com).
\[3\] Christian Rudder, [Dataclysm](http://dataclysm.org), Crown, 2014. ASIN: B00J1IQUX8 .
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=yMtdphT.png&userId=143837
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=oDv5Cm0.png&userId=143837
[3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=UGMy4EW.png&userId=143837
[4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=dMxSpHa.png&userId=143837
[5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=dSSb4KD.png&userId=143837Anton Antonov2016-11-21T10:51:46ZHiggs Boson Classification via Neural Network
http://community.wolfram.com/groups/-/m/t/1016315
![enter image description here][1]
# Introduction
So here is a simple approach that applies Wolfram Language machine learning functions to a classification problem for finding possible Higgs particles. It uses a labeled data set with 30 numerical physical attributes (things like measured spins, angles, energies, etc.) and with labels being either 'signal' (s) or 'background' (b). The attached notebook runs through a sample analysis in the Wolfram Language: importing the training data, cleaning it up for using it, setting up a neural network, training the network with the data, and finally checking how well the trained neural network does at making predictions. Here is the description of data from the source website at KAGGLE:
> Discovery of the long awaited Higgs boson was announced July 4, 2012 and confirmed six months later. 2013 saw a number of prestigious awards, including a Nobel prize. But for physicists, the discovery of a new particle means the beginning of a long and difficult quest to measure its characteristics and determine if it fits the current model of nature.
> A key property of any particle is how often it decays into other particles. ATLAS is a particle physics experiment taking place at the Large Hadron Collider at CERN that searches for new particles and processes using head-on collisions of protons of extraordinarily high energy. The ATLAS experiment has recently observed a signal of the Higgs boson decaying into two tau particles, but this decay is a small signal buried in background noise.
> The goal of the Higgs Boson Machine Learning Challenge is to explore the potential of advanced machine learning methods to improve the discovery significance of the experiment. No knowledge of particle physics is required. Using simulated data with features characterizing events detected by ATLAS, your task is to classify events into "tau tau decay of a Higgs boson" versus "background."
> The winning method may eventually be applied to real data and the winners may be invited to CERN to discuss their results with high energy physicists.
# References and sources
- [Learning to discover: the Higgs boson machine learning challenge][2]
- [KAGGLE: Higgs Boson Machine Learning Challenge][3]
- [Opendata ATLAS][4]
# Training data
Import the training data:
training = Import["D:\\machinelearning\\higgs\\training\\training.csv", "Data"];
Dimensions[training]
`{250001, 33}`
Look at the data fields (they are described in the pdf link above). "EventId" should not be used as part of the training, since it has no predictive value. The last column "Label" is the classification (s=signal, b=background)
training[[1]]
`{"EventId", "DER_mass_MMC", "DER_mass_transverse_met_lep", \
"DER_mass_vis", "DER_pt_h", "DER_deltaeta_jet_jet", \
"DER_mass_jet_jet", "DER_prodeta_jet_jet", "DER_deltar_tau_lep", \
"DER_pt_tot", "DER_sum_pt", "DER_pt_ratio_lep_tau", \
"DER_met_phi_centrality", "DER_lep_eta_centrality", "PRI_tau_pt", \
"PRI_tau_eta", "PRI_tau_phi", "PRI_lep_pt", "PRI_lep_eta", \
"PRI_lep_phi", "PRI_met", "PRI_met_phi", "PRI_met_sumet", \
"PRI_jet_num", "PRI_jet_leading_pt", "PRI_jet_leading_eta", \
"PRI_jet_leading_phi", "PRI_jet_subleading_pt", \
"PRI_jet_subleading_eta", "PRI_jet_subleading_phi", "PRI_jet_all_pt", \
"Weight", "Label"}`
Sample vector:
training[[2]]
`{100000, 138.47, 51.655, 97.827, 27.98, 0.91, 124.711, 2.666, 3.064, \
41.928, 197.76, 1.582, 1.396, 0.2, 32.638, 1.017, 0.381, 51.626, \
2.273, -2.414, 16.824, -0.277, 258.733, 2, 67.435, 2.15, 0.444, \
46.062, 1.24, -2.475, 113.497, 0.00265331, "s"}`
Set up a simple neural network (this can be tinkered with to improve the results):
net=NetInitialize[
NetChain[{
LinearLayer[3000],Ramp,LinearLayer[3000],Ramp,LinearLayer[2],SoftmaxLayer[]
},
"Input"->{30},
"Output"->NetDecoder[{"Class",{"b","s"}}]
]]
![enter image description here][5]
Set up the training data:
data = Map[Take[#, {2, 31}] -> Last[#] &, Drop[training, 1]];
Numerical vectors that each point to a classification (s or b):
RandomSample[data, 3]
`{{87.06, 23.069, 67.711, 162.488, -999., -999., -999., 0.903, 4.245,
318.43, 0.523, 0.839, -999., 105.019, -0.404, 1.612,
54.943, -0.898, 0.855, 13.541, 1.729, 409.232, 1,
158.469, -1.363, -1.762, -999., -999., -999., 158.469} ->
"b", {163.658, 55.559, 116.84, 50.019, -999., -999., -999., 2.855,
38.623, 130.132, 0.906, 1.359, -999., 51.993, 0.36, 2.142,
47.112, -0.932, -1.596, 22.782, 2.663, 191.557, 1, 31.026, -2.333,
0.739, -999., -999., -999., 31.026} ->
"b", {100.248, 27.109, 60.729, 132.094, -999., -999., -999., 1.405,
10.063, 218.474, 0.541, 1.414, -999., 62.519, -0.401, -1.974,
33.817, -0.893, -0.657, 54.857, -1.298, 396.228, 1,
122.137, -2.369, 1.689, -999., -999., -999., 122.137} -> "s"}`
# Training
Length[data]
`250000`
{tdata,vdata}=TakeDrop[data,240000];
result=NetTrain[net,tdata,TargetDevice->"GPU",ValidationSet->Scaled[0.1],MaxTrainingRounds->1000]
![enter image description here][6]
DumpSave["D:\\machinelearning\\higgs\\higgs.mx", result];
# Testing
This is the test data (unlabeled):
test = Import["D:\\machinelearning\\higgs\\test\\test.csv", "Data"];
Extract the validation data:
validate = Map[Take[#, {2, 31}] &, Drop[test, 1]];
Predictions made on the unlabeled data:
result /@ RandomSample[validate, 5]
`{"b", "b", "s", "b", "b"}`
Sample from the labeled data and compute classifier statistics:
cm = ClassifierMeasurements[result, RandomSample[vdata, 1000]]
![enter image description here][7]
cm["Accuracy"]
`0.846`
Plot the confusion matrix:
cm["ConfusionMatrixPlot"]
![enter image description here][8]
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ATLASEXP_image.png&userId=20103
[2]: https://higgsml.lal.in2p3.fr/files/2014/04/documentation_v1.8.pdf
[3]: https://www.kaggle.com/c/higgs-boson
[4]: http://opendata.cern.ch/about/ATLAS
[5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=5393ty56yetjhw4.png&userId=20103
[6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=5256567urytere.png&userId=20103
[7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=rtyee567rutyrd.png&userId=20103
[8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ghf56utyehw65eyhrw.png&userId=20103Arnoud Buzing2017-02-17T14:16:25ZSimulating the Universe (an alternative approach)
http://community.wolfram.com/groups/-/m/t/982238
This is the Community Post presenting the 2016 Wolfram Summer School Project of József Konczer, a Hungarian PhD student of theoretical physics, assisted by Todd Rowland.
The project notebook with the code is attached to this post.
## Fundamental theories in Physics ##
The Holy Grail of theoretical physics would be a theory, which could describe all known phenomena in every situation. This would be called the Theory of Everything ([ToE][1]). In present times, all approaches to a ToE candidate—like String/M theory, Loop quantum gravity ([and many others][2])—incorporate quantum mechanics from the beginning. This approach served surely very useful effective theories like the Standard Model itself, however this approach does not help to find an underlying deterministic theory from which quantum effects would emerge, like Einstein dreamed.
The standard argument for the unavoidability of quantum mechanics and uncertainty is the result of the [Bohr–Einstein debates][3] which was "won" by Bohr because the [EPR paradox][4] was tested by [measurements][5] (there is an ongoing test as well called the [Big Bell test][6]) and the result excludes local hidden variables. (The arguments can be found in details [here][7] and [here][8]). It has to be emphasized that in these arguments locality is a key assumption.
----------
One can ask, what kind of nonlocal theory can be constructed, which still have predictive power, and is not based of conspiracy of Nature? There are only a few researchers who post this question openly, one is them is Gerard ’t Hooft who published recently a [book][9] based on collected [papers][10]. His approach is conservative (from main stream point of view), and mainly suggests, that if one quantize time, then in same basis the unitary time evolution considered in quantum mechanics become a permutation operator between special basis elements or "beable states". However not every time evolution has this property and typically the interacting theories fail to fulfill the requirements. A more bold, however much less understood theory (or framework) is what Stephen Wolfram described in [NKS][11]. The brief summary of his ideas can be found in this [blog post][12]. The main idea here, is to find a simple data structure, for instance a sparse graph, a simple discrete dynamics governed by a replacement rule, an interpretation for this cellular automaton (CA) and then investigate if we can observe similar phenomena what we see in our Universe.
## Hints pointing toward the CA description ##
This is a speculative and highly subjective argumentation, however I think this blog post is an appropriate place to articulate my motives and do not stick to the objective style of research papers.
So first of all without going too deep into metaphysics I don't want to state things about Nature it self, I only talk about our description of it.
The first successful and highly useful description of Nature was Newtons description, which heavily used the idea of continuity of space and time. This idea proved to be useful in description of solids, liquids and gases as well. However some ideas became so useful and popular, that we forgot that all of them is only our description and not Nature its self. Quantum effects and effects related to relativity reminded us, that under non standard circumstances old descriptions can fail. As I see, quantum effects have two message for us. The first is that quantities could be and should be described by discrete variables, and secondly that under a certain level systems can not be observed without disturbance. If we take into account that space and time even as we observe them are influenced by these quantized quantities, it is straightforward to deduce, that space and time should be quantized as well.
Before these findings in physics probability theory was developed. First it was used to analyze gambling situations where one don't know every information about the system. From this point of view it is clearly a strategy to manage our ignorance toward some details in a deterministic situation. However after some point physicists started to use probabilities as they were part of the phenomena, and not only our clever way to make inference from systems where we do not know every detail. Many physicists—including myself—where educated in the spirit of frequentist [interpretation of probability theory][13], which is useful in some cases but as I think, prevents some questions to ask. I think this promotion of probability to an objective property contributed to the interpretation of quantum mechanics as well. As Jayns wrote in his [book][14]:
> In current quantum theory, probabilities express our own ignorance due to our failure
to search for the real causes of physical phenomena; and, worse, our failure even to think
seriously about the problem. This ignorance may be unavoidable in practice, but in our
present state of knowledge we do not know whether it is unavoidable in principle; the
‘central dogma’ simply asserts this, and draws the conclusion that belief in causes, and
searching for them, is philosophically naive. If everybody accepted this and abided by it,
no further advances in understanding of physical law would ever be made; indeed, no such
advance has been made since the 1927 Solvay Congress in which this mentality became
solidified into physics. But it seems to us that this attitude places a premium on stupidity;
to lack the ingenuity to think of a rational physical explanation is to support the supernatural
view.
However even if one thinks, that theories incorporating quantum mechanics are "only" effective theories, probably we can get intuitions from them. There is a [recent result][15] from [AdS/CFT][16] correspondence as an example for the [EPR=ER][17] conjecture. And a connecting [paper][18] of Leonard Susskind, concluding that:
> What all of this suggests to me, and what I want to suggest to you, is that quantum mechanics and gravity are far more tightly related than we (or at least I) had ever imagined. The essential nonlocalities of quantum mechanics (the need for instantaneous communication in order to classically simulate entanglement) parallels the nonlocal potentialities of general relativity: ER=EPR.
The cited papers state, that spacetime structure can be understood as a net of entanglements, however maybe the statement can be reversed, and say that the phenomena of entanglement can be described by a nonlocal spacetime structure.
Among the mentioned hints, the existing theoretical constructions can help to find an appropriate interpretations as well. For example it can happen, that to describe our seemingly 3 dimensional space one has to describe space with higher effective dimensionality and interpret the entangled parts not just as connected regions, but as global structures in the extra dimensions.
After taking hints from existing theoretical constructions one can investigate, what kind of phenomena can appear in simple CA-s which mimic some parts of Nature.
Perhaps the most well known CA is Conway's [Game of Life][19] this is a 2D Cellular automaton where localized objects (called [spaceships][20] or gliders) can propagate, and can interact with each other. This behavior can remind us to particles, however the built in rectangular structure is reflected on the properties of spaceships, and there are no nonlocal connections between these object because of the locality of the rule.
Both problems can be solved, if one tries to construct a CA without built in topology. (This construction will be described in detail.)
Another nice feature of special CA-s called substitution systems, is that for a structure living in the automaton, can not observe the absolute number of steps, or other structures beside him, only the causal net of implemented changes can be recognized from inside. This feature unites the relative space and time for observers or structures inside the system. It can remind us to causal network description of General Relativity.
A third hint from CA point of view is the typical appearance of complex behavior, which can lead to an effective probabilistic description of the system with a higher symmetry what the framework originally allowed. (For example CA description of flows) From disorder new effective order can emerge possible with higher symmetry.
The conjectured computational irreducibility of CA would replace the promised "free will" possibility of quantum mechanics with a different but in some sense similar concept. In this framework the faith of the Universe would be determined, but even an observer outside the system—God if one wishes—could not know the consequences only by letting the simulation run up to the desired point.
Furthermore a multiway CA dynamics is compatible with the many world interpretation of quantum mechanics, with the advantage, that the splitting points of histories are not observer dependent. In this framework the overall dynamics is deterministic, however structures living always on one branch of the evolution will witness an unavoidable true random behavior from the inside point of view.
## Nature and our understanding of it ##
Of course it would be an arrogant attitude to force Nature to fulfill our philosophical expectations, however one can imagine how our description of it can change during time.
There are several situations what one can imagine:
- There is a deterministic description which is valid in any situation (which can appear in our Universe)
- This can be totally discreet
- Or it can be continuous partially or in whole
- It can be, that after some point a truly random (or at least appearing to us) mechanism will appear, which can not be unfolded
- Or it can happen that construction of laws to describe Nature will never come to an end, and our understanding of reality will based on infinite set of possibly deterministic rules.
- And of course it can happen that something unexpected will turn out.
Without favoring any of the listed cases above, my main point is that the very first situation, namely that our universe can be described as a deterministic discrete system is not totally excluded. And the most natural way to understand it can be a CA description.
## CA description candidate for our Universe ##
To have a CA description, one has to choose a data structure, a dynamics and an interpretation. (It has to be pointed out, that any CA can be simulated on another Turing complete CA with different interpretation of the states. Because of that any CA description is highly non unique. However one can try to choose a description which has the "simplest" interpretation.)
For a fundamental CA description one can choose simple graphs as data structure. This seems as a natural choice, because of its simplicity and because of its non fixed topology.
To have a chance to describe deterministic dynamics on this data structure, we even restrict the degree of nodes on the graph. One can try to find the threshold of complexity of the CA, and it seams that cubic graphs can already produce complicated enough structures. So one can set the data structure to a simple cubic graph.
The next step is to define an appropriate dynamics on this data structure. A natural approach is to introduce subgraph replacement rules, which means the following: If one finds a given subgraph pattern $H_1$ in the present graph $G$, then replace it with a compatible new graph $H_2$. It sounds simple, however there are many details which have to be fulfilled to get a dynamics with desired properties.
I mention here two properties of the patterns, which seems to be essential to get a substitution systems which generates a complex behavior and appears completely deterministic from the inside without the specification of the order of replacements in the system.
The first one is a **non overlapping property** of the pattern graph(s) $ H_1 $. This means, that $H_1$ has a special structure such as there is no cubic graph $G$, where two subgraphs can be found which are isomorphic to $H_1$ and have nonzero intersection. The following rule does not fulfill this requirement because there is a cubic graph, where two intersecting copies of $H_1$ can be found
![overlapping rule][21]
![Interesting patterns][22]
The second requirement is **non triviality**, which gives a constraint for $H_2$. In this case we wish to have $H_2$, that there exists a cubic graph $G$, which contains a subgraph $H_1$, where after the replacement $H_1 \rightarrow H_2$ there can be found a new pattern $H_1$, which intersects with $H_2$ but has parts outside $H_2$ as well. (Without this property only self similar or frozen (where there is no more pattern which can be changed) graphs can be generated from finite initial graphs.) The pictures show visually the requirement:
![enter image description here][23]
![enter image description here][24]
After setting some rule, which fulfill these requirements, we have to find an initial graph, apply the rule many times and find an interpretation for the result. It has to be pointed out, that the actual graph structure at a given step can not be observed from an inside point of view. What an inner observer, or a structure can explore is the causal structure which is generated by the replacements. (For details see [NKS chapter 9, section 13][25] )
This is similar to the [causal set program][26].
So a natural way of interpretation of the emerging causal net is that it is a discretization of some kind of spacetime. And local propagating disturbances relative to the overall average structure are particle like excitations, which can have nonlocal connections relative to the average large scale structure. However from AdS/CFT insights it can happen, that we have to interpret particles for example as global structures in a higher dimensional bulk spacetime, which have ends on a boundary-like smaller dimensional surface.
## My contribution to the project ##
During the 3 weeks of 2016 Wolfram summer school I set a framework where the steps of a substitution are precisely defined, and in which the substitutions can be effectively performed even for relatively big graphs. Furthermore I tested a numerical approach to measure the effective dimensionality of the emergent graph structure after sufficiently many steps.
Unfortunately I could not test this framework with rules which could give complex, deterministic behavior, so I could benchmark this machinery on a simple, point to triangle rule, which gives a fractal-like structure. If we interpret this graph as space, then this simple dynamics results a $D=\log(3)/\log(2)=1.58$ dimensional fractal space.
Here is a graph of the generated fractal Universe after 100 steps, started from a tetrahedron:
![Generated fractal Universe after 100 steps, started from a tetrahedron][27]
And the neighborhood structure of this space:
![Local structure in the fractal Universe][28]
## Further directions ##
This project to find deterministic CA description for our Universe is in its infant stage. The framework is more or less set, but it needs tremendous work to investigate possible dynamics and analyze the results of simulations.
An outline of a huge project would be the following:
- List the possible rules, which fulfill the non overlapping and non trivial conditions
- Investigate their long term behavior starting from simple initial graphs
- Find quantities and a method of their measurement which can be determined from generated causal graphs
- Find fixed points of the dynamics which preserve long scale dimensionality and possibly other quantities
- List and investigate local disturbances near these fixed points
- After setting an interpretation analyze the particle-like structures (gliders of this dynamics)
- Develop an effective field theory which can describe an effective behavior of the system near to the fixed points
- Match these field theories with the Standard Model of particle physics
- Find out new predictions of the derived effective field theories, which can be tested by measurements
## Conclusion ##
In my project I could set a framework and show a trivial example for a deterministic graph evolution model.
During the summer school I was not fortunate enough to find dynamics which produce complex behavior, however to find an appropriate rule seems reachable in the near future. Hopefully a dynamics producing complex topology would be interesting enough to inspire much more people and after some point a serious investigation of the field could be started.
I think personally, that proving or even disproving that this framework to describe Nature can be worked out is an extremely interesting challenge and deserves further theoretical research.
In the end I would like to thank my mentor Todd Rowland, and the whole Wolfram summer school team for the organization and I really hope that there will be a continuation of this project.
Last but not least I thank for all the summer school participants for great discussions and a lifelong experience!
![enter image description here][29]
----------
## Further comments ##
I try to collect here some useful comments of my friends and collegues, who kindly read my post, and responded in person:
There is a concept named [Digital physics][30] which has a much longer history what I suggested, and probably the earliest pioneer of the field was Konrad Zuse. Fortunatelly his thesis—[Calculating Space][31] or “Rechnender Raum”—is now translated into English and has a modern, LaTeX typesetting.
Beside NKS there is another relevant book, which can serve as an extended list of references and valuable material in its own, written by Andrew Ilachinski with the title [Cellular Automata A Discrete Universe][32].
There is an ongoing "mini revolution" in the description of AdS/CFT based on [Tensor Networks][33]. The original paper on the topic can be found [here][34].
[1]: https://en.wikipedia.org/wiki/Theory_of_everything
[2]: https://www.quantamagazine.org/20150803-physics-theories-map/
[3]: https://en.wikipedia.org/wiki/Bohr%E2%80%93Einstein_debates
[4]: https://en.wikipedia.org/wiki/EPR_paradox
[5]: https://arxiv.org/abs/1508.05949
[6]: http://thebigbelltest.org/#/science?l=EN
[7]: http://www.cambridge.org/catalogue/catalogue.asp?isbn=9780521818629
[8]: http://www.springer.com/in/book/9783662137352
[9]: http://www.springer.com/us/book/9783319412849
[10]: https://arxiv.org/abs/1405.1548
[11]: http://www.wolframscience.com/
[12]: http://blog.stephenwolfram.com/2015/12/what-is-spacetime-really/
[13]: https://plato.stanford.edu/entries/probability-interpret/
[14]: http://www.cambridge.org/catalogue/catalogue.asp?isbn=0521592712
[15]: http://www.nature.com/news/the-quantum-source-of-space-time-1.18797
[16]: https://en.wikipedia.org/wiki/AdS/CFT_correspondence
[17]: https://en.wikipedia.org/wiki/ER=EPR
[18]: https://arxiv.org/abs/1604.02589
[19]: https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life
[20]: http://conwaylife.com/wiki/Category:Spaceships
[21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=H1H2.png&userId=981213
[22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=GH1H1.png&userId=981213
[23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=H1H2_2.png&userId=981213
[24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=GG.png&userId=981213
[25]: http://www.wolframscience.com/nksonline/section-9.13
[26]: https://en.wikipedia.org/wiki/Causal_sets
[27]: http://community.wolfram.com//c/portal/getImageAttachment?filename=PresentationTemplate_KJ_2.png&userId=981213
[28]: http://community.wolfram.com//c/portal/getImageAttachment?filename=PresentationTemplate_KJ_3.png&userId=981213
[29]: http://community.wolfram.com//c/portal/getImageAttachment?filename=vladstudio_higgs_boson_fluo_800x600_signed.jpg&userId=981213
[30]: https://en.wikipedia.org/wiki/Digital_physics
[31]: http://www.mathrix.org/zenil/ZuseCalculatingSpace-GermanZenil.pdf
[32]: http://www.worldscientific.com/worldscibooks/10.1142/4702
[33]: https://arxiv.org/abs/1306.2164
[34]: https://arxiv.org/abs/0905.1317Jozsef Konczer2016-12-16T07:22:29ZCoffee optimization, how to get your cup of joe just right
http://community.wolfram.com/groups/-/m/t/1024265
## Introduction ##
I take my coffee black, so I had no idea that there was a large controversy behind when you should add your milk to your coffee. [@Gary Bass][at0] however highlighted us to this with [his question][1] in the community. Apparently, the timing of the added milk is critical. If you add the milk directly the coffee will retain its temperature for a longer time, which is perfect if you plan on drinking later. If you are short on time however, and want to save your throat from scalding hot coffee, you might want to save the milk for just before you are about to drink it.
Obviously, this is something that cannot be taken lightly and some serious simulation is required. I compiled the information from that thread here, for anyone looking to perfect their morning routine. The post will include an explanation for how the model was created. I also however attached the actual model so if you are just looking for the simulation, check out the summary. Hopefully however, you also gain some insights into how to model events involving states in [SystemModeler][2]:
## Adding Milk to Coffee ##
In SystemModeler, there are at least two approaches you could take when adding the milk to the coffee. Either you could have everything collected into a single component that has a event in it corresponding to the addition of the milk, or you could have a separate component that specifies the addition of milk as a flow over time. I explored both those scenarios in the attached model.
Approach 1, with a discrete event in the coffee involves creating a copy of the [HeatCapacitor][3] component and adding some parameters that separates the heat capacity of the milk, the amount of milk added, when it is added, etc. As you say, the mixed Cp is unknown. A naïve initial approach could be to just add the two heat capacities together. If C is the total heat capacity of the coffee, with or without milk, you could add an equation that says:
C = if time > AddTime then Ccoffee * Vcoffee * 1000 + Cmilk * Vmilk * 1000 else Ccoffee * Vcoffee * 1000;
The coefficients are just there to convert the different units.
The temperature is a bit more difficult, since it varies continuously over time it is state so and can't be as easily changed as the capacity (which changes values only at one discrete interval).
What you have to do with states is using the [reinit(var,newValue)][4] function to reinitialize variable var, to the new value newValue. If you mix fluids together, the new temperature is the new total enthalpy divided by the new heat capacity:
t = (m1 c1 t1 + m2 c2 t2 + ... + mn cn tn) / (m1 c1 + m2 c2 + ... + mn cn)
(from [Engineering Toolbox][5])
In Modelica, we could reinitialize the temperature when the simulation time exceeds the time when the milk should be added, using the following:
when time > AddTime then
reinit(T, (Ccoffee * Vcoffee * 1000 * T + Cmilk * Vmilk * 1000 * MilkTemperature) / C);
end when;
Adding the coffee component and connecting it to a [ThermalConductor][6] component (to represent the cup) and connecting that in turn to an [FixedTemperature][7] component (to represent the room temperature) results in a fairly compact model:
![Diagram with coffee component][8]
If milk is added after 300 seconds, it produces the following simulation:
![Simulation with coffee component][9]
Approach 2 is having a short flow of milk, instead of a instantaneous addition of milk. The benefit of this is you could create your own addition strategy. For example, you could add half of the milk at the beginning, and half after 300 seconds. Or any arbitrary strategy. For now, I focused on doing it as a 1 second pulse.
An input is added to the coffee, corresponding to the flow of milk. The volume of the milk in the coffee is no longer a parameter but increases with the flow:
der(Vmilk) = u;
And the heat capacity increases with the milk volume:
C = Ccoffee * Vcoffee * 1000 + Cmilk * Vmilk * 1000;
Adding milk will increase the enthalpy in the system, but the increased heat capacity will still cause a drop in temperature:
T = H/C;
der(H) = port.Q_flow + Cmilk * 1000 * u * MilkTemperature;
With H being the enthalpy.
The milk component is simply a pulse from [Pulse][10] that has some additional parameters.
![milk addition component diagram][11]
Everything taken together, we now have an additional component in the coffee cooling model:
![diagram with coffee and milk components][12]
As it should, this approach gives a similar plot as the first one. The only difference is that the milk is added over a duration of 1 second. As the duration approaches zero, the two approaches would converge.
![simulation with coffee and milk][13]
You could use this approach to fit parameters, using the methodology from the [electric kettle][14] example.
## Other Cooling Processes ##
In the model above, we had a very naïve cooling process for our coffee. We assumed it could be described by Newtons law of cooling (which the heat conduction component is based on). In the [original thread][15] a paper is linked that goes into detail on how you might expand the a coffee model to include some other forms of cooling.
I will here use a [HeatCapacitor][16] component instead of the coffee component to simplify things, but the two should interchangeable. The experiment numbers are in reference to the attached article.
**Experiment 1**
Experiment 1 can be described using standard components from the Modelica.Thermal.HeatTransfer package. The pot will be HeatCapacitor component, the ambient temperature will be modeled using FixedTemperature and the convection is modeled using a ThermalConductor, which follows Newtons law of cooling.
![experiment 1 diagram][17]
The G parameter in the ThermalConductor is equivalent to the k parameter they use. From what I could tell, the paper did not include any measurement of the heat capacitance or ambient temperature so I went with 3 dl of water and 20 degrees Celsius. However, both of these would probably need to be higher to fit their experimental data.
**Experiment 2**
To create experiment 2 I first duplicated experiment 1 by selecting it and pressing Ctrl+D, you can also right click and select Duplicate. Experiment 2 requires a component like the ThermalConductor but one that has an exponent that causes nonlinear behaviour in the heat flow. No such component exist in the Modelica Standard Library, but we can easily create one. I created a new component to be used in experiment 2 by dragging the normal ThermalConductor into Experiment2.
![cope class][18]
And gave it a new name, "ArbitraryExponentConductor"
Now I had to modify it to use the exponent. After opening the new component I first added a new parameter by right clicking the parameter view and and selecting Insert > Parameter
![Adding new parameter to model][19]
I used the name x as in the paper and used type Real.
![new parameter window][20]
Now I had to modify the equations so I went into the Modelica Text View (Ctrl+3) and changed the line:
Q_flow = G * dT;
to
Q_flow = G * dT ^ x;
dT corresponds to the temperature difference (tc-ts) in the paper.
Going back into Experiment 2, I changed the normal ThermalConductor by right clicking it and selected Change Type. In the dialog, I gave the name of the new type (CofeeCooling.Experiment2.ArbitraryExponentConductor). You can also drag the component from the component browser directly into the field.
![change model quickly][21]
or of course, delete the component, drag the new one in and make new connections.
**Experiment 3**
For experiment 3, you need to add a some more stuff. Start by douplicating experiment 1. Connect the ThermalConductor to a new HeatCapacitor instead of the FixtedTemperature. That heat capacitor will be the pot, while the original one will be the coffee. The first ThermalConductor then represents equation 1 in the paper, transfer of heat from coffee to the pot. Add another ThermalConductor and connect it between the pot and the FixedTemperature to represent equation 5. Also add two BodyRadiation components and connect them from each capacitor to the FixedTemperature. These will represent all the radition effects described. They are bidirectional so they represent two equations each (3,4 and 6,7). For evaporation, I created a custom component which is described by the equation
port.Q_flow = k * port.T;
Where k is the product of the P, l and v parameters described in the paper. You could add individual parameters for each of them instead, as described in the text for experiment 2.
Connect the evaporation to the coffee capacitor.
![experiment 3 model diagram][22]
The 4:th experiment is much like the the 3:d one. I modified the evaporation component to have the equation
port.Q_flow = k * port.T ^ z;
## Summary & Simulation ##
Okey, so that was the *how*. Now we want to use this model to draw conclusions. I'll use the simplest model here and encourage you to try out the more advanced models yourselves.
Say we want to drink our coffee in 2 minutes, starting from 80°C. Everyone knows that the optimal coffee drinking temperature is 72.34°C. When should we add our milk to get there in 2 minutes?
We can do a parametric simulation in Mathematica to try out two different timings:
addTimes = {0, 110};
sim = WSMSimulate["CoffeeAndMilk.Scenarios.Approach1", WSMParameterValues -> {"AddTime" -> addTimes}];
In the plot, I will add a point that is the optimum temperature at time = 120s. I'll also use a trick to get some nice legends to better understand which curve corresponds to which:
WSMPlot[sim, "coffee.T",
PlotRange -> {{60, 180}, {70, 80}},
Epilog -> Point[{120, 72.34}],
PlotLegends -> Map["Add time = " <> ToString[#] &, addTimes]
]
This produces the following plot:
![plot 1 with 0 and 110][23]
So close. But we can't give up just now. Let us adjust the timing a bit and add the milk right before we want to drink the coffee:
![plot 2 with 0 and 120][24]
That just about does it I'd say.
[at0]: http://community.wolfram.com/web/bassgarys
[1]: http://community.wolfram.com/groups/-/m/t/1021383
[2]: http://www.wolfram.com/system-modeler/
[3]: https://reference.wolfram.com/system-modeler/libraries/Modelica/Modelica.Thermal.HeatTransfer.Components.HeatCapacitor.html
[4]: https://reference.wolfram.com/system-modeler/libraries/ModelicaReference/ModelicaReference.Operators.%27reinit%28%29%27.html
[5]: http://www.engineeringtoolbox.com/mixing-fluids-temperature-mass-d_1785.html
[6]: https://reference.wolfram.com/system-modeler/libraries/Modelica/Modelica.Thermal.HeatTransfer.Components.ThermalConductor.html
[7]: https://reference.wolfram.com/system-modeler/libraries/Modelica/Modelica.Thermal.HeatTransfer.Sources.FixedTemperature.html
[8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=9088mod1.png&userId=554806
[9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=mod1sim.png&userId=554806
[10]: https://reference.wolfram.com/system-modeler/libraries/Modelica/Modelica.Blocks.Sources.Pulse.html
[11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=milk.png&userId=554806
[12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=mode2.png&userId=554806
[13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=mod2sim.png&userId=554806
[14]: https://www.wolfram.com/system-modeler/examples/consumer-products/electric-kettle-fluid-heat-transfer.html
[15]: http://community.wolfram.com/groups/-/m/t/1021383
[16]: https://reference.wolfram.com/system-modeler/libraries/Modelica/Modelica.Thermal.HeatTransfer.Components.HeatCapacitor.html
[17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=4316mod3.png&userId=554806
[18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Copy.png&userId=554806
[19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Addnewparameter.png&userId=554806
[20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=newparameter.png&userId=554806
[21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=quickchange.png&userId=554806
[22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=mod3%281%29.png&userId=554806
[23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=test1plot.png&userId=554806
[24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=test2plot.png&userId=554806Patrik Ekenberg2017-03-02T17:16:24ZMetaprogramming in Wolfram Language
http://community.wolfram.com/groups/-/m/t/1121273
*NOTE: Please see the original version of this post [**HERE**][1]. Cross-posted here per suggestion of [Vitaliy Kaurov][2]*
*Also note: This post has been reposted verbatim, and as such is rather dated. While I believe that it is still mostly accurate, it does not necessarily fully reflect my current views on the subject matter. In particular, a number of newer internal projects have been using metaprogramming techniques in ways not fully reflected here.*
----------
##What this answer is and is not
To avoid some confusion and misunderstanding, let me state right away what is the intended status of this answer.
This answer ***is not***
- A tutorial to the subject
- A systematic, or complete, introduction to the subject
- An authoritative answer putting the final word on the subject
This answer hopefully is
- An (subjective!) overview of various meta-programming techniques in Mathematica, *in the way they are known to me*. I want to explicitly state that I am ***not*** trying to convey any kind of the "common wisdom" here, since the answer is largely based on my own experiences, and I have not seen an overwhelming number of meta-programming examples in Mathematica-related resources I had a chance to get acquainted with (so I may have no idea what the common wisdom is :)).
- A collection of (hopefully relevant) links with some minimal explanations, which would allow the reader to see some examples and applications of metaprogramming in Mathematica, or at least examples of what I consider meta-programming in Mathematica.
- A possible stub for some future answers, so that this larger one could be eventually rewritten and/or split into more focused and narrow ones, as the interest towards some particular forms of metaprogramming in Mathematica is being developed in our community.
##Preamble
Ok, let me give it a shot. I'll start by claiming that Mathematica is very well suited for meta-programming, and one can write much more powerful programs in Mathematica by utilizing it. However, while it *allows* for very interesting and powerful meta-programming techniques, it does not IMO provide a convenient layer of tools to make these techniques more standard and effortless. Particularly painful is the evaluation control (preventing pieces of code from premature evaluation), because of the *absence* of the true quotation mechanism (here I will disagree with some other answers), the infinite evaluation model of Mathematica, and a quite complex core evaluator.
##Enumerating some meta-programming techniques
There are several forms of meta-programming, so let me give a partial list first, and discuss afterwards
- Introspection-based metaprogramming
- Reflection-based metaprogramming (like in say, Java)
- Run-time code generation
- Macros (like in Lisp)
- DSL (domain-specific-language) creation
- ...?
In addition to these, Mathematica has its own meta-programming devices, such as rule-based metaprogramming and the `Block`-related techniques.
##Introspection
Mathematica is IMO very strong here. There are a couple of reasons for this:
- Homoiconic language (programs written in own data structures - Mathematica expressions. This is code-as-data paradigm, like Lisp which uses lists for this)
- One can access global definitions for symbols stored in `OwnValues`, `DownValues`, `SubValues`, `UpVaulues`, etc, and various other global properties, programmatically.
- Rule-based destructuring techniques (using `Cases` etc) seriously simplify many introspection-related operations
- Mathematica code is "over-transparent" - even pure functions are expressions, available to introspection and destructuring, rather than black boxes. This has its downsides (for example, making a functional abstraction leaky in Mathematica, see the end of [this answer][3]), but it also allows for things like `withGlobalFunctions` macro from [this answer][4], where global function definitions are expanded inside pure functions (that macro also illustrates other meta-programming techniques).
###Automatic dependency tracking
I will give a single simple explicit example of what I mean by introspection here, and supply some references to more involved cases. The following line of code gives all the symbols used to build a given expression `expr`, kept unevaluated:
Cases[Unevaluated[expr],s_Symbol:>HoldComplete[s],{0,Infinity},Heads->True]
Note that this will work for *any* Mathematica expression, including a piece of (perhaps unevaluated) Mathematica code.
A good illustration of introspection-based meta-programming is the symbol dependency analysis. I gave it a shot [here][5], where I fully used all of the above-mentioned features (homoiconic language, low-level access to symbol's properties, rule-based destructuring). A simpler but practical application of dependency analysis can be found e.g. in the `getDependencies` function from [this answer][6], where I do use the dependencies to dynamically construct a set of symbols which are encapsulated (not easily available on the top-level) but whose definitions must be saved during the serialization of the list object being constructed.
###Working around some language limitations
Sometimes, introspection-based metaprogramming can be also used to go around certain limitations of the language, or to make the language constructs behave in the way you want while minimally affecting them. Some examples off the top of my head: [changing the default behavior of `SaveDefinitions` option for `Manipulate`][7], [making patterns to match only children of certain elements][8], and also two functions from [this answer][9]: a function `casesShielded` which implements a version of `Cases` that shields certain sub-expressions (matching specific pattern) from the pattern-matcher. and a (rather hacky) function `myCases` which implements a modified depth-first search, where the head is inspected before the elements (this is not what is happening in standard `Cases`, which sometimes has unwanted consequences). Yet another example here is the tiny framework I wrote to deal with the leaks of standard lexical scoping mechanism in Mathematica, which can be found [here][10].
###Summary
To conclude this section, I think that introspection-based meta-programming is a very useful and powerful technique in Mathematica, and the one that is relatively easy to implement without engaging in a fight with the system. I am also positive that it is possible to factor out the most useful introspection primitives and have a higher-level introspection-based metaprogramming library, and hope such a library will emerge soon.
##Reflection - based metaprogramming
This may probably be considered a subset of the introspection-based metaprogramming, but it is particularly powerful for languages which impose more rigid rules on how code is written, particularly OO languages (Java for example). This uniform and rigid structure (e.g. all code is in classes, etc) allows for automatic querying of, for example, the methods called on the object, etc. Mathematica per se is not particularly powerful here, because "too many ways of doing things" are allowed for this to be effective, but one can surely write frameworks and / or DSLs in Mathematica which would benefit from this meta-programming style.
##Run-time code generation
This type of meta-programming can be used relatively easily and brings a lot to the table in Mathematica.
###Automation and adding convenient syntax
I will give a small example from [this answer][11], where an ability to generate a pure function (closure) at run-time allows us to easily define a version of SQL `select` with a more friendly Mathematica syntax, and based on the in-memory Mathematica representation of an SQL table as a nested list:
ClearAll[select, where];
SetAttributes[where, HoldAll];
select[table : {colNames_List, rows__List}, where[condition_]] :=
With[{selF = Apply[Function, Hold[condition] /.
Dispatch[Thread[colNames -> Thread[Slot[Range[Length[colNames]]]]]]]},
Select[{rows}, selF @@ # &]];
Please see the aforementioned answer for examples of use. Further developments of these ideas (also based on meta-programming) can be found in [this][12] and [this][13] discussions.
###Making JIT-compiled functions, and using `Compile` in more flexible ways
An important class of applications of run-time code-generation is in improving the flexibility of `Compile`. A simple example would be to create a JIT-compiled version of `Select`, which would compile `Select` with a custom predicate:
ClearAll[selectJIT];
selectJIT[pred_, listType_] :=
selectJIT[pred, Verbatim[listType]] =
Block[{lst},
With[{decl = {Prepend[listType, lst]}},
Compile @@
Hold[decl, Select[lst, pred], CompilationTarget -> "C",
RuntimeOptions -> "Speed"]]];
This function actually illustrates several techniques, but let me first show how it is used:
test = RandomInteger[{-25, 25}, {10^6, 2}];
selectJIT[#[[2]] > 0 &, {_Integer, 2}][test] // Short // AbsoluteTiming
selectJIT[#[[2]] > 0 &, {_Integer, 2}][test] // Short // AbsoluteTiming
(*
==> {0.4707032,{{-6,9},{-5,23},{-4,4},{13,3},{-5,7},{19,22},<<489909>>,{11,25},{-6,5},
{-24,1},{-25,18},{9,19},{13,24}}}
==> {0.1250000,{{-6,9},{-5,23},{-4,4},{13,3},{-5,7},{19,22},<<489909>>,{11,25},{-6,5},
{-24,1},{-25,18},{9,19},{13,24}}}
*)
The second time it was several times faster because the compiled function was memoized. But even including the compilation time, it beats the standard `Select` here:
Select[test,#[[2]]>0&]//Short//AbsoluteTiming
(*
==> {1.6269531,{{-6,9},{-5,23},{-4,4},{13,3},{-5,7},{19,22},<<489909>>,{11,25},{-6,5},
{-24,1},{-25,18},{9,19},{13,24}}}
*)
The other techniques illustrated here are the use of constructs like `Compile@@Hold[...]` to fool the variable-renaming scheme (see e.g. [this answer][14] for a detailed explanation), and the use of `With` and replacement rules (pattern-based definitions) as a code-injecting device (this technique is used very commonly). Another example of a very similar nature is [here][15], and yet another, very elegant example is [here][16].
###Custom assignment operators and automatic generation of function's definitions
Another class of run-time code-generation techniques (which is somewhat closer to macros in spirit) is to use custom assignment operators, so that you can generate rather complex or large (possibly boilerplate) code from relatively simple specifications. Applications range from relatively simple cases of adding some convenience/ syntactic sugar, such as e.g. [here][17] (where we define a custom assignment operator to allow us to use option names directly in code), to somewhat more complex cases like making replacements in definitions at the definition-time, as say in the function `lex` from [this answer][18] (see also the code for a `LetL` macro below), to quite sophisticated generation of boilerplate code, happening e.g. in JLink behind the scenes (which, for JLink, is a big deal, because *this* (plus of course the great design of JLink and Java reflection) is the reason why JLink is so much easier to use than Mathlink).
###Automating error-handling and generating boilerplate code
Yet another use for run-time code generation (similar to the previous) is to automate error-handling. I discussed one approach to that [here][19], but it does not have to stop there - one can go much further in factoring out (and auto-generating) the boilerplate code from the essential code.
###A digression: one general problem with various meta-programming techniques in Mathematica
The problem with this and previous classes of use cases however is the lack of composition: you can not generally define several custom assignment operators and be sure that they will always work correctly in combinations. To do this, one has to write a framework, which would handle composition. While this is possible to do, the development effort can rarely be justified for simple projects. Having a general library for this would be great, provided that this is at all possible. In fact, I will argue that the lack of composibility ("out of the box") is plaguing many potentially great meta-programming techniques in Mathematica, particularly macros.
Note that I don't consider this being a fundamental core language-level problem, since the relevant libraries / frameworks can surely be written. I view it more as a consequence of the extreme generality of Mathematica and it being in a transition from a niche scientific language to a general-purpose one (in terms of its typical uses, not just capabilities), so I am sure this problem has a solution and will eventually be solved.
###Proper (macro-like) run-time generation of Mathematica code
A final use case for the run-time code generation I want to mention is, well, run-time Mathematica code generation. This is also similar to macros (as they are understood in Lisp) in spirit, in fact probably the closest to them from all techniques I am describing here. One relatively simple example I discuss [here][20], and a similar approach is described [here][21]. A more complex case involving generation of entire packages I used for the real-time cell-based code highlighter described [here][22]. There are also more sophisticated techniques of run-time Mathematica code generation - one of which (in a very oversimplified form) I described [here][23]
###Summary
To summarize this section, I view run-time code generation as another meta-programming technique which is absolutely central to make non-trivial things with Mathematica.
##Macros
First, what I mean by macros is probably not what is commonly understood by macros in other languages. Specifically, by macro in Mathematica I will mean a construct which:
- Manipulates pieces of Mathematica code as data, possibly preventing them from (premature) evaluation
- Expands code at run-time (not "read-time" or "compile-time", which are not so well defined in Mathematica)
###Some simple examples
Here is the simplest macro I know of, which allows one to avoid introducing an intermediate variable in cases when something must be done after the result has been obtained:
SetAttributes[withCodeAfter,HoldRest];
withCodeAfter[before_,after_]:=(after;before)
The point here is that the argument `before` is computed before being passed in the body of `withCodeAfter`, therefore evaluating to the result we want, while the code `after` is being passed unevaluated (due to the `HoldRest` attribute), and so is evaluated already inside the body of `withCodeAfter`. Nevertheless, the returned result is the value of `before`, since it stands at the end.
Even though the above macro is very simple, it illustrates the power of macros, since this kind of code manipulation requires special support from the language and is not present in many languages.
###Tools used for writing macros
The main tools used for writing macros are tools of evaluation control, such as
- `Hold*`- attributes,
- `Evaluate` and `Unevaluated`
- code injection using `With` and / or replacement rules
- Pure functions with `Hold` - attributes
Even in the simple example above, 2 of these tools were used (`Hold`-attribute and replacement rules, the latter hidden a bit by using global replacement rules / definitions). The discussion of the evaluation control constructs proper is outside the scope of the present discussion but a few places you can look at are [here][24] and [here][25]
###Typical classes of macros
Macros can widely range in their purpose. Here are some typical classes
- Making new scoping constructs or environments (very typical use case)
- Used in combination with run-time code generation to inject some unevaluated code
- Used in combination with some dynamic scoping, to execute code in some environments where certain global rules are modified. In this case, the "macro" - part is used to delay the evaluation until the code finds itself in a new environment, so strictly speaking these are rather custom dynamic scoping constructs.
###Examples of new scoping constructs / environments
There are plenty of examples of the first type of macros available in the posts on StackOverlflow and here. One of my favorite macros, which I will reproduce here, is the `LetL` macro which allows consecutive bindings for `With` scoping construct:
ClearAll[LetL];
SetAttributes[LetL, HoldAll];
LetL /: Verbatim[SetDelayed][lhs_, rhs : HoldPattern[LetL[{__}, _]]] :=
Block[{With}, Attributes[With] = {HoldAll};
lhs := Evaluate[rhs]];
LetL[{}, expr_] := expr;
LetL[{head_}, expr_] := With[{head}, expr];
LetL[{head_, tail__}, expr_] :=
Block[{With}, Attributes[With] = {HoldAll};
With[{head}, Evaluate[LetL[{tail}, expr]]]];
What it does is to expand a single declaration like `LetL[{a=1,b=a+1,c = a+b},a+b+c]` into a nested `With` at run-time, and it also works for function definitions. I described in more fully [here][26] (where some subtleties associated with it are also described), and used it extensively e.g. [here][27]. A very similar example can be found in [this answer][28]. Yet another example I already mentioned - it is the macro `withGlobalFunctions` from [this answer][29], which expands all generically-defined (via patterns) global functions. The last example I want to include here (although it also is relevant for the third use case) is a macro for performing a code cleanup, discussed [here][30], and I particularly like the version by @WReach, which I will reproduce here:
SetAttributes[CleanUp, HoldAll]
CleanUp[expr_, cleanup_] :=
Module[{exprFn, result, abort = False, rethrow = True, seq},
exprFn[] := expr;
result =
CheckAbort[
Catch[Catch[result = exprFn[]; rethrow = False; result], _,
seq[##] &], abort = True];
cleanup;
If[abort, Abort[]];
If[rethrow, Throw[result /. seq -> Sequence]];
result]
It is not fully "bullet-proof", but does a really good job in the majority of cases.
###Examples of run-time code generation / new functionality
Actually, many of the above examples also qualify here. I'll add just one more here (in two variations): the abortable table from [this answer][31] (I will reproduce the final version here):
ClearAll[abortableTableAlt];
SetAttributes[abortableTableAlt, HoldAll];
abortableTableAlt[expr_, iter : {_Symbol, __} ..] :=
Module[{indices, indexedRes, sowTag, depth = Length[Hold[iter]] - 1},
Hold[iter] /. {sym_Symbol, __} :> sym /. Hold[syms__] :> (indices := {syms});
indexedRes = Replace[#, {x_} :> x] &@ Last@Reap[
CheckAbort[Do[Sow[{expr, indices}, sowTag], iter], Null],sowTag];
AbortProtect[
SplitBy[indexedRes, Array[Function[x, #[[2, x]] &], {depth}]][[##,1]] & @@
Table[All, {depth + 1}]
]];
(it accepts the same syntax as `Table`, including the multidimensional case, but returns the partial list of accumulated results in the case of Abort[] - see examples of use in the mentioned answer), and its version for a conditional `Table`, which only adds an element is certain condition is fulfilled - it is described [here][32]. There are of course many other examples in this category.
###Examples of dynamic environments
Dynamic environments can be very useful when you want to modify certain global variables or, which is much less trivial, functions, for a particular piece of code, so that the rest of the system remains unaffected. The typical constructs used to achieve this are `Block` and ``Internal`InheritedBlock``.
The simplest and most familiar dynamic environment is obtained by changing the values of `$RecursionLimit` and / or `$IterationLimit` inside a `Block`. Some examples of use for these are in [my answer][33] in the discussion of tail call optimization in Mathematica. For a more complex example, see [my suggestion][34] for the recent question on convenient string manipulation. Some more examples can be found in my answer to [this question][35]. An example of application of this to profiling can be found [here][36].
Again, there are many more examples, many of which I probably missed here.
###Problems with writing macros in Mathematica
To my mind, the main problems with writing and using macros consistently in Mathematica are these:
- Hard to control evaluation. No *real* quotation mechanism (`Hold` and `HoldComplete` don't count because they create extra wrappers, and `Unevaluated` does not count since it is not permanent ans is stripped during the evaluation)
- Macros as described above are expanded from outside to inside. Coupled with the lack of *real* quotation mechanism, this leads to the absence of true macro composition out of the box. This composition can be achieved, but with some efforts
- The lack of the real compilation stage (The definition-time does not fully count since most definitions are delayed).
To circumvent these issues, one has to apply various techniques, such as
- [Trott - Strzebonski in-place evaluation technique][37] to evaluate parts of held expressions in-place (see also [this answer][38] for some more details on that)
- A technique which I call (for the lack of a better name) "inverse rule-dressing", which exploits the properties of delayed rule substitution (delayed, plus intrusive), to inject some unevaluated code. I used it in the first solution in [this answer][39], in more complex way in the `SavePointers` function in [this answer][40], and in a number of other cases. It has also been used very elegantly in [this answer][41].
- using a custom `Hold`-like wrapper which is first mapped on (possibly all) parts of an expression, and later removed using rules. Two examples of this techniques are [here][42] and [here][43]
- ...
Despite all these techniques being useful, and in total covering most of the needs for macro-writing, the need to use them (often in combinations) and the resulting code complexity shows, to my mind, the serious need for a generic library which would provide simpler means for macro-writing. I would prefer to be able to nest macros and not think about zillion of things that may go wrong because of some unwanted evaluation, but rather about things that really matter (such as variable captures).
###Summary
Macros are another very powerful meta-programming technique. While it *is* possible to write them in Mathematica, it is, as of now, a rather involved undertaking, and composing macros is an even harder task. Because composition in the key, I attribute the fact that macros are not in widespread use in Mathematica programming, to this lack of composition, plus the complexity of writing individual macros. That said, I think this is a very promising direction, and hope that some time soon we will have the tools which would make writing macros a more simple and automatic process.
##DSL creation
I won't say almost anything here, except noting that this is entirely possible in Mathematica, and some nice syntax can be added easily via `UpValues`.
##Final remarks
I think that meta-programming is one of the most important and promising directions in the present and future of Mathematica programming. It is also rather complex, and IMO, largely unexplored in Mathematica still. I hope that this justifies this post being so long.
I tried to summarize various approaches to meta-programming in Mathematica, which I am aware of, and give references to examples of these approaches, so that the reader can look for him/herself. Since meta-programming is a complex topic, I did not attempt to write a tutorial, but rather tried to summarize various experiences of myself and others to produce a kind of a reference.
One may notice that the references are dominated by the code I wrote. One reason for that is that I am a heavy user of meta-programming in Mathematica. Another reason is that everyone remembers own code the most. I have to apologize for not including some other references which did not come to my mind right away. I invite everyone to edit this post and add more references, which I missed.
[1]: https://mathematica.stackexchange.com/a/2352/81
[2]: http://community.wolfram.com/web/vitaliyk
[3]: https://stackoverflow.com/questions/4430998/mathematica-what-is-symbolic-programming/4435720#4435720
[4]: https://mathematica.stackexchange.com/questions/704/functions-vs-patterns/746#746
[5]: https://stackoverflow.com/questions/8867757/has-anyone-written-any-function-to-automatically-build-a-dependency-graph-of-an/8869545#8869545
[6]: https://mathematica.stackexchange.com/questions/36/file-backed-lists-variables-for-handling-large-data/209#209
[7]: https://stackoverflow.com/questions/6579644/savedefinitions-considered-dangerous/6580284#6580284
[8]: https://stackoverflow.com/questions/6451802/pattern-to-match-only-children-of-certain-elements/6453673#6453673
[9]: https://stackoverflow.com/questions/8700934/why-is-cases-so-slow-here-are-there-any-tricks-to-speed-it-up/8701756#8701756
[10]: https://gist.github.com/1683497
[11]: https://stackoverflow.com/questions/4787901/data-table-manipulation-in-mathematica/4788373#4788373
[12]: https://stackoverflow.com/questions/8240943/data-table-manipulation-in-mathematica-step-2
[13]: https://stackoverflow.com/questions/6130276/conditionnal-data-manipulation-in-mathematica
[14]: https://stackoverflow.com/questions/6236458/plot-using-with-versus-plot-using-block-mathematica/6236808#6236808
[15]: https://stackoverflow.com/questions/4973424/in-mathematica-how-do-i-compile-the-function-outer-for-an-arbitrary-number-of/4973603#4973603
[16]: https://stackoverflow.com/questions/8204784/how-to-compile-a-function-that-computes-the-hessian/8210224#8210224
[17]: https://stackoverflow.com/questions/4682742/optional-named-arguments-without-wrapping-them-all-in-optionvalue/4683924#4683924
[18]: https://mathematica.stackexchange.com/questions/1602/resource-management-in-mathematica/1603#1603
[19]: https://stackoverflow.com/questions/6560116/best-practices-in-error-reporting-mathematica/6563886#6563886
[20]: https://stackoverflow.com/questions/6214946/how-to-dynamically-generate-mathematica-code/6215394#6215394
[21]: https://stackoverflow.com/questions/8741671/unevaluated-form-of-ai/8742627#8742627
[22]: https://mathematica.stackexchange.com/questions/1315/customizing-syntax-highlighting-for-private-cell-styles/1320#1320
[23]: https://stackoverflow.com/questions/8741671/unevaluated-form-of-ai/8746584#8746584
[24]: https://stackoverflow.com/questions/4856177/preventing-evaluation-of-mathematica-expressions
[25]: https://stackoverflow.com/questions/1616592/mathematica-unevaluated-vs-defer-vs-hold-vs-holdform-vs-holdallcomplete-vs-etc
[26]: https://stackoverflow.com/questions/5866016/question-on-condition/5869885#5869885
[27]: https://mathematica.stackexchange.com/questions/36/file-backed-lists-variables-for-handling-large-data/209#209
[28]: https://stackoverflow.com/questions/8373526/error-generating-localized-variables-as-constants/8377522#8377522
[29]: https://mathematica.stackexchange.com/questions/704/functions-vs-patterns/746#746
[30]: https://stackoverflow.com/questions/3365794/reliable-clean-up-in-mathematica
[31]: https://stackoverflow.com/questions/6470625/mathematica-table-function/6471024#6471024
[32]: https://stackoverflow.com/questions/6367932/generate-a-list-in-mathematica-with-a-conditional-tested-for-each-element/6368770#6368770
[33]: https://stackoverflow.com/questions/4481301/tail-call-optimization-in-mathematica/4627671#4627671
[34]: https://mathematica.stackexchange.com/questions/344/convenient-string-manipulation/377#377
[35]: https://mathematica.stackexchange.com/questions/1162/alternative-to-overloading-set
[36]: https://mathematica.stackexchange.com/questions/1786/workbench-profile-question/1798#1798
[37]: http://library.wolfram.com/conferences/devconf99/villegas/UnevaluatedExpressions/Links/index_lnk_30.html
[38]: https://stackoverflow.com/questions/6633236/replace-inside-held-expression/6633334#6633334
[39]: https://stackoverflow.com/questions/6234701/how-to-block-symbols-without-evaluating-them/6236264#6236264
[40]: https://stackoverflow.com/questions/6579644/savedefinitions-considered-dangerous/6580284#6580284
[41]: https://mathematica.stackexchange.com/questions/1929/injecting-a-sequence-of-expressions-into-a-held-expression/1937#1937
[42]: https://stackoverflow.com/questions/5747742/uses-for-mapall/5749275#5749275
[43]: https://mathematica.stackexchange.com/questions/2137/truncate-treeform-to-show-only-the-top/2139#2139Leonid Shifrin2017-06-16T12:08:12Z[GIF] Curve running on top a curve
http://community.wolfram.com/groups/-/m/t/1120928
Gif inspired by [beesandbombs][1].
![A][2]
![B][3]
![C][4]
![D][5]
Code:
DynamicModule[{\[Sigma] = (2*Pi)/25, r, f, \[Phi], P},
r[t_] := 8 {Cos[3 t], Sin[2 t]};
f[t_, t0_, \[Phi]_] := Exp[-((t - t0)^2/(2 \[Sigma]^2))] Cos[17 t - \[Phi]] 2;
P[t0_, \[Phi]_, c_] := ParametricPlot[If[\[Phi] == -1, r[t], r[t] + {-1, 1} f[t, t0, \[Phi]] Reverse@Normalize[r'[t]]],
{t, -Pi + t0, Pi + t0}, PlotRange -> 11 {{-1, 1}, {-1, 1}},
Axes -> False, PlotStyle -> Thickness[0.008],
ColorFunction -> If[\[Phi] == -1, Black &, (Blend[{{-1.5, Transparent}, {1, RGBColor@c}}, #] &)], Background -> RGBColor@"#FAF9FA"];
res = Manipulate[
Show@{P[t0, 0, "#F2D12E"], P[t0, Pi/2, "#DE2DAE"],
P[t0, 3 Pi/2, "#55B7B4"], P[t0, -1, Black]}
, {t0, 0, 2 Pi, 0.1}]
]
[1]: https://twitter.com/beesandbombs/status/874565733807730689
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=test.GIF&userId=845022
[3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=7492test.GIF&userId=845022
[4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2493test.GIF&userId=845022
[5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=3558test.GIF&userId=845022Thales Fernandes2017-06-15T16:44:02ZInflation forecasting with SARIMA model
http://community.wolfram.com/groups/-/m/t/1117808
We demonstrate the application of SARIMA model for the UK inflation forecasting. Using long-span historical data, we build robust and well-behaved model that captures structural variations in the UK inflation. Forecasts and predictions are reasonable and found within the model’s bands. Rich Mathematica functionality on time series modeling makes the entire effort much more compelling and faster.
![enter image description here][1]
#Introduction#
Inflation forecasting is one of the most demanding predictions central institutions do to devise their monetary policies. It is therefore not s uprising that this activity attracts a lot of public attention. Good inflation forecasts are not only desirable, but also important for correct policy formulation and future adjustments.
Inflation forecasting is based on models. Various approaches exist and model appropriateness is generally judged by predictive accuracy and evaluation of provided output.
We review the application of **SARIMA** (*Seasonal autoregressive integrated moving average*) model to the UK inflation using longer historical period of 25+ years. Using stationarity test, we establish that the UK inflation exhibits both deterministic and stochastic patterns, with later being a dominant factor of the inflation process. We demonstrate that SARIMA model fits the UK inflation well and, as such, can be well used to forecast future inflation based on the fitted model.
SARIMA model belongs to the to the family of ARIMA models that are known to be practical and useful when modelling economic time series. They generally tend to outperform more complex econometric modes in short-term forecasting. Although the modelling framework is agnostic and neglects the requirements for richer explanatory variables, this simplicity and dependence only on past values makes the model flexible, powerful and dynamic in nature that adapts well to the inflation process. Nowadays it is difficult to prove that other modelling choices will perform better when it comes to inflation prediction on a short or even longer-time basis.
#Economic data collection#
Our input data for the inflation model is the UK RPI statistics from 1998. Monthly index data will be used an our primary input:
ukrpi = {119.5, 120.2, 121.4, 125.1, 126.2, 126.7, 126.8, 128.1,
129.3, 130.3, 130., 129.9, 130.2, 130.9, 131.4, 133.1, 133.5,
134.1, 133.8, 134.1, 134.6, 135.1, 135.6, 135.7, 135.6, 136.3,
136.7, 138.8, 139.3, 139.3, 138.8, 138.9, 139.4, 139.9, 139.7,
139.2, 137.9, 138.8, 139.3, 140.6, 141.1, 141., 140.7, 141.3,
141.9, 141.8, 141.6, 141.9, 141.3, 142.1, 142.5, 144.2, 144.7,
144.7, 144., 144.7, 145., 145.2, 145.3, 146., 146., 146.9, 147.5,
149., 149.6, 149.8, 149.1, 149.9, 150.6, 149.8, 149.8, 150.7,
150.2, 150.9, 151.5, 152.6, 152.9, 153., 152.4, 153.1, 153.8,
153.8, 153.9, 154.4, 154.4, 155., 155.4, 156.3, 156.9, 157.5,
157.5, 158.5, 159.3, 159.5, 159.6, 160., 159.5, 160.3, 160.8,
162.6, 163.5, 163.4, 163., 163.7, 164.4, 164.5, 164.4, 164.4,
163.4, 163.7, 164.1, 165.2, 165.6, 165.6, 165.1, 165.5, 166.2,
166.5, 166.7, 167.3, 166.6, 167.5, 168.4, 170.1, 170.7, 171.1,
170.5, 170.5, 171.7, 171.6, 172.1, 172.2, 171.1, 172., 172.2,
173.1, 174.2, 174.4, 173.3, 174., 174.6, 174.3, 173.6, 173.4,
173.3, 173.8, 174.5, 175.7, 176.2, 176.2, 175.9, 176.4, 177.6,
177.9, 178.2, 178.5, 178.4, 179.3, 179.9, 181.2, 181.5, 181.3,
181.3, 181.6, 182.5, 182.6, 182.7, 183.5, 183.1, 183.8, 184.6,
185.7, 186.5, 186.8, 186.8, 187.4, 188.1, 188.6, 189., 189.9,
188.9, 189.6, 190.5, 191.6, 192., 192.2, 192.2, 192.6, 193.1,
193.3, 193.6, 194.1, 193.4, 194.2, 195., 196.5, 197.7, 198.5,
198.5, 199.2, 200.1, 200.4, 201.1, 202.7, 201.6, 203.1, 204.4,
205.4, 206.2, 207.3, 206.1, 207.3, 208., 208.9, 209.7, 210.9,
209.8, 211.4, 212.1, 214., 215.1, 216.8, 216.5, 217.2, 218.4,
217.7, 216., 212.9, 210.1, 211.4, 211.3, 211.5, 212.8, 213.4,
213.4, 214.4, 215.3, 216., 216.6, 218., 217.9, 219.2, 220.7, 222.8,
223.6, 224.1, 223.6, 224.5, 225.3, 225.8, 226.8, 228.4, 229.,
231.3, 232.5, 234.4, 235.2, 235.2, 234.7, 236.1, 237.9, 238.,
238.5, 239.4, 238., 239.9, 240.8, 242.5, 242.4, 241.8, 242.1, 243.,
244.2, 245.6, 245.6, 246.8, 245.8, 247.6, 248.7, 249.5, 250.,
249.7, 249.7, 251., 251.9, 251.9, 252.1, 253.4, 252.6, 254.2,
254.8, 255.7, 255.9, 256.3, 256., 257., 257.6, 257.7, 257.1, 257.5,
255.4, 256.7, 257.1, 258., 258.5, 258.9, 258.6, 259.8, 259.6,
259.5, 259.8, 260.6, 258.8, 260., 261.1, 261.4, 262.1, 263.1,
263.4, 264.4, 264.9, 264.8, 265.5, 267.1, 265.5, 268.4, 269.3};
We build Time Series object from the above data
rpita = TimeSeries[ukrpi, {{1990, 1, 1}, {2017, 3, 1}, "Month"}];
DateListPlot[rpita, PlotTheme -> "Business", PlotStyle -> Black,
Filling -> Axis, FillingStyle -> Directive[Opacity[0.1], Pink],
PlotLabel -> Style["UK RPI: 1990-2017", 17]]
![enter image description here][2]
and obtain UK i inflation data by log-differencing of the RPI
tsdata = TimeSeries[rpita];
logdata = Log[tsdata];
dlogdata = Differences[logdata, 1, 12];
DateListPlot[dlogdata, PlotTheme -> "Business", PlotStyle -> Red,
PlotLabel -> Style["UK Inflation: 1990-2017", 17]]
![enter image description here][3]
#Data examination#
We first want to examine the underlying data and perform the **normality tests**.
ftest = DistributionFitTest[dlogdata, Automatic, "HypothesisTestData"];
ftest["TestDataTable", All]
![enter image description here][4]
All above tests uniformly confirm the *non-normal distribution* of the UK inflation data. This is also obvious from the above chart where we observe some structural breaks that lead to skewed and peaked distribution.
The non-normality is also confirmed at the histogram comparison to fitted normal distribution. Peaked and fatter tails point out at non-normal patterns.
Show[Histogram[dlogdata, Automatic, "ProbabilityDensity",
ColorFunction -> Function[{height}, ColorData["Rainbow"][height]],
PlotLabel -> Style["UK Inflation: Normality test", 17]],
Plot[PDF[ftest["FittedDistribution"], x], {x, -0.01, 0.09},
PlotStyle -> {Black, Thick}]]
![enter image description here][5]
This is further proved by localising four distributional moments over time - we clearly observe oscillating mean, noisy volatility and non-stable skewness, kurtosis and other related measures.
The above analysis shows that the UK inflation evolution over the past 25+ years has primarily been stochastic with less prominent deterministic trends. This is not surprising as during that period the UK economy went through a number of economic cycles with high and low inflation.
f = {Mean, StandardDeviation, Skewness, Kurtosis, QuartileDeviation,
QuartileSkewness};
c = {Blue, Red, Brown, Black, Green, Purple};
Table[DateListPlot[MovingMap[f[[i]], dlogdata, "Year"],
PlotLabel -> f[[i]], PlotStyle -> c[[i]]], {i, Length[f]}] // Quiet
![enter image description here][6]
#Stationarity and autocorrelation tests#
Aside from normality, we may want to look at the additional data properties to determine if the UK inflation data are well-suited for SARIMA modelling.
*Stationarity*: we want to establish if stochastic properties of the data (distributional moments - such as mean, variance, skewness etc) are time-invariant, To achieve this, we apply a number of Unit Root Tests to the underlying data.
Let's recall that the unit root existence in the data will indicate *non-stationarity* and a small *p*-value returned by the tests will confirm that the unit root is *not present*.
urt = UnitRootTest[dlogdata, Automatic, "HypothesisTestData"];
urt["TestDataTable", All]
![enter image description here][7]
Low p-value in the above three tests confirms the absence of unit root and therefore confirmation that the series is stationary.
*Autocorrelation*: the purpose is to determine if there is autocorrelation (i.e.serial correlation) in the error terms of the underlying data. Similarly to the above, we refer to the p-static as an indicator of autocorrelation presence in the data. Low p-value will confirm that the time series data is autocorrelated.
act = AutocorrelationTest[dlogdata, Automatic, "HypothesisTestData"];
act["TestDataTable", All]
![enter image description here][8]
The autocorrelation test confirms that the inflation data is path-wise correlated. This is again non-surprising as the inflation data typically display serial correlation and linear co-dependency.
#SARIMA model fitting#
Having determined data properties and having completed data tests, we can now proceed with SARIMA model fitting to the UK inflation data. This can be easily done with time series model fit function.
inflsarima = TimeSeriesModelFit[dlogdata, {"SARIMA", 12}]
![enter image description here][9]
inflsarima["BestFit"]
> SARIMAProcess[-0.000103279, {0.927709}, 0, {0.376708, 0.175456}, {12,
> {}, 1, {-0.484877, 0.0838243}}, 0.0000259715]
Mathematica fits **SARIMA** (p,d,q) (P,D,Q) model with single parameter non-seasonal integration order (p),two-parameter non-seasonal autoregressive order (q), single parameter seasonal integration order (D) and two-parameter seasonal moving average order (Q). There is notable absence of non-seasonal integration order (d) and seasonal autoregressive order P in the fitted model.
We can easily extract the information about the automated model fitting by requesting model selection statistics:
inflsarima["CandidateSelectionTable"]
![enter image description here][10]
We can see that the **AIC** criterion has been used to select the most optimal model for our task. We can also visualise parameter statistics to see how each value has been determined and what are the errors in their estimates.
inflsarima["ParameterTable"]
![enter image description here][11]
#Model diagnostics#
We can now examine how well the model fits the data and how accurate it is w.r.t. predicted valued. We will run the correlogram examination of the residuals (difference between the actual data point and the estimates) to check if any significant patterns about the data have not be left out in our model. If all information has been captured, then the ACF and PACF plots should resemble white noise.
inflsarima["ACFPlot"]
inflsarima["PACFPlot"]
![enter image description here][12]
![enter image description here][13]
The visual examination of the model confirms well-behaved residuals on both graphs. All autocorrelations in the ACF and PACF plots are within the confidence band, and therefore not significantly different from zero. The only exception with marginal significance can be seen at lag 9 which can be smoothly ignored on grounds of minor substance. The plots do not reveal any other significant outliers or heteroskedasticity. This confirms that selected SARIMA model fits the inflation data *well*.
We can further enhance our diagnostics by running the autocorrelation test again - now on the model residuals. We can additionally performs the Box-Ljung test on the residuals, with null hypothesis that the residuals are not different from white noise.
If a Box-Ljung test returns a large p-value, it suggests that the residuals have no remaining autocorrelations, i.e., they resemble white noise.
inflsarima["LjungBoxPlot"]
![enter image description here][14]
This is indeed the case, which merely confirms that residuals do not exhibit any correlation pattern. The same conclusion can be derived from the residual fit that should clearly display white noise-alike patterns
DateListPlot[inflsarima["FitResiduals"],
ColorFunction -> Function[{x, y}, Hue[y]],
PlotLabel -> Style["SARIMA - fitted residuals", 15]]
![enter image description here][15]
#Forecasting inflation with SARIMA model#
We are now in a position to use the SARIMA model to forecast future inflation evolution. We first use the **TimeSeriesForecast** function to predict the UK inflation for the next 3 years:
forecast = TimeSeriesForecast[inflsarima["Process"], dlogdata, {0, 3*12}];
DateListPlot[{TimeSeriesWindow[
dlogdata, {{2013, 1, 1}, {2017, 3, 1}}], forecast}, Joined -> True,
Filling -> Bottom, PlotTheme -> "Web",
PlotLegends -> {"past inflation", "forecast"},
PlotLabel ->
Style["Historical and predicted inflation: 3 years outlook", 15]]
![enter image description here][16]
We use *'automatic'* method for the forecast, which is the best choice to get the optimised response from the predicted outcome. As the chart above shows, the model predicts subdued inflation outcome with the levels below 3%.
How confident can we be with our prediction? We compute standard errors and review the entire model with the error bands.
moderror = Sqrt[forecast["MeanSquaredErrors"]];
uperr = TimeSeriesThread[{1, 1}.# &, {forecast, moderror}];
dnerr = TimeSeriesThread[{1, -1}.# &, {forecast, moderror}];
DateListPlot[{TimeSeriesWindow[dlogdata, {{2013, 1, 1}, {2017, 3, 1}}], forecast, uperr, dnerr},
Joined -> True, PlotTheme -> "Web", Filling -> {3 -> {4}},
PlotLegends -> {"past inflation", "forecast", "upp err band",
"low err band"},
PlotLabel -> Style["Inflation forecst with prediction errors", 15]]
![enter image description here][17]
We can see where the predicted inflation value can go in the next three years. Whilst 6% inflation (or deflation) does not seem to be realistic at present, it is still the possibility that the model takes into consideration when deriving future paths.
We can certainly go beyond standard model analysis to derive further insight from the setting. Once useful approach is the examination of stochastic nature of the model. We will simulate future inflation paths, determine distributional quantiles and visualise them on the graph.
sim = RandomFunction[inflsarima, {0, 12*3}, 100];
stmean = TimeSeriesThread[Mean, sim];
stup = TimeSeriesThread[Quantile[#, 0.95] &, sim];
stdown = TimeSeriesThread[Quantile[#, 0.05] &, sim];
sim2 = DateListPlot[sim, PlotStyle -> Directive[Opacity[.30]],
Joined -> True, PlotRange -> All];
Show[DateListPlot[{stmean, stup, stdown},
PlotStyle -> {Blue, Green, Red},
PlotLabel -> Style["Projected inflation with confidence bands", 15],
PlotLegends -> {"Projection", "Upper band", "Lower band"}], sim2]
![enter image description here][18]
Simulation approach is handy tool to observe seasonality effect on the future inflation modelling. As this is present in all future predictions, we can detect where seasonality effect can influence inflation forecast. This can be seen as additional 'risk factor' in the outcome prediction.
From the above analysis we can easily locate the quantile values as *'worst outcomes'* within prediction tolerance.
DateListPlot[{TimeSeriesWindow[
dlogdata, {{2013, 1, 1}, {2017, 3, 1}}], stup, stdown},
PlotStyle -> Thick,
PlotLabel -> Style["Where are future inflation limits?", 16],
PlotLegends -> {"Hist inflation", "Upper limit", "Lower limit"},
PlotTheme -> "Web", Filling -> {2 -> {3}}]
![enter image description here][19]
The chart above reveals interesting pattern: deflation scenario is now steeper and deeper. This may indicate higher downside risk as turn to deflation will be more abrupt with potentially higher severity.
#Conclusion#
We have demonstrated the ease of using Mathematica for inflation modelling tasks using rich library of built-in routines and functions for time series analysis. As many functions are built to operate in automatic mode, the task of inflation modelling can be reduced to data sourcing and outcome analysis. All hard work of data preparation, model fitting and diagnostics can be left to excellent Mathematica capabilities in this field.
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=Inflationmodels.jpg&userId=387433
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1.png&userId=20103
[3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2.png&userId=20103
[4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=3.png&userId=20103
[5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=4.png&userId=20103
[6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-06-13at09.48.40.png&userId=20103
[7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=6.png&userId=20103
[8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=7.png&userId=20103
[9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=8.png&userId=20103
[10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=9.png&userId=20103
[11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=10.png&userId=20103
[12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=11.png&userId=20103
[13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=12.png&userId=20103
[14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=13.png&userId=20103
[15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=14.png&userId=20103
[16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=15.png&userId=20103
[17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=16.png&userId=20103
[18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=17.png&userId=20103
[19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=18.png&userId=20103Igor Hlivka2017-06-09T12:29:10Z