Community RSS Feed
https://community.wolfram.com
RSS Feed for Wolfram Community showing any discussions in tag Staff Picks sorted by activeAI vision via Wolfram Language
https://community.wolfram.com/groups/-/m/t/3072318
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/c298ab8f-16de-4178-bb27-268c24bd8485Anton Antonov2023-11-27T03:45:43ZDirect API access to new features of GPT-4 (including vision, DALL-E, and TTS)
https://community.wolfram.com/groups/-/m/t/3062403
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/34ac42c4-10de-4201-a158-6af47eda2bd8Marco Thiel2023-11-08T21:48:25ZA primer on Association and Dataset
https://community.wolfram.com/groups/-/m/t/1167544
*NOTE: all Wolfram Language code and data are available in the attached notebook at the end of the post.*
----------
For my class this fall, I developed a little primer on Association and Dataset that I think might be useful for many people. So, I'm sharing the attached notebook. It's somewhat about the concepts embedded inside these features. It's intended for people at a beginner-intermediate level of Mathematica/Wolfram Language programming, but might be of value even to some more advanced users who have not poked about the Dataset functionality.
The sections of the notebook are:
1. The world before Associations and Datasets
2. Datasets without Associations
3. Enter the Association
4. Creating a Dataset from a List of Associations
5. Nice queries with Dataset
6. Query
7. Some Recipes
#The world before Associations and Datasets#
Here' s an array of data. The data happens to represent the cabin class, age, gender, and survival of some of the passengers on the Titanic.
t = {{"1st", 29, "female", True}, {"1st", 30, "male", False}, {"1st",
58, "female", True}, {"1st", 52, "female", True}, {"1st", 21,
"female", True}, {"2nd", 54, "male", False}, {"2nd", 29, "female",
False}, {"3rd", 42, "male", False}};
As it stands, our data is a List of Lists.
Head[t]
> List
Head /@ t
> {List, List, List, List, List, List, List, List}
Suppose I wanted to get the second and fifth rows of the data. This is how I could do it.
t[[{2, 5}]]
> {{"1st", 30, "male", False}, {"1st", 21, "female", True}}
Suppose we want to group the passengers by gender and then compute the mean age. We could do this with the following pretty confusing code.
Use and enjoy. Constructive feedback appreciated.
grouped = GatherBy[t, #[[3]] &];
justTheAges = grouped[[All, All, 2]];
Mean /@ justTheAges
> {189/5, 42}
Or I could write it as a one liner this way.
Map[Mean, GatherBy[t, #[[3]] &][[All, All, 2]]]
> {189/5, 42}
But either way, realize that I have to remember that gender is the third column and that age is the second column. When there is a lot of data, this can get hard to remember.
#Datasets without Associations#
I could, if I wanted, convert this data into a Dataset. I do this below simply by wrapping Dataset about t. You see there is now some formatting about the data. But there are no column headers (because no one has told Dataset what to use). And there are no row headers, again because no one has told Dataset what to use.
t2 = Dataset[t]
![enter image description here][1]
The head of the expression has changed.
Head[t2]
> Dataset
Now, I can now access the data in a different way.
Query[{2, 5}][t2]
![enter image description here][2]
Or, I can do this. Mathematica basically converts this expression into Query[{2,5}][t2]. The expression t2[{2,5}] is basically syntactic sugar.
t2[{2, 5}]
![enter image description here][3]
##Digression : Using Query explicitly or using syntactic sugar##
Why, by the way would anyone use the longer form if Mathematica does the work for you? Suppose you want to store a Dataset operation -- perhaps a complex series of Dataset operations -- but you want it to work not just on a particular Dataset but on any Dataset (that is compatible). Here's how you could do it.
q = Query[{2, 5}]
> Query[{2, 5}]
q[t2]
![enter image description here][4]
Now, let' s create a permutation of the t2 Dataset so that the rows are scrambled up.
t2Scrambled = t2[{1, 4, 8, 3, 2, 7, 5}]
![enter image description here][5]
We can now run the q operation on t2Scrambled. Notice that the output has changed even though the query has stayed the same.
q[t2Scrambled]
![enter image description here][6]
We can also generate Query objects with functions. Here's a trivial example. There are very few languages of which I am aware that have the ability to generate queries by using a function. The one other example is Julia.
makeASimpleQuery[n_] := Query[n]
makeASimpleQuery[{3, 4, 7}][t2]
![enter image description here][7]
##MapReduce operations on Dataset objects##
Now, if I want to know the mean ages of the genders I can use this code. This kind of grouping of data and then performing some sort of aggregation operation on the groups is sometimes known as a MapReduce. (I'm not a fan of the name, but it is widely used). It's also sometimes known as a rollup or an aggregation.
Query[GroupBy[#[[3]] &], Mean, #[[2]] &][t2]
![enter image description here][8]
Or this shorthand form in which the Query is constructed.
t2g = t2[GroupBy[#[[3]] &], Mean, #[[2]] &]
![enter image description here][9]
I think this is a little cleaner. But we still have to remember the numbers of the columns, which can be challenging.
By the way, just to emphasize how we can make this all functional, here's a function that creates a query that can run any operation (not just computing the mean) on the Dataset grouped by gender and then working on age.
genderOp[f_] := Query[GroupBy[#[[3]] &], f, #[[2]] &]
genderOp[Max][t2]
![enter image description here][10]
To test your understanding, see if you can find the minimum age for each class of passenger on the Titanic in our Dataset **t2**.
Query[GroupBy[#[[1]] &], Min, #[[2]] &][t2]
![enter image description here][11]
#Enter the Association#
##Review of Association##
If you feel comfortable with Associations, you can skip this section; otherwise read it carefully. Basically the key to understanding most Dataset operations is understanding Associations.
###Construction of Associations###
Now let' s alter the data so that we don't have to remember those facts. To do this we will create an **Association**. Here's an example called **assoc1**. Notice that we do so by creating a sequence of rules and then wrapping it in an Association head. Notice that the standard output does not preserve the word "Association" as the head but, just as List is outputted as stuff inside curly braces, Association is outputted as stuff inside these funky "<|" and "|>" glyphs.
assoc1 = Association["class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True]
> <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True|>
I could equivalently have created a list of rules rather than a sequence. Mathematica would basically unwrap the **List** and create a sequence.
assoc1L = Association[{"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True}]
> <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True|>
We can use **AssociationThread** to create Associations in a different way. The first argument is the list of things that go on the left hand side of the Rules -- the "keys" -- and the second argument is the list of things that go on the right hand side of the Rules -- the "values".
assoc1T = AssociationThread[{"class", "age", "gender", "survived"}, {"1st", 29, "female", True}]
> <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True|>
Now let's use **AssociationThread** function to create a list of Associations similar to our original data.
convertListToAssociation =
list \[Function]
AssociationThread[{"class", "age", "gender", "survived"}, list]
> Function[list, AssociationThread[{"class", "age", "gender", "survived"}, list]]
I start with t and Map the **convertListToAssociation** function over the rows of the data. I end up with a list of Associations.
t3 = Map[convertListToAssociation, t]
![enter image description here][12]
###Keys and Values###
Associations have keys and values. These data structures are used in other computer languages but known by different names: *Python* and *Julia* call them dictionaries. *Go* and *Scala* call them maps. *Perl* and *Ruby* call them hashes. *Java* calls it a *HashMap*. And *Javascript* calls it an object. But they all work pretty similarly. Anyway, the keys of an **Association** are the things on the left hand side of the Rules.
Keys[assoc1]
> {"class", "age", "gender", "survived"}
And the values of an Association are the things on the right hand side of the Rules.
Values[assoc1]
> {"1st", 29, "female", True}
That' s about all there is too it. Except for one thing. Take a look at the input and output that follows.
assoc2 = Association["a" -> 3, "b" -> 4, "a" -> 5]
> <|"a" -> 5, "b" -> 4|>
You can' t have duplicate keys in an Association. So, when Mathematica confronts duplicate keys, it uses the last key it saw. You might think this is a minor point, but it is actually very important in coding. We will see why soon.
###Nested Associations###
A funny thing happens if you nest an **Association** inside another **Association**.
Association[assoc1, assoc2]
> <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True, "a" -> 5, "b" -> 4|>
You end up with a single un - nested (flat) association. That's a little unusual for Mathematica, but we can exploit this flattening as a way of adding elements to an Association.
Association[Association["dances" -> False], assoc1]
> <|"dances" -> False, "class" -> "1st", "age" -> 29, "gender" ->
> "female", "survived" -> True|>
Or, here' s a function that exploits the flattening to add elements to an **Association**.
addstuff = Association[#, "dances" -> False, "sings" -> True] &
> Association[#1, "dances" -> False, "sings" -> True] &
addstuff[assoc1]
> <|"class" -> "1st", "age" -> 29, "gender" -> "female", "survived" -> True, "dances" -> False, "sings" -> True|>
###Extracting Values from Associations###
Just as the values contained in a **List** can be accessed by using the **Part** function, the values contained in an **Association** can likewise be accessed. Suppose, for example that I wanted to compute double the age of the person in **assoc1**.
It turns out there are a lot of ways of doing this. The first is to treat the Association as a list except that the indices, instead of being integers, are the "keys" that are on the left hand side of the rules.
2*Part[assoc1, "age"]
> 58
2*assoc1[["age"]]
> 58
A second way is to use Query. We can wrap the "key" in the head **Key** just to make sure Mathematica understands that the thing is a Key.
2*Query[Key["age"]][assoc1]
> 58
Usually we can omit the Key and everything works fine.
2*Query["age"][assoc1]
> 58
A third way is to write a function that has an association as its argument.
af = Function[Slot["age"]]
> "#age &"
Now look what we can do.
2*Query[af][assoc1]
> 58
We can shorten this approach by using a simpler syntax for a function.
2*Query[#age &][assoc1]
> 58
Note, though that this still will not work. Basically, Mathematica is confused. It thinks the function itself is the key.
2*assoc1[af]
> 2 Missing["KeyAbsent", #age &]
But here' s a simple workaround. For very simple functions, I can just use the name of the key.
2*assoc1["age"]
> 58
##A Note on Slot Arguments##
And please pay attention to this : sometimes the Mathematica parser gets confused when it confronts a "slot argument" written as #something. If you see this happening, write it as Slot["something"].
Slot["iamaslot"] === #iamaslot
> True
Here' s another problem. What if the key in the association has spaces or non-standard characters in it. Any of these, for example, are perfectly fine keys: the string "I have a lot of spaces in me", the string "I_have_underscores", the symbol True, the integer 43. But if we try to denote those keys by putting a hash in front of them, it will lead to confusion and problems.
problemAssociation = Association["I have a lot of spaces in me" -> 1, "I_have_underscores" -> 2, True -> 3, 43 -> 4]
> <|"I have a lot of spaces in me" -> 1, "I_have_underscores" -> 2,
True -> 3, 43 -> 4|>
{Query[#I have a lot of spaces in me &][problemAssociation],
Query[#I _have _underescores &][problemAssociation]}
![enter image description here][13]
Here' s a solution.
{Query[Slot["I have a lot of spaces in me"] &][problemAssociation],
Query[Slot["I_have_underscores"] &][problemAssociation]}
> {1, 2}
Here' s how we solve the use of True and an integer as keys. We preface them with **Key**.
{Query[#True &][problemAssociation], Query[#43 &][problemAssociation]}
![enter image description here][14]
{Query[Key[True]][problemAssociation],
Query[Key[43]][problemAssociation]}
> {3, 4}
##Working with Associations and Lists of Associations##
Here' s something we can do with the data in the form of an Association. I could ask for the gender of the person in the third row as follows. Notice I did not have to remember that "gender" was generally in the third position.
t3[[3]][["gender"]]
> "female"
So, even if I scramble the rows, I can still use the same code.
t3Scrambled = Map[convertListToAssociation, t[[All, {4, 1, 3, 2}]]]
![enter image description here][15]
t3Scrambled[[3]][["gender"]]
> female
I could also group the people according to their cabin class. Here I use Query on a list of Associations.
Query[GroupBy[#class &]][t3]
![enter image description here][16]
Again, the following code, which does not explicitly use **Query**, won' t work. Basically, nothing has told Mathematica to translate t3[stuff___] \[RightArrow]Query[stuff][t3]. If t3 had a head of Dataset, Mathematica would know to make the translation.
t3[GroupBy[#class &]]
![enter image description here][17]
I can also get certain values for all the Associations in a list of Associations.
Query[All, #age &][t3]
> {29, 30, 58, 52, 21, 54, 29, 42}
I can also map a function onto the result. I don't have to go outside the Query form to do so.
Query[f, #age &][t3]
> f[{29, 30, 58, 52, 21, 54, 29, 42}]
Or, without exiting the Query form, I can map a function onto each element of the result.
Query[Map[f], #age &][t3]
> {f[29], f[30], f[58], f[52], f[21], f[54], f[29], f[42]}
I could also do the same thing as follows.
Query[All, #age &, f][t3]
> {f[29], f[30], f[58], f[52], f[21], f[54], f[29], f[42]}
#Creating a Dataset from a List of Associations#
To get full use out of Query and to permit syntactic shorthands, we need for Mathematica to understand that the list of Associations is in fact a Dataset. Here' s all it takes.
d3 = Dataset[t3]
![enter image description here][18]
We can recover our original list of associations by use of the **Normal** command.
t3 === Normal[d3]
> True
With the data inside a Dataset object we now have pretty formatting. But we have more.
We can still do this. We get the same result but in a more attractive form.
d3g = Query[GroupBy[#class &]][d3]
![enter image description here][19]
But now this shorthand works too.
d3g = d3[GroupBy[#class &]]
![enter image description here][20]
And compare these two elements of code. When the data is in the form of a dataset, Mathematica understands that the stuff in the brackets is not intended as a key but rather is intended to be transformed into a Query.
{Query[#age &][t3[[1]]], d3[[1]][#age &]}
> {29, 29}
##A Dataset that is an Association of Associations##
Let' s look under the hood of **d3g**.
d3gn = Normal[d3g]
![enter image description here][21]
Note : if you *really* want to look under the hood of a **Dataset** ask to see the **Dataset** in **FullForm**. You can also get more information by running the undocumented package Dataset`, but this is definitely NOT recommended for the non-advanced user.
What we see is an Association in which each of the values is itself a list of Associations.
We can map a function over d3gn.
Map[f, d3gn]
![enter image description here][22]
I can of course do the mapping within the **Query** construct.
Query[All, f][d3gn]
![enter image description here][23]
If I try synactic sugar, it doesn' t work because d3gn is not a Dataset.
d3gn[All, f]
> Missing["KeyAbsent", All]
But, if I use the Dataset version, it does work. (The first line may be an ellipsis depending on your operating system and display, but if you look under the hood it looks just like the values for 2nd and 3rd. I have no idea why an ellipsis is being inserted.
d3g[All, f]
![enter image description here][24]
##A Dataset that just has a single Association inside.##
We can also have a Dataset that just has a single Association inside. Mathematica presents the information with the keys and values displayed vertically.
Dataset[d3[[1]]]
![enter image description here][25]
In theory, we could have a Dataset that just had a single number inside it.
Dataset[6]
![enter image description here][26]
#Nice queries with Dataset#
Now I can construct a query that takes a dataset and groups it by the gender column. It then takes each grouping and applies the Mean function to at least part of it. What part? The "age" column part. Notice that I no longer have to remember that gender is the third column and age is the second column.
qd = Query[GroupBy[#gender &], Mean, #age &]
> Query[GroupBy[#gender &], Mean, #age &]
Now I can run this query on t3.
qd[d3]
![enter image description here][27]
We can now learn a lot about Query. So long as our data is in the form of a Dataset we can write the query as either a formal Query or use syntactic sugar.
#Query#
A major part of working with data is to understand **Query**. Let's start with a completely abstract **Query**, that we will call **q1**.
q1 = Query[f];
Now let' s run q1 on t3.
q1[t3]
![enter image description here][28]
We end up with a list of Associations that has f wrapped around it at the highest level. It's the same as if I wrote the following code.
f[t3] === q1[t3]
> True
Now, let' s write a **Query** that applies the function g at the top level of the list of associations and the function **f** at the second level, i.e. to each of the rows. Why does it work at the second level? Because it's the second argument to **Query**.
q2 = Query[g, f];
q2[t3]
![enter image description here][29]
The result is the same as if I mapped **f** onto t3 at its first level and then wrapped **g** around it.
g[Map[f, t3, {1}]] === q2[d3]
Query[All, MapAt[StringTake[#, 1] &, #, {{"class"}, {"gender"}}] &][d3]
Here' s a function **firstchar** that takes the first character in a string.
firstchar = StringTake[#, 1] &
> StringTake[#1, 1] &
Now, let' s construct a query **cg1** that applies **firstchar** to the class and gender keys in each row.
cg1 = Query[All,
a \[Function] MapAt[firstchar, a, {{"class"}, {"gender"}}]]
> Query[All, Function[a, MapAt[firstchar, a, {{"class"}, {"gender"}}]]]
We apply **cg1** to our little dataset **d3**.
cg1[d3]
![enter image description here][30]
What if we want to apply the same function to every element of the Dataset. We just apply it at the lowest level. Here's one way.
Query[Map[f, #, {-1}] &][d3]
![enter image description here][31]
We can also combine it with column wise and entirety wise operations. For reasons that are not clear, Mathematica can't understand this as a Dataset and returns the Normal form.
Query[(Map[f, #, {-1}] &) /* entiretywise, columnwise][d3]
![enter image description here][32]
Here' s how we could actually a multilevel **Query**.
Suppose we want to write a function that computes the fraction of the people in this little dataset that survived. The first step is simply going to be to extract the survival value and convert it to 1 if True and 0 otherwise. There's a built in function Boole that does this.
{Boole[True], Boole[False]}
> {1, 0}
q3 = Query[something,
assoc \[Function] assoc["survived"] /. {True -> 1, _ -> 0}]
> Query[something, Function[assoc, assoc["survived"] /. {True -> 1, _
> -> 0}]]
q3[t3]
> something[{1, 0, 1, 1, 1, 0, 0, 0}]
So, now we have something wrapping a list of 1 s and 0 s. By making **something** the **Mean** function, we can achieve our result.
q4 = Query[Mean, Boole[#survived] &]
> Query[Mean, Boole[#survived] &]
q4[d3]
> 1/2
We can also examine survival by gender. Notice that **Query** is a little like **Association**: it gets automatically flattened.
Query[GroupBy[#gender &], q4][t3]
> <|"female" -> 4/5, "male" -> 0|>
If the data is held in a **Dataset**, we can also write the final step as follows.
d3[GroupBy[#gender &], q4]
![enter image description here][33]
Notice that even if we omit the "Query", this code works. Mathematica just figures out that you meant Query.
The code immediately above is in the form we typically see and often use.
#Some Recipes#
titanic = ExampleData[{"Dataset", "Titanic"}]
![enter image description here][34]
How to add a value to the Dataset based on values external to the existing columns.
Here' s some additional data. Notice that the data is the same length as the titanic dataset.
stuffToBeAdded =
Table[Association["id" -> i,
"weight" -> RandomInteger[{80, 200}]], {i, Length[titanic]}]
![enter image description here][35]
We use **Join** at level 2.
augmentedTitanic = Join[titanic, stuffToBeAdded, 2]
![enter image description here][36]
##How to add a column to a Dataset based on values in the existing columns and to do so row-wise##
Notice that the query below does NOT change the value of the titanic dataset. To change the value of the titanic dataset, one would need to set titanic to the result of the computation. Remember, Mathematica generally does not have side effects or do modifications in place.
Query[All, Association[#, "classsex" -> {#class, #sex}] &][titanic]
![enter image description here][37]
We can add multiple columns this way.
Query[All,
Association[#, "classsex" -> {#class, #sex},
"agesqrt" -> Sqrt[#age]] &][titanic]
![enter image description here][38]
##How to change the value of an existing column : row - wise##
Age everyone one year.
Query[All, Association[#, "age" -> #age + 1] &][titanic]
![enter image description here][39]
How to change the value of columns selectively.
Query[All,
Association[#,
"age" -> If[#sex === "male", #age + 1, #age]] &][titanic]
![enter image description here][40]
How to create a new column based on some aggregate operator applied to another column.
With[{meanAge = Query[Mean, #age &][titanic]},
Query[All,
Association[#, "ageDeviation" -> #age - meanAge] &]][titanic]
![enter image description here][41]
Can you develop your own recipes?
[1]: http://community.wolfram.com//c/portal/getImageAttachment?filename=17751.png&userId=20103
[2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=47222.png&userId=20103
[3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=60813.png&userId=20103
[4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=83664.png&userId=20103
[5]: http://community.wolfram.com//c/portal/getImageAttachment?filename=44435.png&userId=20103
[6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=105416.png&userId=20103
[7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=49777.png&userId=20103
[8]: http://community.wolfram.com//c/portal/getImageAttachment?filename=16898.png&userId=20103
[9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=28239.png&userId=20103
[10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=262710.png&userId=20103
[11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=749611.png&userId=20103
[12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=209912.png&userId=20103
[13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=925313.png&userId=20103
[14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=851614.png&userId=20103
[15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=627315.png&userId=20103
[16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=679516.png&userId=20103
[17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=674717.png&userId=20103
[18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=707518.png&userId=20103
[19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1053319.png&userId=20103
[20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=579820.png&userId=20103
[21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=980221.png&userId=20103
[22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=450322.png&userId=20103
[23]: http://community.wolfram.com//c/portal/getImageAttachment?filename=113723.png&userId=20103
[24]: http://community.wolfram.com//c/portal/getImageAttachment?filename=932624.png&userId=20103
[25]: http://community.wolfram.com//c/portal/getImageAttachment?filename=754825.png&userId=20103
[26]: http://community.wolfram.com//c/portal/getImageAttachment?filename=836826.png&userId=20103
[27]: http://community.wolfram.com//c/portal/getImageAttachment?filename=393227.png&userId=20103
[28]: http://community.wolfram.com//c/portal/getImageAttachment?filename=577928.png&userId=20103
[29]: http://community.wolfram.com//c/portal/getImageAttachment?filename=158329.png&userId=20103
[30]: http://community.wolfram.com//c/portal/getImageAttachment?filename=984930.png&userId=20103
[31]: http://community.wolfram.com//c/portal/getImageAttachment?filename=664831.png&userId=20103
[32]: http://community.wolfram.com//c/portal/getImageAttachment?filename=1090332.png&userId=20103
[33]: http://community.wolfram.com//c/portal/getImageAttachment?filename=450733.png&userId=20103
[34]: http://community.wolfram.com//c/portal/getImageAttachment?filename=740234.png&userId=20103
[35]: http://community.wolfram.com//c/portal/getImageAttachment?filename=143835.png&userId=20103
[36]: http://community.wolfram.com//c/portal/getImageAttachment?filename=36.png&userId=20103
[37]: http://community.wolfram.com//c/portal/getImageAttachment?filename=37.png&userId=20103
[38]: http://community.wolfram.com//c/portal/getImageAttachment?filename=38.png&userId=20103
[39]: http://community.wolfram.com//c/portal/getImageAttachment?filename=39.png&userId=20103
[40]: http://community.wolfram.com//c/portal/getImageAttachment?filename=40.png&userId=20103
[41]: http://community.wolfram.com//c/portal/getImageAttachment?filename=41.png&userId=20103Seth Chandler2017-08-20T19:26:30ZVisualizing the Fibonacci subproblem dependency graph
https://community.wolfram.com/groups/-/m/t/3071742
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/7756ffe4-7a43-4c3b-94d5-1b90a3fb58d3Chase Marangu2023-11-26T07:33:55ZExploring Real-World Problems: Simplifying complexity of nonlinear PDE models
https://community.wolfram.com/groups/-/m/t/3072567
![enter image description here][1]
&[Wolfram Notebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=3310hero.jpg&userId=20103
[2]: https://www.wolframcloud.com/obj/0d314114-0ddb-44d2-bcad-974bcc113b52Vedat Senol2023-11-27T19:20:06Z[BOOK] Demystifying Epidemics: A Beginner's Guide to Infectious Disease Modeling
https://community.wolfram.com/groups/-/m/t/3070525
![enter image description here][1]
&[Wolfram Notebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=9515Cover.png&userId=20103
[2]: https://www.wolframcloud.com/obj/97559e36-c684-41f1-a543-00317cdb6709Athanasios Paraskevopoulos2023-11-23T22:22:06ZFibonacci Day 2023: Computing and visualizing the first 100 digits
https://community.wolfram.com/groups/-/m/t/3070628
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/bf11b939-1545-4981-80b1-f215587558e0Daniel Carvalho2023-11-24T00:26:28Z[WSRP23] On the mergers of numerical multiway systems
https://community.wolfram.com/groups/-/m/t/2965206
![Multiway graph][1]
&[Wolfram Notebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Untitled.png&userId=2964459
[2]: https://www.wolframcloud.com/obj/b6ad5a15-e8d2-4359-ae89-b72d3e3d06d4Eric Archerman2023-07-13T23:11:59ZIntegrating C++ and the Wolfram Language with LibraryLink Utilities
https://community.wolfram.com/groups/-/m/t/2133603
If you are a Wolfram Language user with C/C++ background, at some point you might find yourself thinking:
> *I have this great C++ library, I wish I could use it in my Wolfram Language code.*
or
> *I love programming in the WL, but some things are easier for me to implement in C++. If only there was a way to combine these two languages.*
A quick google search reveals two main options: [WSTP](https://reference.wolfram.com/language/guide/WSTPCLanguageFunctions.html) (formerly known as MathLink
) and [LibraryLink](https://reference.wolfram.com/language/guide/LibraryLink.html). They provide interfaces for C programs to exchange data with the
Wolfram Language and to perform operations on either side. Both frameworks are documented with lots of examples.
Let us take a look at a sample WSTP function for sending list of integers from the Wolfram Language to a C program:
[![WSTPFunction][1]](https://reference.wolfram.com/language/ref/c/WSGetInteger32List.html)
If your code already uses error codes, manual memory management, out-parameters, etc. both WSTP and LibraryLink will
perfectly fit into your codebase.
However, if you prefer to use exceptions for error handling, RAII for resource management, namespaces, templates and other modern C++ techniques, you might
find the integration slightly more challenging.
The remaining part of this text assumes that the reader has basic knowledge of LibraryLink.
# LibraryLink Utilities
One thing that can help you achieve seamless integration of C++ and WL is called *LibraryLink Utilities* (abbr. **LLU**) which is a set of modern C++ wrappers
over
LibraryLink and WSTP. LLU is developed and maintained by Wolfram Research and it has been released under the MIT license on GitHub:
https://github.com/WolframResearch/LibraryLinkUtilities
With documentation and examples also available online:
https://wolframresearch.github.io/LibraryLinkUtilities/
We do not provide prebuilt binaries of LLU, so the only way to use it is by building locally from sources. The most common approach is to clone the repository
from one of the following URLs:
- **[ssh]:** `git@github.com:WolframResearch/LibraryLinkUtilities.git`
- **[https]:** `https://github.com/WolframResearch/LibraryLinkUtilities.git`
Alternatively, a zip package can be downloaded from GitHub containing a snapshot from any branch.
# Demo
Let's see how to write a complete C++ package for Wolfram Language using LLU. We will analyze a toy paclet named "Demo" which is shipped with LLU (under
`/tests/Demo`). This paclet exposes two functions to the Wolfram Language:
- `CaesarCipherEncode[message_String, shift_Integer]` - encodes given message by shifting every character by shift positions in the English alphabet
- `CaesarCipherDecode[cipherText_String, shift_Integer]` - restores the original message encoded with Caesar's cipher given the encoded text and the shift
Such simple functionality in practice would easily be implemented directly in WL without invoking library functions implemented in C++, but when you learn
the general structure of paclets written with LLU, you can have arbitrarily complex C++ code with all the rest being as simple as in this demo.
## Paclet structure
The demo paclet resides at the `/tests/Demo` directory inside the LLU repo root directory and has the following structure.
```
Demo
├── Demo
│ ├── Kernel
│ │ └── Demo.wl
│ └── PacletInfo.wl
├── Sources
│ └── demo.cpp
├── Tests
│ └── test.wl
└── CMakeLists.txt
```
The first thing you may notice is that Demo paclet uses CMake as a build system (via the presence of CMakeLists.txt). This is the most natural choice because
LLU
itself uses CMake and it provides a number of useful CMake utilities. It is possible to build and install LLU with CMake but then use different build system
in paclets, or even just ``CCompilerDriver`CreateLibrary``.
As you can see, our Demo follows the general paclet structure described in the
[Paclet Development Guide](https://www.wolframcloud.com/obj/tgayley/Published/PacletDevelopment.nb) - we have a Kernel directory with WL sources and the
required `PacletInfo.wl` with paclet's metadata. Additionally, we store C++ sources in `Sources` directory and a minimal set of tests in `Tests`. Having
unit tests in a paclet is not required but often a good practice.
## Installation
To be able to use the Wolfram Language functions that the Demo provides, it is enough to follow these steps:
1. Configure, build and install LLU as described
[in the documentation](https://wolframresearch.github.io/LibraryLinkUtilities/basic/how_to_use.html#configure).
Let’s say you chose `/my/workspace/LLU` as the install directory.
2. Navigate to `tests/Demo` in the LLU source directory - this is where the sources of the Demo project reside.
3. Run the following commands (or equivalent for your system):
```commandline
cmake -DLLU_ROOT=/my/workspace/LLU -DWolframLanguage_ROOT=/path/to/WolframDesktop/ -B build
cd build/
cmake --build . --target install
```
(where `/path/to/WolframDesktop` is a path to the Wolfram product (i.e. Wolfram Desktop/Mathematica/Wolfram Engine) you have installed)
This will put a complete paclet directory structure under `tests/Demo/build/Demo`. You can copy this directory into
[\$UserBaseDirectory/Applications](https://reference.wolfram.com/language/ref/$UserBaseDirectory.html) and then load the paclet by calling
Needs["Demo`"]
in a notebook.
However, a preferred way to make the paclet discoverable by the Wolfram Language is to build another target called `paclet`
```commandline
cmake --build . --target paclet
```
When built, the `paclet` target will take the directory structure created by the `install` target and turn it into a proper `.paclet` file. It can optionally
validate paclet contents, run a test file or install the paclet to a directory where the Wolfram Language can automatically find it. Investigate the
`tests/Demo/CMakeLists.txt` file for details on how to create and use this target.
Finally, after building the `paclet` target or manually copying the Demo paclet into the user paclet directory, you should be able to run the following code
in a notebook:
```mathematica
In[1]:= Needs["Demo`"]
In[2]:= Demo`CaesarCipherEncode["HelloWorld", 5]
Out[2]= "Mjqqtbtwqi"
In[3]:= Demo`CaesarCipherDecode[%, 5]
Out[3]= "HelloWorld"
```
## Using Demo as paclet template
Demo paclet is intended to serve as a short showcase of LLU capabilities but also as a convenient base for developers to create custom paclets. Let's see
what changes are necessary to turn Demo into a project of your own.
1. Copy all the sources to a new directory named the same as your new paclet, e.g.
```commandline
cd /path/to/LLU
cp -r tests/Demo ~/projects/MyPaclet
```
2. Rename all occurrences of "Demo" in directory and file names to "MyPaclet" (or whatever name you chose in step 1). On a Linux system, this step may be
accomplished as follows:
```commandline
cd ~/projects/MyPaclet
mv Demo MyPaclet
mv MyPaclet/Kernel/Demo.wl MyPaclet/Kernel/MyPaclet.wl
mv Sources/demo.cpp Sources/mypaclet.cpp
```
3. Modify source files (you can use `sed` on Linux or `Find & Replace` function in your favorite IDE):
1. Change the ``Demo` `` context in *MyPaclet.wl* and *Tests/test.wl* to ``MyPaclet` ``
2. Rename the main target and variable names in *CMakeLists.txt*
3. In *MyPaclet.wl* change the library name in line `` `LLU`InitializePacletLibrary["Demo"];`` to `"MyPaclet"`.
4. Unleash your creativity! Remove the two library functions defined by the Demo project and write your own code instead.
# Examples
Now that you know how to easily create WolframLanguage paclets that use LLU, let's see a number of common use cases and applications where LLU is especially
handy compared to plain LibraryLink or WSTP. All the C++ code snippets shown below require `LLU/LLU.h` and possibly other header files to be included but
this is omitted for brevity. For the WL snippets, whenever a library function is loaded and used we assume that this was preceded with a proper paclet
loading routine which includes an evaluation of ``LLU`InitializePacletLibrary["LibraryName"]``. This part should be clear after following the instructions in
the previous section.
## Working with numeric data
Exchanging numerical data between C++ and the Wolfram Language is one of the most popular tasks for LibraryLink. Initially, one could transfer scalars or
tensors of type: integer, real or complex. Although this seems to cover all the common cases, it is often desired to have a finer control over the data type.
For instance, if we know that all our data is in range 0 - 100 we can store each number in 1 byte, whereas tensors (a.k.a packed arrays) use 4 or 8 bytes
per element. For this reason
LibraryLink now supports [NumericArrays](https://reference.wolfram.com/language/ref/NumericArray.html).
Consider a simple function that takes a packed array of real numbers and returns a copy of the input array but with all the numbers clipped to
[0, 1] interval:
EXTERN_C DLLEXPORT int ClipToUnitInterval(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
// Create MArgumentManager to manage all the input and output arguments for the library function
LLU::MArgumentManager mngr(libData, Argc, Args, Res);
// The first argument should be our tensor of real numbers, otherwise the following line will throw an exception
const auto data = mngr.getTensor<double>(0);
// We will return a copy of the input array with some values modified
auto clipped = data.clone();
// We iterate over all elements of the container in a single loop even if the Tensor is multidimensional
for (auto& d : clipped) {
d = std::clamp(d, 0., 1.);
}
// Set function result
mngr.set(clipped);
return LLU::ErrorCode::NoError;
}
or, if one is not afraid to use a macro, we can hide all the boilerplate code:
```cpp
LLU_LIBRARY_FUNCTION(ClipToUnitInterval) {
const auto data = mngr.getTensor<double>(0);
auto clipped = data.clone();
for (auto& d : clipped) {
d = std::clamp(d, 0., 1.);
}
mngr.set(clipped);
}
```
Such library function can be loaded and used in the Wolfram Language as follows:
```mathematica
In[1]:= LLU`PacletFunctionSet[ClipToUnitInterval, "ClipToUnitInterval", {{Real, 1}}, {Real, _}];
In[2]:= ClipToUnitInterval[{{0.2, 0.3}, {-0.1, 1.4}}]
Out[2]= {{0.2, 0.3}, {0., 1.}}
```
Our `ClipToUnitInterval` works great but is limited to the input arrays of type Real. Oftentimes we want to have a function that works for any type of
tensor. The simplest solution is to provide 3 almost identical functions: for integers, reals and complex numbers. This gets less feasible for NumericArray,
which supports 12 different element types. LLU comes to the rescue here with "generic" containers. For every container `X` that is templated with element type
(such as `LLU::NumericArray` or `LLU::Image`) an `LLU::GenericX` is provided, which is a type-unaware counterpart of `X`.
For instance, we can write
```cpp
const auto numericArr = mngr.getNumericArray<double>(0);
```
if the only thing that we want to accept as the first argument passed to the function is a NumericArray of type "Real64". If we want to accept any NumericArray,
we can instead write:
```cpp
const auto numericArr = mngr.getGenericNumericArray(0);
```
our code is thus more flexible but we cannot access the underlying data of `numericArr` directly (because we don't know its type). We could write a `switch`
statement that would detect the data type at run-time and act accordingly; we could also use a helper function from LLU called `LLU::asTypedNumericArray` which
takes
a generic NumericArray, and an action to perform on this array, which behaves as if the data type of the array was known. For instance
LLU_LIBRARY_FUNCTION(Reverse) {
// Receive a NumericArray of any type as the first argument to the library function
auto numericArr = mngr.getGenericNumericArray(0);
// Perform an action (second argument) on the numericArr. The typedNA passed to the lambda is a strongly typed LLU::NumericArray.
LLU::asTypedNumericArray(numericArr, [&mngr](auto&& typedNA) {
// Extract data type from the argument, this can be simplified in C++20 with a template parameter list for the lambda
using T = typename std::remove_reference_t<decltype(typedNA)>::value_type;
// Create a "reversed" copy of the array preserving the type and dimensions.
auto reversedArr = LLU::NumericArray<T>(std::crbegin(typedNA), std::crend(typedNA), LLU::MArrayDimensions {typedNA.getDimensions(), typedNA.getRank()});
// Set the reversed array as the result
mngr.set(std::move(reversedArr));
});
}
## Error handling
If every function that we write always succeeded with no errors, programming would be much easier. Alas, errors do happen for many various reasons;
proper handling and reporting to the caller is one of the most challenging tasks, especially for API functions. LibraryLink, like many C libraries,
uses integer error codes with [7 predefined values](https://reference.wolfram.com/language/LibraryLink/tutorial/LibraryStructure.html#394079419).
In C++, however, an often preferred method of error handling is via exceptions. LLU conforms with this practice and uses exceptions both to report problems that
occurred in its own code and to allow programmers to throw exceptions in their library functions. Such exceptions may even be propagated up to
the Wolfram Language and transformed into `Failure` objects. This is described in more details in the
[LLU documentation](https://wolframresearch.github.io/LibraryLinkUtilities/modules/error_handling.html).
Imagine we expect two different exceptional situations that may occur in our code. For each of them, we register a separate kind of exception, with different
name and short textual description which may contain template slots populated at run time. The registration should be done inside `WolframLibrary_initialize`
and may look like this:
```cpp
EXTERN_C DLLEXPORT int WolframLibrary_initialize(WolframLibraryData libData) {
LLU::LibraryData::setLibraryData(libData);
LLU::ErrorManager::registerPacletErrors({
{"NoSourceError", "Requested data source does not exist."},
{"EmptySourceError", "Requested data source has `elemCount` elements, but required at least `elemReq`."}
});
return LLU::ErrorCode::NoError;
}
```
With this simple setup we can now throw exceptions from the function that reads data:
```cpp
void readData(DataSource* source) {
if (!source) {
LLU::ErrorManager::throwException("NoSourceError");
}
if (source->elemCount() < 3) {
LLU::ErrorManager::throwException("EmptySourceError", source->elemCount(), 3);
}
//...
}
```
Each call to `LLU::ErrorManager::throwException` causes an exception of class `LLU::LibraryLinkError` with predefined name and error code to be thrown. All
parameters of `throwException` after the first one are used to populate consecutive template slots in the error message. The only thing left to do now
is to catch the exception. Usually, you catch only in the interface functions (the ones with `EXTERN_C DLLEXPORT`), extract the error code from exception
and return it:
```cpp
EXTERN_C DLLEXPORT int ReadData(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
auto err = LLU::ErrorCode::NoError; // no error initially
try {
//...
} catch (const LLU::LibraryLinkError& e) {
err = e.which(); // extract error code from LibraryLinkError
} catch (...) {
err = LLU::ErrorCode::FunctionError; // to be safe, one may want to handle non-LLU exceptions as well and return generic error codes
}
return err;
}
```
LLU can later recognize this error code on the WL side and create a proper `Failure` object corresponding to the exception that was thrown on the C++ side.
#### Custom exception class
It is possible to make `ErrorManager` throw exceptions of different class than `LibraryLinkError` as long as this class defines a publicly accessible
constructor which takes `const LibraryLinkError&` as the first argument. This feature might be useful when we want a custom action to happen right before
throwing, for instance logging some information to a text file. Consider a simple example:
// Define custom error class. LoggingError inherits from LibraryLinkError and additionally logs file name, function name and line number to a file.
struct LoggingError : LLU::LibraryLinkError {
// Where to log exception details
static constexpr const char* logFileName = "LLUErrorLog.txt";
template<typename... T>
LoggingError(const LLU::LibraryLinkError& e, WolframLibraryData ld, int line, const std::string& file, const std::string& func, T&&... params)
: LLU::LibraryLinkError(e) {
// Pass LibraryLinkError's parameters to top-level
setMessageParameters(ld, std::forward<T>(params)...);
sendParameters(ld);
// Extend debug info to contain file and function names and append to the log file
setDebugInfo("Exception " + name() + " in " + file + ":" + std::to_string(line) + " in " + func + ": " + debug());
std::ofstream log {logFileName, std::ios_base::app};
log << debug() << "\n";
}
};
// Helper macro to conveniently throw LoggingErrors with current line, filename and function name
#define THROW_LOGGING_ERROR(name, ...) LLU::ErrorManager::throwCustomException<LoggingError>(name, libData, __LINE__, __FILE__, __func__, __VA_ARGS__)
This class can be used as follows:
```cpp
EXTERN_C DLLEXPORT int ReadDataWithLoggingError(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
auto err = LLU::ErrorCode::NoError;
try {
LLU::MArgumentManager mngr(Argc, Args, Res);
auto fileName = mngr.getString(0);
if (fileName.find(':') != std::string::npos) {
THROW_LOGGING_ERROR("DataFileError", fileName, 0, R"(file name contains a possibly problematic character ":")");
}
auto fileNameLen = static_cast<wsint64>(fileName.length());
if (fileNameLen > 16) {
THROW_LOGGING_ERROR("DataFileError", fileName, fileNameLen, "file name is too long");
}
THROW_LOGGING_ERROR("DataFileError", fileName, fileNameLen, "data type is not supported");
}
// The error handling code can remain as before, because our custom exception class inherits from LLU::LibraryLinkError
catch (const LLU::LibraryLinkError& e) {
err = e.which();
} catch (...) {
err = LLU::ErrorCode::FunctionError;
}
return err;
}
```
#### Top-level layer
Apart from the C++ code, paclets often have nontrivial amount of Wolfram Language code where errors might also occur. In order to achieve uniform
error reporting across C++ and WL, one needs to register errors specific to the WL layer of the paclet:
```mathematica
`LLU`RegisterPacletErrors[<|
"InvalidInput" -> "Data provided to the function was invalid.",
"UnexpectedError" -> "Unexpected error occurred with error code: `errCode`."
|>];
```
`RegisterPacletErrors` takes an `Association` of user-defined errors of the form
error_name -> error_message
Such registered errors can later be issued from the Wolfram Language part of the project like this:
```mathematica
status = DoSomething[input];
If[Not @ StatusOK[status],
`LLU`ThrowPacletFailure["UnexpectedError", "MessageParameters" -> <|"errCode" -> status|>]
]
```
Alternatively, one can use ``LLU`CreatePacletFailure`` which returns the `Failure` object as the result instead of throwing it.
## Monitoring progress
If functions in your library are meant to work on potentially large amounts of data, and are likely to take considerable amount of time to complete, it is a
good idea in terms of user experience to provide some kind of indication to the user of how far the function is from completing and to allow users to abort
the functions execution at any point. LibraryLink offers the latter in the form of
[AbortQ](https://reference.wolfram.com/language/LibraryLink/ref/callback/AbortQ.html)
function which lets you check whether an abort was requested and then act accordingly (e.g. stop the computation, clean up, return).
LLU goes one step further and provides a mechanism for functions to report progress to the caller on the Wolfram Language side. The basic form of reporting
progress is simply a real number that is shared between the WL and C++ code. During a library function execution, the number is updated from the C++ code and
those
changes can be reflected on the WL side in many different ways to give user a visual indication of current progress, e.g. using
[ProgressIndicator](https://reference.wolfram.com/language/ref/ProgressIndicator.html).
Imagine we want to write a simple function `SleepTight` that simply sleeps in a loop moving the progress bar in a steady pace. From the caller's perspective
, this
function takes a single argument - total time (in seconds) for the function to complete. We could load this function in the WL like this:
```mathematica
LLU`PacletFunctionSet[SleepTight, "SleepTight", {Real}, "Void", "ProgressMonitor" -> MyPaclet`PM`SleepTight];
```
Notice that for progress reporting to work on the Wolfram Language side as expected, the library function must be loaded with extra option
`“ProgressMonitor” -> x`, where `x` is a Symbol. Every time your library function reports progress, the new progress value will be assigned to `x`.
By default, `“ProgressMonitor” -> None` is used. It is a good idea to make sure the name for the monitoring symbol will be unique. One suggestion is to use
``PacletName`PM`` as the context, and the name of the symbol to be the same as the function name.
Now, one can run the library function with simple progress bar:
```mathematica
Monitor[
(* the function will run for 5 seconds *)
SleepTight[5],
(* check the value of progress every 0.2 second and display as progress bar *)
ProgressIndicator[Dynamic @ First @ Refresh[MyPaclet`PM`SleepTight, UpdateInterval -> 0.2]]
]
```
The remaining task of implementing the function on the C++ side is fairly straightforward:
```cpp
EXTERN_C DLLEXPORT int SleepTight(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
// Create MArgumentManager to manage all the input and output arguments for the library function
LLU::MArgumentManager mngr(libData, Argc, Args, Res);
// Get the first argument which determines how many seconds should this function take to evaluate
auto totalTime = mngr.getReal(0);
// Calculate number of steps for the progress bar, we want 10 steps per second
auto numOfSteps = static_cast<int>(std::ceil(totalTime * 10));
// Get ProgressMonitor instance, initialize with the number of seconds per step
auto pm = mngr.getProgressMonitor(1.0 / numOfSteps);
// Sleep in a loop, increase progress in each iteration. Increasing progress also automatically checks for Abort.
for (int i = 0; i < numOfSteps; ++i) {
std::this_thread::sleep_for(100ms);
++pm;
}
return LLU::ErrorCode::NoError;
}
```
## Using WSTP link as a stream
WSTP links can be used to sequentially transfer data between C++ code and the Wolfram Language. It makes them perfect candidates to be represented as streams
in the C++ sense (with "stream operators" `>>` and `<<`). LLU provides such a stream class for WSTP, called `WSStream`. The class takes two non-type template
parameters which represent default encodings to be assumed for C-string and `std::string`s that sent or received via the stream.
A sample function that receives a number of UTF8 strings from the link and puts back a total byte-length of all the strings, could look like this:
```cpp
LIBRARY_WSTP_FUNCTION(StringsByteLength) {
WSStream<WS::Encoding::UTF8> stream {wsl};
std::vector<std::string> strs;
stream >> strs;
auto total = std::accumulate(std::cbegin(strs), std::cend(strs), 0UL, [](auto sum, const std::string& s) { return sum + s.length(); });
stream << total << WS::EndPacket;
}
```
Which could then be loaded and used in the Wolfram Language as follows:
```mathematica
In[1]:= LLU`WSTPFunctionSet[StringsByteLength, "StringsByteLength"];
In[2]:= StringsByteLength["a", "bb", "ccc"]
Out[2]= 6
```
A slightly more complicated example could involve a function that accepts a list of integers and returns an Association with a list of divisors for each of the
input integers, or `$Failed` if the number has more than 15 divisors. This example demonstrates how to send expressions whose length is not known a-priori:
```cpp
LLU_WSTP_FUNCTION(FactorsOrFailed) {
WSStream<WS::Encoding::Byte> stream {wsl, 1}; // we expect 1 input argument - a list of integers
std::vector<int> numbers;
stream >> numbers;
stream << WS::Association(numbers.size()); // the output Association will have the same length as the input list even if the list contains duplicates
for (auto n : numbers) { // duplicates will be automatically taken care of on the WL side
stream << WS::Rule << n;
stream << WS::BeginExpr("List"); // we do not know the number of factors so we have to use WS::BeginExpr
int divisors = 0;
for (int j = 1; j <= n; ++j) {
if (n % j == 0) {
if (divisors < 15) {
stream << j;
divisors++;
} else {
stream << WS::DropExpr(); // Drop the List that and send $Failed instead
stream << WS::Symbol("$Failed");
break;
}
}
}
stream << WS::EndExpr();
}
stream << WS::EndPacket;
}
```
We can now load our function and use it like this:
```mathematica
In[1]:= LLU`WSTPFunctionSet[Factors, "FactorsOrFailed"];
In[2]:= Factors[{1, 3, 15, 10000000, 15}]
Out[2]= <|1 -> {1}, 3 -> {1, 3}, 15 -> {1, 3, 5, 15}, 10000000 -> $Failed|>
```
WSTP is very flexible and lets you exchange arbitrary expressions between the Wolfram Language and C++ code but it comes with a price - bigger overhead compared
to data transfer via LibraryLink arguments.
## Passing and returning heterogeneous lists
LibraryLink has a fixed list of types that can be used as library function arguments and return values. For instance, see the "Details and Options" section in
the documentation of [LibraryFunctionLoad](https://reference.wolfram.com/language/ref/LibraryFunctionLoad.html). As you can see, it is straightforward to
pass an array of numeric data, a string or an image but already sending a list of images is hard. One can conform all the images and transfer them via
LibraryLink as a single `Image3D` but this puts a toll on efficiency. Sending a list of strings is simply impossible without tricks or workarounds.
This topic has been brought up by the community quite a few times:
- [Returning multiple results from a LibraryLink function](https://mathematica.stackexchange.com/questions/31545/returning-multiple-results-from-a-librarylink-function)
- [Is it possible to let LibraryLink return multiple results?](https://mathematica.stackexchange.com/questions/121825/is-it-possible-to-let-librarylink-return-multiple-results)
- [LibraryFunctionLoad for multivariate C++ function containing several arguments](https://mathematica.stackexchange.com/questions/128723/libraryfunctionload-for-multivariate-c-function-containing-several-arguments)
- [LibraryFunctionLoad with variable number of arguments](https://community.wolfram.com/groups/-/m/t/366849)
- [Working with a variable number of arguments in LibraryLink](https://community.wolfram.com/groups/-/m/t/190560)
It turns out that the solution to this problem already exists in LibraryLink and is called `DataStore`. The only problem is that for now it remains
undocumented. Below is a quick glimpse of what `DataStore` is, which should also help understand the C++ wrappers of `DataStore` that LLU has to offer.
![DataStore structure][2]
As can be seen in the picture, `DataStore` is a simple unidirectional linked list but with limited functionality exposed, compared for instance to
`std::forward_list`. The API provided in the Wolfram Library allows for
- creating an empty `DataStore`
- copying or deleting existing `DataStore`
- appending new nodes at the end
- iterating over nodes
- obtaining the length of the store (in constant time)
Each node of the `DataStore` carries a value of type `MArgument`, which is a union type of all types that LibraryLink can handle as function arguments or
return values. The `MArgument` union include `DataStore` which means that the store can be nested. Additionally, every node contains an optional "name" which
can be any string. Names do not have to be unique.
A simple library function that takes a list of strings (in the form of a `DataStore`) and returns a new list with each of the input strings reversed,
implemented in pure LibraryLink may look like this:
EXTERN_C DLLEXPORT int StringsReversedLibraryLink(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
// get DataStore which is the first input argument to the library function
DataStore ds_in = MArgument_getDataStore(Args[0]);
mint length = libData->ioLibraryFunctions->DataStore_getLength(ds_in);
// create new DataStore to hold the result
DataStore ds_out = libData->ioLibraryFunctions->createDataStore();
if (ds_out == nullptr) {
// error handling
}
// start traversing the DataStore from the first node
DataStoreNode dsn = libData->ioLibraryFunctions->DataStore_getFirstNode(ds_in);
while (dsn != nullptr) {
MArgument node_data;
if (libData->ioLibraryFunctions->DataStoreNode_getData(dsn, &node_data) != 0) {
// error handling and cleanup
}
if (libData->ioLibraryFunctions->DataStoreNode_getDataType(dsn) != MType_UTF8String) {
// error handling and cleanup
}
// reverse the order of characters in the string and push to the output DataStore
std::string_view s {MArgument_getUTF8String(node_data)};
std::string outStr(s.rbegin(), s.rend()); // create reversed copy
libData->ioLibraryFunctions->DataStore_addString(ds_out, outStr.data());
// move to the next node
dsn = libData->ioLibraryFunctions->DataStoreNode_getNextNode(dsn);
}
// set the newly created DataStore as the result of this library function call
MArgument_setDataStore(Res, ds_out);
return LIBRARY_NO_ERROR;
}
It is clear how to operate on `DataStore`s in C code now, but the Wolfram Language side of LibraryLink also uses a representation of this structure and it is
defined as follows:
```mathematica
Developer`DataStore[node_expr$1, node_expr$2, ..., node_expr$n]
```
where each `node_expr` is of the form `string -> expr` or just `expr` with an extra requirement that `expr` must be an expression supported in LibraryLink.
For example, ``Developer`DataStore["abc", "de", "f"]`` passed to the function implemented above would result in ``Developer`DataStore["cba", "ed", "f"]`` being
returned from the library.
LLU provides two direct wrappers over `DataStore`: `LLU::GenericDataList` and `LLU::DataList<T>`. The former equips `DataStore` with a proper container
interface including methods like `push_back()`, `front()`, `back()` or `length()`. It also offers easier iteration over the list with `begin()` and `end()`.
The second wrapper is templated with a node type and should be used whenever we expect a homogeneous `DataStore` (with all nodes of the same type). The types
that can be passed as template parameter are not the raw LibraryLink types included in the MArgument union but rather their LLU counterparts. For convenience,
they are enclosed in `LLU::NodeType` namespace with following members:
```cpp
/// Boolean type, corresponds to True or False in the Wolfram Language
using Boolean = bool;
/// Machine integer type
using Integer = mint;
/// Double precision floating point type
using Real = double;
/// Complex number type, bitwise-compatible with mcomplex defined in WolframLibrary.h
using Complex = std::complex<double>;
/// Tensor stands for a GenericTensor - type agnostic wrapper over MTensor
using Tensor = MContainer<MArgumentType::Tensor>;
/// SparseArray type corresponds to the "raw" MSparseArray as LLU does not have its own wrapper for this structure yet
using SparseArray = MSparseArray;
/// NumericArray stands for a GenericNumericArray - type agnostic wrapper over MNumericArray
using NumericArray = MContainer<MArgumentType::NumericArray>;
/// Image stands for a GenericImage - type agnostic wrapper over MImage
using Image = MContainer<MArgumentType::Image>;
/// String values from LibraryLink (char*) are wrapped in std::string_view
using UTF8String = std::string_view;
/// DataStore stands for a GenericDataList - type agnostic wrapper over DataStore
using DataStore = MContainer<MArgumentType::DataStore>;
```
Additionally, there is also `LLU::NodeType::Any` which can be used to make `LLU::DataList` work with a heterogeneous `DataStore`. `LLU::DataList` compared to
`LLU::GenericDataList` provides more iteration options (iteration over node values or node names only) and a function to immediately create a `std::vector` out
of the stored values.
Let us implement the same function as above using `LLU::GenericDataList`:
EXTERN_C DLLEXPORT int StringsReversedGeneric(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
auto err = LLU::ErrorCode::NoError;
try {
using NodeT = LLU::NodeType::UTF8String;
LLU::MArgumentManager mngr(libData, Argc, Args, Res);
auto dsIn = mngr.getGenericDataList(0);
GenericDataList dsOut;
// iterate over the GeneridDataList with a range-based for loop
for (auto node : dsIn) {
// we are dealing with generic DataList, so we need to be explicit about the actual node type
std::string_view s = node.as<NodeT>();
std::string reversed {s.rbegin(), s.rend()}; // create reversed copy
dsOut.push_back(std::string_view(reversed)); // passing a view is fine because DataStore will copy the string immediately
}
mngr.set(dsOut);
} catch (const LLU::LibraryLinkError& e) {
err = e.which();
} catch (...) {
err = LLU::ErrorCode::FunctionError;
}
return err;
In the function above we know that all nodes should contain strings, so we could use `LLU::DataList<LLU::NodeType::UTF8String>` instead of the generic data
list. The code would stay almost the same, except we could simply call `node.value()` to get the corresponding `std::string_view`.
For a different example, consider a function `SeparateKeysAndValue` that takes a data store of named complex numbers and separates it into two data stores:
one holding names of the original data store and another one with values. For instance,
```mma
Developer`DataStore["a" -> 1 + 2.5 * I, "b" -> -3. - 6.I, 2I, "d" -> -4]
```
would be transformed into
```
Developer`DataStore[
"Keys" -> Developer`DataStore["a", "b", "", "d"],
"Values" -> Developer`DataStore[1. + 2.5 * I, -3. - 6.I, 2.I, -4.]
]
```
The implementation of such library function may look as follows:
EXTERN_C DLLEXPORT int SeparateKeysAndValues(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {
LLU::MArgumentManager mngr(libData, Argc, Args, Res);
auto dsIn = mngr.getDataList<LLU::NodeType::Complex>(0);
DataList<LLU::NodeType::UTF8String> keys;
DataList<LLU::NodeType::Complex> values;
// we use structured bindings to immediately split each node into name and value
for (auto [name, value] : dsIn) {
keys.push_back(name);
values.push_back(value);
}
DataList<GenericDataList> dsOut; // the output type is a DataList of DataLists
dsOut.push_back("Keys", std::move(keys));
dsOut.push_back("Values", std::move(values));
mngr.set(dsOut);
}
Instead of a single loop with structured bindings, we could also make two passes over the input data list:
DataList<LLU::NodeType::UTF8String> keys;
for (auto name : LLU::NameAdaptor {dsIn}) {
keys.push_back(name);
}
DataList<LLU::NodeType::Complex> values;
for (auto value : LLU::ValueAdaptor {dsIn}) {
values.push_back(value);
}
Notice that we use iterator adaptors `LLU::NameAdaptor` and `LLU::ValueAdaptor` to iterate over a specific property of the nodes (either name or value,
respectively).
# Summary
LibraryLink Utilities is a new addition to the Wolfram ecosystem providing a convenient way for developers to integrate the Wolfram Language with external
libraries written in modern C++.
It builds upon existing LibraryLink framework making it more fitting into C++ codebases and providing a number of utilities for easier paclet development
including strongly typed container classes, enhanced error reporting and progress monitoring capabilities, a way to bring object-oriented paradigm into the
Wolfram Language with Managed Library Expressions, lazy loading of library functions and much more.
All of this comes with a good integration with CMake and a rich documentation in 2 flavors: doxygen comments for all public entities in the library
([see here](https://wolframresearch.github.io/LibraryLinkUtilities/doxygen/)) and tutorial-like Sphinx-based documentation
([here](https://wolframresearch.github.io/LibraryLinkUtilities/)) that covers
the most important aspects of the projects with examples and detailed instructions.
Last but no least, LLU is free and open-source, so anyone is welcome to contribute to this project on GitHub.
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=WSTPFunction.png&userId=824316
[2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=DataStore.png&userId=824316Rafal Chojna2020-12-09T10:47:37ZElastic bouncing in circular bowl
https://community.wolfram.com/groups/-/m/t/3070150
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/2e8ef962-e944-4ab2-a759-bbae56a38670Sander Huisman2023-11-23T18:24:14ZGuide 1: The Wolfram Plugin for ChatGPT
https://community.wolfram.com/groups/-/m/t/3070428
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/ce4a1961-7813-4d3f-a60e-9ec2b6ae3595Michael Trott2023-11-23T17:51:06ZOn Graph Products and Matrix Products
https://community.wolfram.com/groups/-/m/t/3069212
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/44d6ce0c-5cac-473e-80a1-edc40c56592bJ. M.2023-11-22T08:14:10ZParabola on the complex plane
https://community.wolfram.com/groups/-/m/t/3069438
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/3e2e8157-fd16-48db-b260-fb5bc682a51fRobert Rimmer2023-11-22T18:30:02ZNone of the countries bordering Poland before 1990 exist today: the fall of the Berlin Wall and USSR
https://community.wolfram.com/groups/-/m/t/3067969
![enter image description here][1]
&[Wolfram Notebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=_2_ezgif.com-optimizecopy.gif&userId=11733
[2]: https://www.wolframcloud.com/obj/34c7205a-40cd-4587-b6a1-f1fadc1ed8ceVitaliy Kaurov2023-11-21T01:14:10ZArtificial intelligence and the skill premium
https://community.wolfram.com/groups/-/m/t/3068735
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/b347ea32-be91-4666-8721-584a93daa737Jamel Saadaoui2023-11-21T19:37:19ZPhase Transitions in the Quantum Transverse Field Ising Chain
https://community.wolfram.com/groups/-/m/t/2554882
&[Wolfram Notebook][1]
[1]: https://www.wolframcloud.com/obj/f3893352-8dea-43e3-a022-3dc23c33fce7Aditya Dhumuntarao2022-06-21T20:40:21ZFinding all structural breaks in time series
https://community.wolfram.com/groups/-/m/t/1749226
## Introduction
In this document we show how to find the so called "structural breaks",
[[Wk1](https://en.wikipedia.org/wiki/Structural_break)],
in a given time series.
The algorithm is based in on a systematic application of Chow Test,
[[Wk2](https://en.wikipedia.org/wiki/Chow_test)],
combined with an algorithm for local extrema finding in noisy time series,
[[AA1](https://mathematicaforprediction.wordpress.com/2015/09/27/finding-local-extrema-in-noisy-data-using-quantile-regression/)].
The algorithm implementation is based on the packages
["MonadicQuantileRegression.m"](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m),
[[AAp1](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m)],
and ["MonadicStructuralBreaksFinder.m"](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicStructuralBreaksFinder.m),
[[AAp2](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicStructuralBreaksFinder.m)].
The package [[AAp1](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m)]
provides the software monad QRMon that allows rapid and concise specification of
[Quantile Regression](https://en.wikipedia.org/wiki/Quantile_regression) workflows.
The package
[[AAp2](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicStructuralBreaksFinder.m)]
extends QRMon with functionalities related to structural breaks finding.
### What is a structural break?
It looks like at least one type of "structural breaks" are defined through regression models, [[Wk1](https://en.wikipedia.org/wiki/Structural_break)]. Roughly speaking a structural break point of time series is a regressor point that splits the time series in such way that the obtained two parts have very different regression parameters.
One way to test such a point is to use Chow test, [[Wk2](https://en.wikipedia.org/wiki/Chow_test)]. From [[Wk2](https://en.wikipedia.org/wiki/Chow_test)] we have the definition:
The Chow test, proposed by econometrician Gregory Chow in 1960, is a test of whether the true coefficients in two linear regressions on different data sets are equal. In econometrics, it is most commonly used in time series analysis to test for the presence of a structural break at a period which can be assumed to be known a priori (for instance, a major historical event such as a war).
### Example
Here is an example of the described algorithm application to the data from [[Wk2](https://en.wikipedia.org/wiki/Chow_test#/media/File:Chowtest4.svg)].
QRMonUnit[data]?QRMonPlotStructuralBreakSplits[ImageSize -> Small];
![IntroductionsExample](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Introductions-example.png)
## Load packages
Here we load the packages [AAp1] and [AAp2].
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicQuantileRegression.m"]
Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MonadicProgramming/MonadicStructuralBreaksFinder.m"]
## Data used
In this section we assign the data used in this document.
### Illustration data from Wikipedia
Here is the data used in the Wikipedia article "Chow test", [[Wk2](https://en.wikipedia.org/wiki/Chow_test#/media/File:Chowtest4.svg)].
data = {{0.08, 0.34}, {0.16, 0.55}, {0.24, 0.54}, {0.32, 0.77}, {0.4,
0.77}, {0.48, 1.2}, {0.56, 0.57}, {0.64, 1.3}, {0.72, 1.}, {0.8,
1.3}, {0.88, 1.2}, {0.96, 0.88}, {1., 1.2}, {1.1, 1.3}, {1.2,
1.3}, {1.3, 1.4}, {1.4, 1.5}, {1.4, 1.5}, {1.5, 1.5}, {1.6,
1.6}, {1.7, 1.1}, {1.8, 0.98}, {1.8, 1.1}, {1.9, 1.4}, {2.,
1.3}, {2.1, 1.5}, {2.2, 1.3}, {2.2, 1.3}, {2.3, 1.2}, {2.4,
1.1}, {2.5, 1.1}, {2.6, 1.2}, {2.6, 1.4}, {2.7, 1.3}, {2.8,
1.6}, {2.9, 1.5}, {3., 1.4}, {3., 1.8}, {3.1, 1.4}, {3.2,
1.4}, {3.3, 1.4}, {3.4, 2.}, {3.4, 2.}, {3.5, 1.5}, {3.6,
1.8}, {3.7, 2.1}, {3.8, 1.6}, {3.8, 1.8}, {3.9, 1.9}, {4., 2.1}};
ListPlot[data]
![DataUsedWk2](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Data-used-Wk2.png)
### S&P 500 Index
Here we get the time series corresponding to [S&P 500 Index](https://en.wikipedia.org/wiki/S%26P_500_Index).
tsSP500 = FinancialData[Entity["Financial", "^SPX"], {{2015, 1, 1}, Date[]}]
DateListPlot[tsSP500, ImageSize -> Medium]
![DataUsedSP500](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Data-used-SP500.png)
## Application of Chow Test
The Chow Test statistic is implemented in [AAp1].
In this document we rely on the relative comparison of the Chow Test statistic values: the larger the value of the Chow test statistic,
the more likely we have a structural break.
Here is how we can apply the Chow Test with a QRMon pipeline to the [Wk2] data given above.
chowStats =
QRMonUnit[data]?
QRMonChowTestStatistic[Range[1, 3, 0.05], {1, x}]?
QRMonTakeValue;
We see that the regressor points $\text{$\$$Failed}$ and $1.7$ have the largest Chow Test statistic values.
Block[{chPoint = TakeLargestBy[chowStats, Part[#, 2]& , 1]},
ListPlot[{chowStats, chPoint}, Filling -> Axis, PlotLabel -> Row[{"Point with largest Chow Test statistic:",
Spacer[8], chPoint}]]]
![ApplicationOfChowTestchowStats](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Application-of-Chow-Test-chowStats.png)
The first argument of QRMonChowTestStatistic is a list of regressor points or Automatic.
The second argument is a list of functions to be used for the regressions.
Here is an example of an automatic values call.
chowStats2 = QRMonUnit[data]?QRMonChowTestStatistic?QRMonTakeValue;
ListPlot[chowStats2, GridLines -> {
Part[
Part[chowStats2, All, 1],
OutlierIdentifiers`OutlierPosition[
Part[chowStats2, All, 2], OutlierIdentifiers`SPLUSQuartileIdentifierParameters]], None}, GridLinesStyle -> Directive[{Orange, Dashed}], Filling -> Axis]
![ApplicationOfChowTestchowStats2](https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Application-of-Chow-Test-chowStats2.png)
For the set of values displayed above we can apply simple 1D outlier identification methods,
[[AAp3](https://github.com/antononcube/MathematicaForPrediction/blob/master/OutlierIdentifiers.m)],
to automatically find the structural break point.
chowStats2[[All, 1]][[OutlierPosition[chowStats2[[All, 2]], SPLUSQuartileIdentifierParameters]]]
(* {1.7} *)
OutlierPosition[chowStats2[[All, 2]], SPLUSQuartileIdentifierParameters]
(* {20} *)
We cannot use that approach for finding all structural breaks in the general time series cases though as exemplified with the following code using the time series S&P 500 Index.
chowStats3 = QRMonUnit[tsSP500]?QRMonChowTestStatistic?QRMonTakeValue;
DateListPlot[chowStats3, Joined -> False, Filling -> Axis]
![ApplicationOfChowTestSP500](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Application-of-Chow-Test-SP500.png)
OutlierPosition[chowStats3[[All, 2]], SPLUSQuartileIdentifierParameters]
(* {} *)
OutlierPosition[chowStats3[[All, 2]], HampelIdentifierParameters]
(* {} *)
In the rest of the document we provide an algorithm that works for general time series.
## Finding all structural break points
Consider the problem of finding of **all** structural breaks in a given time series.
That can be done (reasonably well) with the following procedure.
1. Chose functions for testing for structural breaks (usually linear.)
2. Apply Chow Test over dense enough set of regressor points.
3. Make a time series of the obtained Chow Test statistics.
4. Find the local maxima of the Chow Test statistics time series.
5. Determine the most significant break point.
6. Plot the splits corresponding to the found structural breaks.
QRMon has a function, QRMonFindLocalExtrema, for finding local extrema; see [AAp1, AA1].
For the goal of finding all structural breaks, that semi-symbolic algorithm is the crucial part in the steps above.
## Computation
### Chose fitting functions
fitFuncs = {1, x};
### Find Chow test statistics local maxima
The computation below combines steps 2,3, and 4.
qrObj =
QRMonUnit[tsSP500]?
QRMonFindChowTestLocalMaxima["Knots" -> 20,
"NearestWithOutliers" -> True,
"NumberOfProximityPoints" -> 5, "EchoPlots" -> True,
"DateListPlot" -> True,
ImageSize -> Medium]?
QRMonEchoValue;
![ComputationLocalMaxima](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Computation-local-maxima.png)
### Find most significant structural break point
splitPoint = TakeLargestBy[qrObj?QRMonTakeValue, #[[2]] &, 1][[1, 1]]
### Plot structural breaks splits and corresponding fittings
Here we just make the plots without showing them.
sbPlots =
QRMonUnit[tsSP500]?
QRMonPlotStructuralBreakSplits[(qrObj? QRMonTakeValue)[[All, 1]],
"LeftPartColor" -> Gray, "DateListPlot" -> True,
"Echo" -> False,
ImageSize -> Medium]?
QRMonTakeValue;
The function QRMonPlotStructuralBreakSplits returns an association that has as keys paired split points and Chow Test statistics; the plots are association's values.
Here we tabulate the plots with plots with most significant breaks shown first.
Multicolumn[
KeyValueMap[
Show[#2, PlotLabel ->
Grid[{{"Point:", #1[[1]]}, {"Chow Test statistic:", #1[[2]]}}, Alignment -> Left]] &, KeySortBy[sbPlots, -#[[2]] &]], 2]
![ComputationStructuralBreaksPlots](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Computation-structural-breaks-plots.png)
## Future plans
We can further apply the algorithm explained above to identifying time series states or components.
The structural break points are used as knots in appropriate Quantile Regression fitting. Here is an example.
The plan is to develop such an identifier of time series states in the near future.
(And present it at [WTC-2019](https://www.wolfram.com/events/technology-conference/2019/).)
![FuturePlansTimeSeriesStates](https://github.com/antononcube/MathematicaForPrediction/raw/master/MarkdownDocuments/Diagrams/Finding-all-structural-breaks-in-time-series/Future-plans-time-series-states.png)
## References
### Articles
\[Wk1\] Wikipedia entry, [Structural breaks](https://en.wikipedia.org/wiki/Structural_break).
\[Wk2\] Wikipedia entry, [Chow test](https://en.wikipedia.org/wiki/Chow_test).
\[AA1\] Anton Antonov, ["Finding local extrema in noisy data using Quantile Regression"](https://mathematicaforprediction.wordpress.com/2015/09/27/finding-local-extrema-in-noisy-data-using-quantile-regression/), (2019), [MathematicaForPrediction at WordPress](https://mathematicaforprediction.wordpress.com/2015/09/27/finding-local-extrema-in-noisy-data-using-quantile-regression/).
\[AA2\] Anton Antonov, ["A monad for Quantile Regression workflows"](https://github.com/antononcube/MathematicaForPrediction/blob/master/MarkdownDocuments/A-monad-for-Quantile-Regression-workflows.md), (2018), [MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction/).
### Packages
\[AAp1\] Anton Antonov, [Monadic Quantile Regression Mathematica package](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicQuantileRegression.m), (2018), [MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction/).
\[AAp2\] Anton Antonov, [Monadic Structural Breaks Finder Mathematica package](https://github.com/antononcube/MathematicaForPrediction/blob/master/MonadicProgramming/MonadicStructuralBreaksFinder.m), (2019), [MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction/).
\[AAp3\] Anton Antonov, [Implementation of one dimensional outlier identifying algorithms in Mathematica](https://github.com/antononcube/MathematicaForPrediction/blob/master/OutlierIdentifiers.m), (2013), [MathematicaForPrediction at GitHub](https://github.com/antononcube/MathematicaForPrediction/).
### Videos
\[AAv1\] Anton Antonov, Structural Breaks with QRMon, (2019), YouTube.Anton Antonov2019-07-31T19:26:28Z[BOOK] Explore Algebra with Wolfram Language: New High School Textbook Available!
https://community.wolfram.com/groups/-/m/t/3067312
![enter image description here][1]
&[Wolfram Notebook][2]
[1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=10328hero.png&userId=20103
[2]: https://www.wolframcloud.com/obj/4ecce92c-d212-4e82-8ec6-4ee75be60facAthanasios Paraskevopoulos2023-11-18T17:06:36Z