Empirically derived conditional distribution of System symbols
Range->Length->Part
is commonly found in:
Given that Mathematica is a symbolic language it would be interesting to view the relationships between these symbols e.g. Length[List[1,2,3]]
implies a relationship of Length -> List
.
Since we do not know this information, we will have to retrieve it. For that, I headed to GitHub. Note: GitHub's public api limits the rate you can query to
$1$ query every
$30$ seconds (so the crawl takes many hours).
(* Get repositories marked with Mathematica as a language *)
githubResults = SearchRepositories["Mathematica"];
"8586 public repositories with language(s): Mathematica.
Only the first 1000 are available via the public api."
(* Get repository names *)
repositoriesWithMathematica = Normal[githubResults[All, "full_name"]];
(* Search for .nb files in these repos *)
notebookResults = SearchNotebooks[repositoriesWithMathematica];
"3525 notebooks found."
(* Now go download all of those .nbs *)
files = DownloadRawURLs[notebookResults, CreateDirectory[]]
With our
$3,525$ notebooks, we are ready to try and extract the Symbol->Symbol
relationship (thanks to Carl Woll for helping with this part):
Test for system symbol
SetAttributes[systemSymbol, {Listable, HoldFirst}]
systemSymbol[Symbol[_String]] = False;
systemSymbol[s_Symbol] := Context[s] === "System`"
systemSymbol[_] = False;
Extract symbol relationships
Note, we replace all non system symbols with a placeholder "NonWolfram"
.
NBExpressionRelations[file_] := Module[
{
nb = NotebookImport[file, "Input"],
expr =
Cases[nb,
p : _Symbol?systemSymbol[s_Symbol, ___] :> Hold[p], {3, Infinity}]
},
Cases[nb,
p_Symbol?systemSymbol[s_Symbol[___], ___] :>
If[Context[s] === "System`", RuleDelayed[p, s],
RuleDelayed[p, "NonWolfram"]], {3, Infinity}]
]
Just loop this over all the files to get the data, and then
(* if you want to keep them by file *)
results = AssociationThread[fileNames, expressionData]
(* or *)
data = Flatten[expressionData]
Explore System symbol relationships
(* Tally over the keys (symbols used as Heaed) *)
tally = Tally[data[[;; , 1]]];
If we just look at the symbols by occurrence, we see that unsurprisingly List
is the most prominent:
We can view the interaction network as well, where the vertices are colored and sized by occurrence.
ToString[#[[1]]] \[DirectedEdge] ToString[#[[2]]] & /@ data;
The light blue dot near the center is List
.
This isn't all that informative, but we can make the frequency distribution of the symbols
NormalizeAssociation[assoc_] :=
With[{tots = Total[assoc]}, Reverse[Sort[Map[N[#/tots] &, assoc]]]]
(* Occurances of symbols *)
symOccur = Association @@ Rule @@@ tally;
(* Probability of symbols *)
symProb = NormalizeAssociation[symOccur];
and then the conditioned distribution
(* Conditional occurances of symbols *)
symCondOccur =
Map[Association @@ Rule @@@ Tally[#[[;; , 2]]] &,
GroupBy[Rule @@@ data, First]];
(* Conditional probability of symbols *)
symCondProb = Map[NormalizeAssociation, symCondOccur];
with this we can traverse our network (in this case we take only the most likely at each step, which may not be the best approach overall)
nextSymbol[symbol_] := Module[
{
dist, max, sel, key
},
If[symbol == Nothing, Return[Nothing]];
dist = symCondProb[symbol];
(* Maybe we never saw this symbol as head in our data *)
If[MissingQ@dist, Return[Nothing]];
max = Max[symCondProb[symbol]];
sel = Select[dist, max == # &];
(* Likewise perhaps selection failed as it was not connected to anything *)
If[sel == <||>, Return[Nothing]];
key = First@Keys[sel];
(* Since we always take max,
if key\[Equal]symbol we will go in circles *)
If[key == symbol, Return[Nothing]];
Return[key];
]
```
Now we can see some promising results:
NestList[nextSymbol[#] &, Range, 5]
``
yields:
{Range, Length, Part}`
NestList[nextSymbol[#] &, Import, 4]
yields: {Import, StringJoin, NotebookDirectory, EvaluationNotebook}
so it seems people tend to write Range[Length[Part[...]]]
e.g. Range@Length@myData[[;;,1]] and Import[StringJoin[NotebookDirectory[],...]]
Whats next?
Next it would be interesting to throw these notebooks into a language model.
Code
Attached is the code for crawling, everything else of relevance is above.
Attachments: