Could Wolfram Language construct a network diagram of its functionality?

[antelope (Aeyoss Antelope)]

To me, if it were, Information[] would emit something other than null.

[jpd (John Doty)]

A network of the cross-references in the documentation won’t help you understand the language itself. How could it make sense of the following?

ffmod[g12_][f_][state[n_, v_, {q1_, q2_}, out_]] := 
 state[n + 1, f[n], {q1 + v - b2q[out], q2 + g12*q1}, q2b[q2 + q1]]

This is part of a simulation of a delta-sigma modulator as an inhomogeneous iterated function system. But you can deconstruct it from the “everything is an expression” viewpoint. Note that b2q and q2b are defined functions, while state is an undefined head.

[mthiel (Marco Thiel)]

Hi Everyone,

I also had the idea of using the documentation:

http://reference.wolfram.com/language/guide/AlphabeticalListing.html

It is easy to import all commands from that website:

helppages = Select[Import["http://reference.wolfram.com/language/guide/AlphabeticalListing.html", "Hyperlinks"], StringMatchQ[#, "http://reference.wolfram.com/language/ref/" ~~ __ ~~ ".html"] &];

The following function imports all the links to other functions in the “See Also” section. Please do not run this function to keep the load on Wolfram Inc’s servers as low as possible. The data is attached below to this post. Sorry to Wolfram Inc for downloading all this, it was not a DoS attack

Monitor[For[k = 1, k <= Length[helppages], k++, dummy = URLFetch[helppages[[k]]]; 
  AppendTo[links, helppages[[k]] \[DirectedEdge] "http://reference.wolfram.com/language/ref/" <> # <> ".html" & /@
     If[Length[StringSplit[dummy, "See Also"]] > 1, (StringSplit[StringJoin[StringSplit[StringSplit[StringSplit[StringSplit[dummy, "See Also"][[2]], "Related Guides"][[1]], "href=\""], "\"><span"]], {"/language/ref/", ".html"}][[2 ;; ;; 2]]), {}]]], N[k/Length[helppages]]]

It is a good idea to save the data:

Export["~/Desktop/MMA-funcs.txt", links]

The following lines work to import the data again from that file:

data = Import["~/Desktop/MMA-funcs.txt", "Plaintext"];
links = ToExpression@Delete[Delete[StringReplace[StringReplace[(# <> "]" & /@ (StringReplace[
          StringReplace[StringSplit[StringReplace[data, "}\n{" -> ","], "],"], {"{Directed" -> "Directed", "]}" -> "]"}], 
          "]" ~~ ___ ~~ "Directed" -> "], Directed"])),  ",Directed" -> "Directed"], "]]" -> "]"], 17648], 17647]

I will attach the txt-file to this post for everyone to play with. We can now graph the “relationship-graph” of all Mathematica functions.

g = Graph[DeleteDuplicates[links]]

One interesting thing is to calculate the Communities:

communities = FindGraphCommunities[g];

We can look at a couple of the smaller ones to see whether the network makes intuitive sense:

Grid[Table[
  StringSplit[StringSplit[#, "ref/"][[2]], ".html"][[1]] & /@ communities[[k]], {k, -25, -1}], Frame -> All]

Every row corresponds to one community. It appears that the communities make sense. The first row for examples is related to Graph commands. We can also plot the CommunityGraph:

CommunityGraphPlot[g, communities]

We can now also try to gauge the importance of different functions, here the 20 most important ones, for example using the BetweenessCentrality:

Grid[Reverse[SortBy[Transpose[{vertices, BetweennessCentrality[g]}], #[[2]] &]][[1 ;; 20]], Frame -> All]

We can compare that to the VertexDegree:

Grid[Reverse[SortBy[Transpose[{vertices, VertexDegree[g]}], #[[2]] &]][[1 ;; 20]], Frame -> All]

or else the PageRankCentrality:

Grid[Reverse[SortBy[Transpose[{vertices, PageRankCentrality[g, 0.85]}], #[[2]] &]][[1 ;; 20]], Frame -> All]

We see that in each of the cases “Graph” ranks very high. Of course we can now also look for specific functions like so:

Part[PageRankCentrality[g, 0.85], VertexIndex[g, "http://reference.wolfram.com/language/ref/" <> # <> ".html"]] & /@ {"ColorSeparate"}
(*{0.000265413}*)

Just replace the “ColorSeparate” by your favourite function. Another thing that I found interesting was the word cloud of all functions. I basically collect all instances of a function being an in- or out-vertex and applying the WordCloud function:

WordCloud[StringSplit[StringSplit[#, "/ref/"][[2]], ".html"][[1]] & /@ Flatten[{links[[All, 1]], links[[All, 2]]}]]

It was interesting for me that “Automatic” was so prominent. This becomes clear if we look at the VertexInDegree.

VertexInDegree[g, "http://reference.wolfram.com/language/ref/Automatic.html"]
(*93*)

as opposed to

VertexInDegree[g, "http://reference.wolfram.com/language/ref/Graph.html"]
(*76*)

and

VertexInDegree[g, "http://reference.wolfram.com/language/ref/Plot3D.html"]
(*14*)

I am quite aware that this post does not address the initial question, but I hope it is of some interest anyway.

Cheers,

M.

PS: Of course, you can now study everything hierarchically. For example we can take the largest Community:

g2 = Graph[Select[links, MemberQ[communities[[1]], #[[2]]] &]];

determine its Communities:

communities2 = FindGraphCommunities[g2];

and plot:

CommunityGraphPlot[g2, communities2]

Here are members of the respective sub-Communities:

StringSplit[StringSplit[communities2[[1]], "/ref/"][[All, 2]], ".html"][[All, 1]]

{“ClickPane”, “AngularGauge”, “ClockGauge”, “VerticalGauge”,
“Slider”, “VerticalSlider”, “ProgressIndicator”, “Locator”,
“Slider2D”, “ColorSlider”, “Control”, “LocatorPane”,
“ControlsRendering”, “Animate”, “ListAnimate”, “Manipulate”,
“Animator”, “Dynamic”, “AutorunSequencing”, “ActionMenu”,
“ButtonBar”, “PopupMenu”, “DockedCells”, “Bookmarks”,
“AnimationDirection”, “Arrow”, “Inset”, “Arrowheads”,
“AnimationRunning”, “Trigger”, “Pause”, “Manipulator”,
“BaselinePosition”, “Antialiasing”, “Rasterize”, “AnySubset”,
“FormFunction”, “FormObject”, “CheckboxBar”, “TogglerBar”,
“ListPicker”, “AppearanceRules”, “FormLayoutFunction”, “Full”,
“AutoSpacing”, “TextJustification”, “Alignment”, “TextAlignment”,
“AlignmentPoint”, “Item”, “ControlPlacement”, “Axes”, “AxesLabel”,
“Frame”, “PlotLabel”, “FrameLabel”, “Background”, “ImageMargins”,
“Pane”, “Backward”, “Bottom”, “MenuView”, “BarSpacing”, “Spacings”,
“CellBaseline”, “Blend”, “BulletGauge”, “Setter”, “RadioButton”,
“Panel”, “SetterBar”, “TabView”, “ContentPadding”, “FrameMargins”,
“ImageSize”, “GestureHandler”, “EventHandler”, “Refresh”,
“MousePosition”, “Labeled”, “ControlType”, “CellMargins”,
“CellBracketOptions”, “CellDingbat”, “CellFrame”,
“CellDynamicExpression”, “CellEventActions”, “NotebookEventActions”,
“FrontEndEventActions”, “CellFrameLabels”, “CellFrameMargins”,
“CellFrameColor”, “CellFrameLabelMargins”, “CellLabelMargins”,
“OpenerView”, “ColorSetter”, “ColumnAlignments”, “ItemSize”,
“ChartLabels”, “Checkbox”, “Toggler”, “Opener”, “RadioButtonBar”,
“ConstantImage”, “FieldSize”, “ContentSize”, “Update”, “Darker”,
“DataRange”, “Dividers”, “Delimiter”, “FinishDynamic”, “Setting”,
“DynamicWrapper”, “FormTheme”, “Forward”, “ForwardBackward”,
“Framed”, “FrameTicks”, “FrameBox”, “FrameBoxOptions”, “BoxFrame”,
“FlipView”, “SlideView”, “PaneSelector”, “ImageSizeAction”,
“GeoMarker”, “GaugeFaceElementFunction”, “GaugeFaceStyle”,
“GaugeFrameElementFunction”, “GaugeMarkers”, “HorizontalGauge”,
“GaugeFrameStyle”, “GaugeStyle”, “ScaleRangeStyle”, “GrayLevel”,
“TouchPosition”, “GridLines”, “GeoBackground”, “ThermometerGauge”,
“IntervalSlider”, “Hue”, “IconData”, “ImageAspectRatio”, “ImageCrop”,
“ImageDimensions”, “InitializationCell”, “ImageFormattingWidth”,
“$ImageFormattingWidth”, “ImageResolution”, “PageWidth”,
“format/JPEG”, “format/GIF”, “ImagePadding”, “Text”, “Magnify”,
“ImageResize”, “RasterSize”, “Magnification”, “ImageScaled”,
“ImageSizeMultipliers”, “PixelConstrained”, “ItemStyle”,
“Scrollbars”, “ItemAspectRatio”, “LabelingFunction”, “ListPickerBox”,
“ScrollPosition”, “Large”, “Lighter”, “LocatorAutoCreate”,
“LocatorRegion”, “Left”, “LineIndent”, “LineIndentMaxFraction”,
“Legended”, “LegendFunction”, “LegendMargins”, “TouchscreenAutoZoom”,
“MinIntervalSize”, “Medium”, “NotebookDynamicExpression”,
“Overscript”, “OverscriptBox”, “PackingMethod”, “Placed”,
“PreserveImageOptions”, “PlotMarkers”, “PlotRangeClipping”,
“PlotRangePadding”, “PlotRegion”, “Point”, “PassEventsDown”,
“PassEventsUp”, “PolarAxes”, “PolarTicks”, “RGBColor”, “Right”,
“Raster”, “RotateLabel”, “RotationAction”, “RoundingRadius”,
“RowAlignments”, “Rectangle”, “Scale”, “Small”, “ScaleRanges”,
“ScalingMatrix”, “ScriptBaselineShifts”, “ScriptSizeMultipliers”,
“Set”, “SetDelayed”, “SynchronousInitialization”,
“SynchronousUpdating”, “TouchscreenControlPlacement”, “Ticks”,
“TrackedSymbols”, “TrackingFunction”, “Tiny”, “Top”, “Underscript”,
“UpdateInterval”, “WordCloud”, “WordOrientation”, “XYZColor”}

Using DeleteDuplicates on the links cleans up the graph considerably:

[antelope (Aeyoss Antelope)]

Thanks for doing this Marco! I believe some of this data will help me learn more Mathematica…

[bernatep (Bernat Espigulé)]

Marco, this is amazing Attached is an Interactive Graph of the Wolfram Language Documentation Center that I’ve created by adding tooltips and hyperlinks to your first CommunityGraph:

g = Graph[Map[Hyperlink[Tooltip[#, DirectedEdge[#[[1]], #[[2]]] &@Flatten[StringCases[{First[#], Last[#]}, "http://reference.wolfram.com/language/ref/" ~~ x__ ~~ ".html" -> x]]], Last[#]] &, DeleteDuplicates[links]]];
communities = FindGraphCommunities[g];
CommunityGraphPlot[g, communities]

This is an excellent way to explore and learn about new functions.

Enjoy!

[fkampas (Frank Kampas)]

Marco, Bernat,
I’m really impressed by what you’ve been able to do.

[antelope (Aeyoss Antelope)]

This should be taken over by WRI and called MathematicaFunctionsData[]..

[jpd (John Doty)]

Where’s Mathematica in this? What I see is almost entirely rarely used symbols from what used to be the “add-ons”. Mathematica used to be factored rather than presented to the user as a big flat lump. These graphics are an excellent illustration of the fog that now obscures the core functionality. If you want to master Mathematica, this is the stuff you should generally ignore. Only use it when it’s applicable to your very specific, specialized problem.

[fkampas (Frank Kampas)]

Yes, what we’re seeing is the infrastructure. I had no idea there was so much of it.

[antelope (Aeyoss Antelope)]

I expected this much since just the valid options for Graphics3D is probably a huge graph by itself.

If you include the options for all the functions and all of the constants (which any large system has), you get this type of graph. Anybody ever heard of PlotRangeClipping? Ever used it?

It would be no different than if you see the anatomy of a human elbow, with all of the skin, blood vessels, musculature, lymph nodes, bone, all displayed at once on one graph. Actually it would be very tough for most layman to understand. But if you could select a location in the elbow (maybe where it hurts) and through views or overlays, you could expose or mask different levels of detail, you might have a chance to find where to look for the root cause.

That’s why what we need for this is: a set of annotated functions (not including options, constants) with a way to relate/group these and a way to traverse these groups

This is why I wanted to strip the constants (starts with ‘$’) and all the options. I guess the options are in the See Also lists.

[fkampas (Frank Kampas)]

When I was teaching Mathematica years ago, I started out with numbers, lists, symbols, patterns, rules, and functions, which I regarded as the fundamental components. I don’t know how one would extract that sort of thing from the documentation.

[Vitaliyk (Vitaliy Kaurov)]

Due to release of new functionality WolframLanguageData there is another way of doing it. Let’s take a look at only Graphics functions (one can generalize). Draw a directed network illustrating connectivity of Wolfram Language documentation reference (leaf nodes) and guide (internal nodes) pages:

graphicssym = EntityList[EntityClass["WolframLanguageSymbol", {"FunctionalityArea", "GraphicsFunctions"}]];

linktrails = EntityValue[graphicssym, 
   EntityProperty["WolframLanguageSymbol", "LinkTrails"]];

edges = Flatten[(DirectedEdge @@@ Partition[#, 2, 1]) & /@ 
    Select[Flatten[linktrails, 1], Length[#] > 1 &]];

verts = Flatten[List @@@ edges] // Union;

Graph[edges, VertexLabels -> Placed["Name", Tooltip], 
 VertexStyle -> ((# -> If[StringMatchQ[#, "*ref/*"], Red, Green]) & /@verts)]

Also fun related things:

WolframLanguageData["Cos", "RelationshipCommunityGraph"]

[schandler (Seth Chandler)]

Great stuff. The LinkTrails property does not appear to be documented fully, though. It is used in ref/WolframLanguageData but it is not one of the listed properties in the details section.

[antelope (Aeyoss Antelope)]

I cannot believe it! They made what I asked for a few months ago (i.e. WolframLanguageData[])..

Now I wish I could extend the meta-data system to create AntelopeData[] (or MySpecialData[]). That would extend the meta-data system itself. The next request after that would be to extend one of the datasets, such as the ability to add chemical compounds to ChemicalData[] or the like!

Anyways, thanks WRI team!