Message Boards Message Boards

2
|
18251 Views
|
33 Replies
|
29 Total Likes
View groups...
Share
Share this post:

Could Wolfram Language construct a network diagram of its functionality?

POSTED BY: Frank Kampas
33 Replies
Sort By:

I cannot believe it! They made what I asked for a few months ago (i.e. WolframLanguageData[])..

Now I wish I could extend the meta-data system to create AntelopeData[] (or MySpecialData[]). That would extend the meta-data system itself. The next request after that would be to extend one of the datasets, such as the ability to add chemical compounds to ChemicalData[] or the like!

Anyways, thanks WRI team!
POSTED BY: Aeyoss Antelope

Great stuff. The LinkTrails property does not appear to be documented fully, though. It is used in ref/WolframLanguageData but it is not one of the listed properties in the details section.
POSTED BY: Seth Chandler

Due to release of new functionality WolframLanguageData there is another way of doing it. Let's take a look at only Graphics functions (one can generalize). Draw a directed network illustrating connectivity of Wolfram Language documentation reference (leaf nodes) and guide (internal nodes) pages:

graphicssym = EntityList[EntityClass["WolframLanguageSymbol", {"FunctionalityArea", "GraphicsFunctions"}]];

linktrails = EntityValue[graphicssym, 
   EntityProperty["WolframLanguageSymbol", "LinkTrails"]];

edges = Flatten[(DirectedEdge @@@ Partition[#, 2, 1]) & /@ 
    Select[Flatten[linktrails, 1], Length[#] > 1 &]];

verts = Flatten[List @@@ edges] // Union;

Graph[edges, VertexLabels -> Placed["Name", Tooltip], 
 VertexStyle -> ((# -> If[StringMatchQ[#, "*ref/*"], Red, Green]) & /@verts)]

enter image description here

Also fun related things:

WolframLanguageData["Cos", "RelationshipCommunityGraph"]

enter image description here
POSTED BY: Vitaliy Kaurov
POSTED BY: Frank Kampas

I expected this much since just the valid options for Graphics3D is probably a huge graph by itself.

If you include the options for all the functions and all of the constants (which any large system has), you get this type of graph. Anybody ever heard of PlotRangeClipping? Ever used it?

It would be no different than if you see the anatomy of a human elbow, with all of the skin, blood vessels, musculature, lymph nodes, bone, all displayed at once on one graph. Actually it would be very tough for most layman to understand. But if you could select a location in the elbow (maybe where it hurts) and through views or overlays, you could expose or mask different levels of detail, you might have a chance to find where to look for the root cause.

That's why what we need for this is: a set of annotated functions (not including options, constants) with a way to relate/group these and a way to traverse these groups

This is why I wanted to strip the constants (starts with '$') and all the options. I guess the options are in the See Also lists. :-(
POSTED BY: Aeyoss Antelope

Yes, what we're seeing is the infrastructure. I had no idea there was so much of it.
POSTED BY: Frank Kampas

This should be taken over by WRI and called MathematicaFunctionsData[]..
POSTED BY: Aeyoss Antelope
POSTED BY: John Doty

Marco, Bernat, I'm really impressed by what you've been able to do.
POSTED BY: Frank Kampas

Thanks for doing this Marco! I believe some of this data will help me learn more Mathematica...
POSTED BY: Aeyoss Antelope
Attachments:
POSTED BY: Marco Thiel

Marco, this is amazing :-) Attached is an Interactive Graph of the Wolfram Language Documentation Center that I've created by adding tooltips and hyperlinks to your first CommunityGraph:

g = Graph[Map[Hyperlink[Tooltip[#, DirectedEdge[#[[1]], #[[2]]] &@Flatten[StringCases[{First[#], Last[#]}, "http://reference.wolfram.com/language/ref/" ~~ x__ ~~ ".html" -> x]]], Last[#]] &, DeleteDuplicates[links]]];
communities = FindGraphCommunities[g];
CommunityGraphPlot[g, communities]

Interactive Documentation Center Graph

This is an excellent way to explore and learn about new functions.

Enjoy!

Attachments:
POSTED BY: Bernat Espigulé
POSTED BY: John Doty

Actually this is (still) not correct...it needs a custom NearestFunction[]....
POSTED BY: Aeyoss Antelope

This should come out using WolframLanguageData[Map, SeeAlsoList]
POSTED BY: Aeyoss Antelope

OK Frank and John:

It seems obvious that Mma cannot possibly attach any semantic meaning to the 3 leters "Map". It only looks up the 3-character symbol in the current Context and begins replacements.

But with all languages, syntax (which Mma is very concerned with) and semantics are in different domains. Certainly as we all work to learn more and more of Mma, the semantic meaning of various functions can be grouped in our minds as "closer" or "farther".

I got stuck on the idea of a visualization of this.

It is interesting to me that Wolfram wants everything to be "computable" but the help system of Mma itself is not computable. Maybe they can make WolframLanguageData[]. But the web is (computable) and Mma's help system is out on the web.

So I posit that a first approximation to this visualization is the "See Also" list of the Mma "man pages" on the web.

Inspired by the example I cited above, I have "sketched some code" in an attached notebook that:

Import[] the man page from the web (e.g. Import["http://reference.wolfram.com/language/ref/Map.html"] )

strips out the "See Also" list of functions

creates Graph[] rules

generates a network for that function

Notes:

Since "See Also" is human-curated, it probably also has limitations.

To run against all the functions, that code needs more error checking.

I could not figure out how to use FindClusters[], since I am not sure how to graph the functions names in 2D.

HTH.

Attachments:
POSTED BY: Aeyoss Antelope

Interesting. This is definitely along the lines I was thinking.
POSTED BY: Frank Kampas

It seems obvious that Mma cannot possibly attach any semantic meaning to the 3 leters "Map". It only looks up the 3-character symbol in the current Context and begins replacements.

Map is a perfect illustration of the difficulty. It's a pure expression rewriter. It certainly has semantics, but in:

Map[head, tails]

as far as I know, there are no restrictions on head and tails: any expression works. Therefore, Map is connected to the entire language.
POSTED BY: John Doty
POSTED BY: Frank Kampas

My guess is that WRI has the documentation in some custom format that could easily be converted to some "computable" format. Maybe this is not prioritized.
POSTED BY: Aeyoss Antelope

I would expect that all the documentation was developed in Mathematica.
POSTED BY: Frank Kampas

To me, if it were, Information[] would emit something other than null.
POSTED BY: Aeyoss Antelope

Hello John:

Actually I completely agree with you.

But in my mind, the list of Mma functions I know are categorized with some custom meta-data, so that Map[] is closer to Apply[] than it is to NComputeChebyshevDoubleIntegral[]. For sure, you can do Map[] using this non-existent function.

Using this "See Also" metric over the space of functions is far from optimal. But a (correct) visualization using this metric would (I guess) be illuminative.
POSTED BY: Aeyoss Antelope

A network of the cross-references in the documentation won't help you understand the language itself. How could it make sense of the following?

ffmod[g12_][f_][state[n_, v_, {q1_, q2_}, out_]] := 
 state[n + 1, f[n], {q1 + v - b2q[out], q2 + g12*q1}, q2b[q2 + q1]]

This is part of a simulation of a delta-sigma modulator as an inhomogeneous iterated function system. But you can deconstruct it from the "everything is an expression" viewpoint. Note that b2q and q2b are defined functions, while state is an undefined head.
POSTED BY: John Doty

OK Frank and John:

It seems obvious that Mma cannot possibly attach any semantic meaning to the 3 leters "Map". It only looks up the 3-character symbol in the current Context and begins replacements.

But with all languages, syntax (which Mma is very concerned with) and semantics are in different domains. Certainly as we all work to learn more and more of Mma, the semantic meaning of various functions can be grouped in our minds as "closer" or "farther".

I got stuck on the idea of a visualization of this.

It is interesting to me that Wolfram wants everything to be "computable" but the help system of Mma itself is not computable. Maybe they can make WolframLanguageData[]. But the web is (computable) and Mma's help system is out on the web.

So I posit that a first approximation to this visualization is the "See Also" list of the Mma "man pages" on the web.

Inspired by the example I cited above, I have "sketched some code" in an attached notebook that:

Import[] the man page from the web (e.g. Import["http://reference.wolfram.com/language/ref/Map.html"] ) strips out the "See Also" list of functions creates Graph[] rules generates a network for that function

Since "See Also" is human-curated, it probably also has limitations.

To run against all the functions, that code needs more error checking.

I could not figure out how to use FindClusters[], since I am not sure how to graph the functions in 2D.

HTH.

Attachments:
POSTED BY: Aeyoss Antelope

I hear what you are saying, the replies stating that Mathematica looks for patterns in EXPRESSIONS doesn't mean it can't get from T initai to T final by way of another mesh all points in the complex are connected.
POSTED BY: jonathan lister

Consider:

Sin[IntegerQ[MatrixForm]]

Now, these symbols have no intended connection. Nevertheless, Mathematica accepts and even partially rewrites the expression:

Sin[False]

And, of course, there are several ways you could coax Mathematica to rewrite that if you choose.

Most expressions that fit the form head[sequence] are legitimate in Mathematica. Even many that throw errors are OK when wrapped in Quiet. So a "network diagram" for the "language" would be uninformative, since it's very nearly a fully connected graph. I suppose you could make a weighted diagram for somebody's specific programming style, but I expect that would mostly illuminate bad programming. Using Mathematica's power involves forging connections.
POSTED BY: John Doty

I thought the original goal of the question was to find some meta-data or annotation for each function that would allow a graph of semantically clustered functions to be constructed. Using that meta-data and Nearest[] could facilitate a classification and/or network of those functions..further down that path would be tool-tips with data from Information[] and links into the documentation. So far I cannot see any such meta-data.
POSTED BY: Aeyoss Antelope

The semantic clustering you seek does not exist, and seeking it will simply mislead you about the nature of Mathematica. "Everything is an expression." Start there.
POSTED BY: John Doty
POSTED BY: Aeyoss Antelope

Yes, the definitions don't provide much more information than the name.
POSTED BY: Frank Kampas

Turns out there are a lot of pattern tests:

In[3]:= Names["*Q"]

Out[3]= {"AcyclicGraphQ", "AlgebraicIntegerQ", "AlgebraicUnitQ", \
"AntihermitianMatrixQ", "AntisymmetricMatrixQ", "ArgumentCountQ", \
"ArrayQ", "AssociationQ", "AtomQ", "BinaryImageQ", "BipartiteGraphQ", \
"BooleanQ", "BoundaryMeshRegionQ", "BoundedRegionQ", "BusinessDayQ", \
"ByteArrayQ", "ColorQ", "CompatibleUnitQ", "CompleteGraphQ", \
"CompositeQ", "ConnectedGraphQ", "ConstantRegionQ", \
"ContinuousTimeModelQ", "ControllableModelQ", "CoprimeQ", \
"DateObjectQ", "DaylightQ", "DayMatchQ", "DeviceOpenQ", \
"DiagonalizableMatrixQ", "DigitQ", "DirectedGraphQ", "DirectoryQ", \
"DiscreteTimeModelQ", "DisjointQ", "DispatchQ", \
"DistributionParameterQ", "DuplicateFreeQ", "EdgeCoverQ", "EdgeQ", \
"EllipticNomeQ", "EmptyGraphQ", "EulerianGraphQ", "EvenQ", \
"ExactNumberQ", "FileExistsQ", "FreeQ", "FrontEndSharedQ", \
"GeoWithinQ", "GraphQ", "GroupElementQ", "HamiltonianGraphQ", \
"HermitianMatrixQ", "HypergeometricPFQ", "ImageInstanceQ", "ImageQ", \
"IndefiniteMatrixQ", "IndependentEdgeSetQ", "IndependentVertexSetQ", \
"InexactNumberQ", "IntegerQ", "IntersectingQ", "IntervalMemberQ", \
"InverseEllipticNomeQ", "IrreduciblePolynomialQ", "IsomorphicGraphQ", \
"JavaObjectQ", "KEdgeConnectedGraphQ", "KernelSharedQ", "KeyExistsQ", \
"KeyFreeQ", "KeyMemberQ", "KnownUnitQ", "KVertexConnectedGraphQ", \
"LeapYearQ", "LegendreQ", "LetterQ", "LinkConnectedQ", "LinkReadyQ", \
"ListQ", "LoopFreeGraphQ", "LowerCaseQ", "MachineNumberQ", \
"ManagedLibraryExpressionQ", "MandelbrotSetMemberQ", "MarcumQ", \
"MatchLocalNameQ", "MatchQ", "MatrixQ", "MemberQ", "MeshRegionQ", \
"MixedGraphQ", "MultigraphQ", "NameQ", "NegativeDefiniteMatrixQ", \
"NegativeSemidefiniteMatrixQ", "NormalMatrixQ", "NumberQ", \
"NumericQ", "ObservableModelQ", "OddQ", "OptionQ", "OrderedQ", \
"OrthogonalMatrixQ", "OutputControllableModelQ", "PacletNewerQ", \
"PartitionsQ", "PathGraphQ", "PermutationCyclesQ", \
"PermutationListQ", "PlanarGraphQ", "PolynomialQ", \
"PositiveDefiniteMatrixQ", "PositiveSemidefiniteMatrixQ", \
"PossibleZeroQ", "PrimePowerQ", "PrimeQ", "PrintableASCIIQ", \
"ProcessParameterQ", "QHypergeometricPFQ", "QuadraticIrrationalQ", \
"QuantityQ", "RegionQ", "RegularlySampledQ", "RootOfUnityQ", \
"SameObjectQ", "SameQ", "SatisfiableQ", "SimpleGraphQ", \
"SquareFreeQ", "SquareMatrixQ", "StringContainsQ", "StringEndsQ", \
"StringFreeQ", "StringMatchQ", "StringQ", "StringStartsQ", "SubsetQ", \
"SymmetricMatrixQ", "SyntaxQ", "TautologyQ", "TensorQ", \
"TimeObjectQ", "TreeGraphQ", "TrueQ", "UnateQ", "UndirectedGraphQ", \
"UnitaryMatrixQ", "UnsameQ", "UpperCaseQ", "URLExistsQ", "ValueQ", \
"VectorQ", "VertexCoverQ", "VertexQ", "WeaklyConnectedGraphQ", \
"WeightedGraphQ"}

In[4]:= Length[%]

Out[4]= 157

I haven't figured out how get their definitions all at once. The following doesn't work:

(? #)& /@ Names["*Q"]
POSTED BY: Frank Kampas

If I wanted this, I would approach this like the following.

  1. get all of the names in the system...use something like words=Names["System`"];

  2. strip out all of the ones that begin with '$'

  3. use this graph code in the man page here: http://reference.wolfram.com/language/ref/Graph.html

(* Generate a network of "nearby" words in a dictionary *)

Flatten[Map[(Thread[# \[DirectedEdge] DeleteCases[Nearest[words, #, 3], #]]) &, words]];

Flatten[Map[(Thread[# \[DirectedEdge] DeleteCases[Nearest[words, #, 3], #]]) &, words]];

Graph[%, VertexLabels -> "Name", ImageSize -> 450]

To me what would tell more is a different Nearest[] that (somehow) shows near in functionality.

HTH.
POSTED BY: Aeyoss Antelope
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract