Message Boards Message Boards

Automatic documentation generation and distribution

Posted 6 years ago

This is a post I initially wrote elsewhere, but which I'm porting here as it shows once more why I think subdomains and normal web operations in the Wolfram Cloud are important features (and which I imagine should not be much work to implement)


This post is going to be long on design and relatively short on code. As usual, it's exposition of something I spent a while developing and have cooked into one of my packages .

I'm going to talk about how to make and distribute documentation in Mathematica, with specific emphasis on the automatic generation of documentation.

Documentation Overview

To start, though, we'll break down the types of things that we find in the documentation and which we'll want to support in a package. At this point there are about 10 different types of documentation formats out there:

  • Symbol pages

  • Guide pages

  • Tutorial pages

  • Message pages

  • Format pages

  • Service Connection pages

  • HowTos

  • Workflows

  • Overviews

Of these, there are really only 4 distinct types of things that we see in the documentation folders:

  • Reference Pages

  • Guides

  • Tutorials

  • Workflows

And of these I opted to only implement the first three as they are the ones I use the most.

Reference Pages (Symbols)

Ref pages come in a few different flavors, but the most common one, and the most common type of documentation in general, is the symbol page.

As of version 11.1 symbol pages look like this:

Rasterize@Documentation`HelpLookup["ref/StatusArea"]

(*Out:*)

making-mathematica-documentation-with-mathematica-526634225825234956

Which can be split into 5 major parts

  • Header bar

  • Usage table

  • Details

  • Examples

  • Related Links

We'll need to include each of these when we build our own docs

Guide Pages

Guides have four parts. In the interest of space I won't show an actual image of a guide, but you can see one via:

Documentation`HelpLookup["guide/PlottingOptions"]

These parts are

  • Header bar

  • Title and abstract

  • Function listing

  • Related links

Tutorials

Tutorials are so much more flexible than guides of symbol pages that it only really makes sense to discuss three sections, with an optional fourth

  • Header bar

  • Jump-links (optional)

  • Tutorial content

  • Related links

This flexibility makes them both easier and harder to handle.

Generating Documentation

Now that we know what kinds of things we need to include we can move to how to include them. Before jumping into the actual code, though, it's worth noting that WRI does provide a tool for building docs.

Workbench

Workbench is Wolfram's primary IDE. It's a plugin to Eclipse

Why not just use Workbench?

There's no absolute reason not to use Workbench. Indeed it's probably got the lowest barrier to entry given that it's semi-battle-tested by WRI.

On the other hand, using Workbench restricts ones possibilities. It doesn't always stay up-to-date (as of writing this the documentation it builds by default is still on version 11.0) and by using it we lose the edge of our knowledge of Mathematica.

In general, Mathematica will be the best tool for manipulating Mathematica documents, so my view is why not simply provide a Mathematica package to do it?

DocGen

The package that I developed is called DocGen (inspired by, but not nearly as clever as, doxygen ). It uses my larger BTools toolchain to provide extra linkages to the entire Mathematica ecosystem.

Documentation Templates

The first place any documentation starts is as a documentation template. This is simply a notebook with cell-types attached that will be post-processed into a full documentation notebook. We can make a new one for a Symbol Page like:

(* to load DocGen *)
<<BTools`Paclets`
DocGen["SymbolPage", MyFunction, Method->"Template"]//CreateDocument;

(*Out:*)

making-mathematica-documentation-with-mathematica-7939871188899330678

We can simply type in this notebook to add content. I'll fille this out, then generate it and we'll see what happens:

making-mathematica-documentation-with-mathematica-5746074635887181910

We can see that for the most part it looks the same, but now the notebook is formatted properly for use with the Documentation Center

A similar workflow is implemented for guides and tutorials

Automatic Generation

This by itself doesn't give us much of a leg-up on Workbench. In fact, this template system may even be a little bit worse than Workbench's DocuTools (although significantly less bloated and quicker to use).

What does make this powerful is how it allows us to now automatically generate documentation, as all we need to do it extract parameters from a Symbol or context and feed them into this type of template.

The actual details behind this can be a bit gory, but you can read about them in my post on StackExchange . In general we simply take the Symbol and extract the usage patterns to fill out the usage table, the calling patterns to fill out the details, make a few sample Examples based on the calling patterns, and provide See Also links based on camel-case similarity in the first hump.

This is what DocGen does by default, so for instance we can do:

DocGen@DocGen

making-mathematica-documentation-with-mathematica-6455205443540996861

And we get a fully-functional documentation notebook automatically

We can take this further, though, and do the same for an entire set of contexts to link a package or multiple packages together:

DocGen@
 {
  "BTools`Paclets`","BTools`FrontEnd`", 
  "BTools`Web`", "BTools`External`",
  "BTools`"
  }

(*Out:*)

making-mathematica-documentation-with-mathematica-5587525071093655299

This gives us a really powerful way to provide accessible documentation with a minimum of effort. In all, that makes it much more likely that the documentation will actually get made.

This also rewards good package design as the better ones definition patterns are, the clearer the documentation built will be.

Distributing Documentation

Simply building the documentation isn't enough, though. Good documentation should serve as an advertisement for one's package. So the next thing to do is design a distribution system that allows us to publicize and distribute our documentation effectively. To do that we'll want to start with building some paclets for our docs.

Documentation Paclets

Paclets

I've talked about paclets before on a number of occasions, so I won't go into depth on them now, but if you want a refresher you can look here .

When we build our documentation paclets, we'll want to make them to have four properties:

  • They are obviously documentation, not code

  • They do not interfere or interact with the package they document in anyway

  • They are modularized as much as possible

  • They can be easily updated and versioned

The last one is easy using the paclet manager and a paclet server. The second to last can be done easily if packages are well partitioned into subcontexts. The first and second, however, are a little bit trickier.

If we have a paclet call "MyPaclet" and we wanted to distribute its documentation separately, we couldn't simply call the documentation paclet "MyPaclet" as well. Instead, we'll follow suit with what WRI does for many of its subpaclets, such as "ServiceConnections" and curated data and append a qualifier to the paclet name. So instead of "MyPaclet" we'll call it "Documentation_MyPaclet" to make it obvious where it comes from.

The issue with this is that it breaks our simple lookup procedure, but happily it's simple enough to fix this. In the "Documentation" extension to a Paclet expression we find the option "LinkBase" . This specifies what the lookup-root for things in the paclet should be. For instance if there is a page at "Guides/MyPaclet" in our "Documentation_MyPaclet" paclet, by using "MyPaclet" as the "LinkBase" this page will resolve to "MyPaclet/guide/MyPaclet" on lookup, and so the documentation will behave as expected.

Overall, then, we'll have our Paclet expression look like:

Paclet[
 Name -> "Documentation_MyPaclet", 
 Version -> "1.0.0", 
 Extensions -> 
  {
    {
      "Documentation", 
      "Language" -> "English", 
      "LinkBase" -> "MyPaclet", 
      "MainPage" -> "Guides/MyPaclet"
      }
    }
  ]

This is something I go over here as well.

Paclet Server

With these documentation paclets in hand, we can go one step further and build a paclet server for our documentation (see the refresher link for paclets for a refresher on paclet servers, too). This will be an entirely generic paclet server, but it will serve as a way to easily share documentation in small chunks, just like curated data. I set up a server for all the documentation I've built here which looks like:

making-mathematica-documentation-with-mathematica-2355226512675681026

People can then install pieces of documentation from there, like:

PacletInstall[
 "Documentation_PacletManager",
 "Site"->
  "http://www.wolframcloud.com/objects/b3m2a1.docs/DocumentationServer"
 ]

(*Out:*)

making-mathematica-documentation-with-mathematica-2019233375110478896

And they'll be immediately ready to use in Mathematica.

Documentation Sites

One last thing to comment on is how we can take our documentation Notebooks and turn them into true HTML documentation which you can peruse on the web. This will involve taking some pieces out of Workbench (but which I've made accessible from a paclet server, so no worries if you don't want to download Workbench).

HTML Documentation

Workbench provides some facilities for generating HTML documentation. These facilities are (as of when I wrote this package) limited to 11.0-style documentation, but that's more than good enough for most things.

The main thing I needed to do was apply a thorough cleaning to the documentation pages I generated to make sure the finicky package that actually generates the HTML (called Transmogrify which is in turn called by a higher-level package called DocumentationBuild ) won't hang when it reaches a directive it doesn't know how to process.

After that, the main issue was simply making sure all the appropriate resources are deployed, and then we're good to go.

I've built this into DocGen as a method. So if you want to build out HTML documentation for a paclet or set of paclets you can do it like:

DocGen["HTML", PacletFind["Documentation_BTools*"]]

(*Out:*)

{
 {"~/Library/Mathematica/Applic"…"Gen/Web/BToolsWeb/guide/BToolsWeb.html",<<24>>,""…""},
 {"~/Library/Mathematica/Applic"…"Gen/Web/BToolsWeb/guide/BToolsWeb.html",<<24>>,""…""},
 {"~/Library/Mathematica/Appli"…"oolsExternal/guide/BToolsExternal.html",<<22>>,""…""},
 {"~/Library/Mathematica/Appli"…"oolsFrontEnd/guide/BToolsFrontEnd.html",<<40>>,""…""},
 {"~/Library/Mathematica/ApplicationData/DocGen/Web/BTools/guide/BTools.html"},
 {"~/Library/Mathematica/Applicat"…"Web/BToolsPaclets/ref/AppAddContent.html"}
 }

And this can be deployed to the web to use as documentation. If we want that we can simply run it with CloudDeploy->True and it will do so.

Alternately DocGen also support deploying built HTML automatically, which we can see in the "Methods" :

DocGen["HTML", "Methods"]

(*Out:*)

{Automatic,"Deploy"}

The best way to do this is passing the directory with all the HTML:

DocGen["HTML", PacletFind["Documentation_BTools*"], Method->"Deploy"];

Documentation Site

We can take this a step further though, and build a wrapper around this type of functionality to get it to upload

DocumentationSiteBuild["BuildOverview"->True, "AutoDeploy"->True];

This creates a site where we can browse all of the exposed documentation, much like the paclet server we had before:

making-mathematica-documentation-with-mathematica-7555384944798244364

And with that, I think, we're done.

POSTED BY: b3m2a1 ​ 
2 Replies

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming!

POSTED BY: Moderation Team
Posted 6 years ago

A follow up to the previous post. Originally here.


Here's a quick example of how we can use some of the stuff I showed in this post to build out full documentation by hand.

Getting DocGen

This post will be using my DocGen system which is built into a large package of mine that supports other parts of the development process as well. This package lives in a paclet on my paclet server. You can install it with:

PacletInstall["BTools",
 "Site"->""
 ]

If you want to learn more about making paclet servers you can read more here .

Documenting the Templating system

We're going to make use of a palette I designed for helping with the documentation system. After you've installed BTools you can open it by looking for BTools?DocGen in the palettes menu:

making-documentation-with-docgen-50323364875999283

Palette Overview

The entire palette looks like this:

making-documentation-with-docgen-7767653871764496452

If we look at it, it essentially has five sections:

making-documentation-with-docgen-4180400016615224181

The first of these chooses the paclet and lets us open the relevant config files. The second lets us open any built documentation pages for this context. The third actually generates the content and optionally deploys it to the cloud. The fourth will make documentation indices for pre version 11.2 and the last creates a new documentation paclet.

Creating a new Documentation Paclet

We start off by making a new paclet to hold our documentation. To do that we just use the making-documentation-with-docgen-7009256029017860706 button. It pops up a window:

making-documentation-with-docgen-2791007331622069100

And we simply provide the name of our paclet. The current project will then switch to "Templating" :

making-documentation-with-docgen-1743412973075559677

And we see the paclet info and config file have been built:

making-documentation-with-docgen-552763968467873143

Making a SymbolPage Template

First we go to the "Context Pages" menu and select "Symbol Page Template"

making-documentation-with-docgen-3053272758509917492

This opens up a notebook with templates for all the symbols in the context:

making-documentation-with-docgen-462167994733675065

Making a Symbol Page

We can fill out the content for a page:

making-documentation-with-docgen-5498733888786341535

And then we use the docked cell on the template page to build out the HTML (with the bracket for the content we're generating selected):

making-documentation-with-docgen-2943174513768772591

This then builds a doc page for us:

making-documentation-with-docgen-725166031214461507

Saving a Symbol Page

Alternately we can do this from the palette using the "Generate and Save" menu to save the page to the current paclet:

making-documentation-with-docgen-6219661019162151278

And then if we go to the "Open Symbol Page" menu we see we've generated and saved the page:

making-documentation-with-docgen-3284837002929066762

Making Multiple Pages

Alternately, if we opted not to select a cell bracket the package will generate a page for every template in the InputNotebook . Or if we had selected multiple brackets pages would be generated for each. This gives us a way to easily update our documentation from a saved template. Here's what we get if we did it with no bracket selected:

making-documentation-with-docgen-6556851610799844716

Deploying the Paclet

The DocGen palette doesn't directly support deploying this paclet to a server, but we can use the Paclet Server Manager palette that also comes with BTools to do that. I won't go into the details, though, as they're documented here .

Generating HTML Documentation

We can also use these template to build HTML documentation. To do this without deployment we can simply select the "Symbol Web Pages" action from the "Generate from Notebook" menu (again with the bracket selected):

making-documentation-with-docgen-6451737925142533680

This builds to a temporary directory (it also mirrors the display assets it needs the first time it's generated). The Cloud Deploy option comes into play when using the "Generate and Save" menu.

After this copies every thing and generates the HTML we end up with a page that looks like:

making-documentation-with-docgen-1179710600402752990

Which of course looks like the 11.0 docs, but until a new version of "Transmogrify" and "DocumentationBuild" come out this is what we're restricted to. That, unfortunately, is outside of my control so we'll stop at this.

POSTED BY: b3m2a1 ​ 
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract