Group Abstract Group Abstract

Message Boards Message Boards

7
|
11.1K Views
|
16 Replies
|
18 Total Likes
View groups...
Share
Share this post:

Cannot generate paclet documentation due to crash Mathematica 13.0

With the new paclet functions and documentation tools in Mathematica 13.0 and the problems with workbench, I finally came around to refactoring my paclet (qmritools) and attempted to do the full build via Mathematica. However, I did not succeed.

Some things have been discussed previously (here and here) and I am aware that the PacletTools are still experimental but i still wanted to share my experiences. Maybe someone has a solution or a workaround or a fix.

My paclet contains the main context QMRTools which loads all the sub packages.

In[9]:= << QMRITools`
SetDirectory[NotebookDirectory[]];
FileBaseName /@ FileNames["*.wl", "QMRITools\\kernel"]
QMRITools`$Contexts

Out[11]= {"CardiacTools", "CoilTools", "DenoiseTools", "DixonTools", \
"ElastixTools", "GeneralTools", "GradientTools", "ImportTools", \
"IVIMTools", "JcouplingTools", "MaskingTools", "NiftiTools", \
"PhysiologyTools", "PlottingTools", "ProcessingTools", "QMRITools", \
"ReconstructionTools", "RelaxometryTools", "SimulationTools", \
"SpectroTools", "TaggingTools", "TensorTools", "TractographyTools", \
"VisteTools"}

Out[12]= {"QMRITools`GeneralTools`", "QMRITools`MaskingTools`", \
"QMRITools`NiftiTools`", "QMRITools`ElastixTools`", \
"QMRITools`PlottingTools`", "QMRITools`DixonTools`", \
"QMRITools`IVIMTools`", "QMRITools`DenoiseTools`", \
"QMRITools`CardiacTools`", "QMRITools`RelaxometryTools`", \
"QMRITools`GradientTools`", "QMRITools`TensorTools`", \
"QMRITools`JcouplingTools`", "QMRITools`SpectroTools`", \
"QMRITools`ReconstructionTools`", "QMRITools`TractographyTools`", \
"QMRITools`VisteTools`", "QMRITools`ProcessingTools`", \
"QMRITools`SimulationTools`", "QMRITools`PhysiologyTools`", \
"QMRITools`CoilTools`", "QMRITools`TaggingTools`", \
"QMRITools`ImportTools`"}

In total it has over 600 functions (and options) wich all have ::usage defined. 

In[13]:= functions = 
  Sort@Flatten[Names[# <> "*"] & /@ QMRITools`$Contexts];
Length[functions]

Out[14]= 647

So to build the paclet the first step is to make all the fucntion and guide pages for which i used the new DocumentaionTools palette.

Notes on Documentation tools "Generate function pages":

  • usage text does not end in "." causes error messages during generation of the function pages.

solution: check usage texts for closing periods.

(*Check usage for end in "."*)
DeleteCases[
 If[StringTake[Information[#]["Usage"], -1] =!= ".", #] & /@ 
  functions, Null]
  • My paclet has over 600 usage texts however it consistantly crashes between 450 and 620 notebooks. The crash is such that mathematica completely crashes, frontend and kernel.

solution: only load half the sub contexts and generate the function pages, then load the other half of the sub contexts and generate those function pages.

  • If I add a single function for which I want to generate a function page I don't want to generate all 650 of them in the "Generate function pages" of the documentation palette it is possible to only generate selected pages. However this only allows for the function in the context QMRITools'* to be selected and not all the functions in the QMRITools'*'*. Also this menu automatically fills in the function found in the context. I can imaging this fails with 600+ function and options. In workbench there was the possibility to only select the missing functions.

partial solution: it is easy to find which function pages I am missing with the following code. However it wont allow me to fill in these functions in the menu since they are not in the top level context.

In[15]:= dir = 
  "QMRITools\\Documentation\\English\\ReferencePages\\Symbols";
missing = 
 If[! FileExistsQ[FileNameJoin[{dir, #}] <> ".nb"], #, Nothing] & /@ 
  functions
Sort[{#, Intersection[missing, Names[# <> "*"]]} & /@ 
  QMRITools`$Contexts]

Out[16]= {"ADCCalc"}

Out[17]= {{"QMRITools`CardiacTools`", {}}, {"QMRITools`CoilTools`", \
{}}, {"QMRITools`DenoiseTools`", {}}, {"QMRITools`DixonTools`", {}}, \
{"QMRITools`ElastixTools`", {}}, {"QMRITools`GeneralTools`", {}}, \
{"QMRITools`GradientTools`", {}}, {"QMRITools`ImportTools`", {}}, \
{"QMRITools`IVIMTools`", {}}, {"QMRITools`JcouplingTools`", {}}, \
{"QMRITools`MaskingTools`", {}}, {"QMRITools`NiftiTools`", {}}, \
{"QMRITools`PhysiologyTools`", {}}, {"QMRITools`PlottingTools`", {}}, \
{"QMRITools`ProcessingTools`", {}}, \
{"QMRITools`ReconstructionTools`", {}}, \
{"QMRITools`RelaxometryTools`", {}}, {"QMRITools`SimulationTools`", \
{}}, {"QMRITools`SpectroTools`", {}}, {"QMRITools`TaggingTools`", \
{}}, {"QMRITools`TensorTools`", {"ADCCalc"}}, \
{"QMRITools`TractographyTools`", {}}, {"QMRITools`VisteTools`", {}}}

enter image description here enter image description here

With some workarounds I eventually managed to generate all the function pages such that I can build my paclet using the PacletTools. However here things get messy and I am not able to build my packlet.

PacletTools:

  • PacletDocumentationBuild[] also consistently crashes between 400 and 500 function pages build.

  • PacletDocumentationBuild[ , , "HTML"] also consistently crashes Mathematica between 400 and 500 function pages build.

  • PacletBuild needs to always rebuild the full documentation. If I have done full documentation build already and only change some function definitions or Asset but no function or guide pages I still have to rebuild the entire documentation.

solution: "manually" copy the changed notebooks or Asset to the build directory and use CreatePacletArchive to pack the Build directory to a packet file. The code below copies all *.wl files and the paclet Extentions to the build folder. This way i can only call PacletDocumentationBuild when I think it is needed.

Options[ApplicationBuild] = {RebuildDoc -> False, 
   RebuildHTMLDoc -> False, MakeArchives -> True};
ApplicationBuild[location_?StringQ, opts : OptionsPattern[]] := 
 ApplicationBuild[LoadApplicationFolder[location], opts]
ApplicationBuild[pac_, OptionsPattern[]] := 
 Block[{location, name, filesApp, filesAppTar, buildDir},
  location = pac["Location"];
  name = pac["Name"];
  buildDir = FileNameJoin[{location, "build", name}];

  Print["Building " <> name <> " in location " <> location];

  Print["   - Building application"];
  filesApp = Prepend[
    Flatten[
     Select[Normal[
        PacletExtensionFiles[pac]], #[[1, 1]] =!= "Documentation" &][[
      All, 2]]],
    FileNameJoin[{location, "PacletInfo.wl"}]
    ];
  filesAppTar = StringReplace[#, location -> buildDir] & /@ filesApp;

  MapThread[(
     Quiet@CreateDirectory[DirectoryName[#2]];
     CopyFile[#1, #2, OverwriteTarget -> True]
     ) &, {filesApp, filesAppTar}];

  If[OptionValue[RebuildDoc],
   Print["   - Rebuilding documentation"];
   PacletDocumentationBuild[pac, DirectoryName[buildDir]]
   ,
   Print["   - Not building documentation"];
   ];

  If[OptionValue[RebuildHTMLDoc],
   Print["   - Rebuilding html documentation"];
   PacletDocumentationBuild[pac, buildDir <> "-html", "HTML"]
   ,
   Print["   - Not building html documentation"];
   ];

  If[OptionValue[MakeArchives],
   Print["   - Making paclet and archive"];
   CreatePacletArchive[buildDir];
   CreateArchive[buildDir, 
    FileNameJoin[
     Append[Drop[FileNameSplit[buildDir], -1], 
      name <> "-" <> pac["Version"] <> ".zip"]], 
    OverwriteTarget -> True];
   ,
   Print["   - Not making paclet and archive"];
   ]]

So in the end I can manage everything to work all the way I want. However, I cannot finish the documentation (and HTML) builds since the Mathematica FrontEnd crashes after processing ~400 files.

enter image description here
POSTED BY: Martijn Froeling
16 Replies
Sort By:

On the topic of PacletTools: I noticed that loading PacletTools and GeneralUtilities at the same time gives a shadowing issue with CatchAll:

<< GeneralUtilities`
<< PacletTools`
POSTED BY: Sjoerd Smit

Hi Martijn, just a quick update. I tried to reproduce the crash but was unable to. But I'm using v13.0 on macOS 10.15.7, and presuming based on your screenshots, you're using Windows, which could explain the different behavior.

A couple of notes on DocumentationBuildHTML:

  • It accepts the same predicate function as a third argument, so like with DocumentationBuildNotebooks, you can work around the crash by building specific notebooks for the time being.
  • You will probably want to specify the following options (PacletDocumentationBuild[.., "HTML"] does this for you):
DocumentationBuildHTML[
    ...,
    "UseLanguageExtension" -> False,  (* Output "file.html", not "file.html.en" *)
    "CompleteHTMLQ" -> True,          (* Include the JavaScript/CSS assets necessary to correctly render the HTML *)
]
POSTED BY: Connor Gray

Thanks! I will give the HTML build another try!

Yes, I'm on Windows:
Edition Windows 10 Enterprise
Version 21H2
OS build 19044.1466
Windows Feature Experience Pack 120.2212.3920.0
POSTED BY: Martijn Froeling

Hi Conor, today I have installed release 13.1.

It seems that the PacletDocumentationBuild is working now, and my Mathematica is no longer crashing. Not sure if anything has changed, but at least it works, which is great!

However, I do get some errors while running, although they don't seem to influence the build. For the documentation, I get this error, the referred file is indeed there but cannot be deleted I guess.

enter image description here

When I build the HTML I get the same but when the build starts with the symbol pages these errors pop up for each page, which is not the case for the guide pages.

enter image description here

And again there are some issues with the reference.nb file.

enter image description here

When installing the Paclet I often get this error. If it happens its always the same file that gives the issue and I have to quit the kernel to recover Mathematica, after that it always works immediately.

enter image description here

Happy to see everything working!
POSTED BY: Martijn Froeling
Updating Name
Updating Name
Posted 3 years ago

Hi Martijn,

I'm glad to see that the build experience has improved for you. There were a number of FrontEnd issue addressed in v13.1; it seems likely that one or more of them may have contributed to the crashes you experienced. The DeleteFile operations are unknown to me, but the Pick[..] messages are an issue I've run into before that occurs when the grid options in one of the 'Usage' sections at the top of a symbol reference page are not formed correctly. I don't have a correct example available off-hand, but you might try re-creating the notebook that is generating those messages; it's likely whatever is wrong was simply an accidentally deleted part of the typesetting of the usage message.

It's on my list to improve that error message in the future.
POSTED BY: Updating Name

I'll give that a try. I indeed remember there was an issue with the formatting of multi-line usage messages using the documentation palette. I'll see if that has changed in the new release and let you know if it will work.

Tnx!
POSTED BY: Martijn Froeling
POSTED BY: Martijn Froeling

After doing some comparison i noticed that for the old and new generated and template Guide and Function pages indeed the header of the documents has changed. The Meta information is now at the bottom but at the top of the pages the "New Modified etc." line is indeed missing.

enter image description here

enter image description here
POSTED BY: Martijn Froeling
POSTED BY: Connor Gray

Awesome! will give it a try!

Tnx for the swift update. Will let you know how things work out.
POSTED BY: Martijn Froeling

Everything seems to work as intended! It also seems a bit quicker, building 700 pages in ~10 min where it used to be almost 20.

Thanks for all the help

enter image description here
POSTED BY: Martijn Froeling

Hi Conner,

Today I tried to also rebuild my HTML documentation after the successful completion of the normal documentation build. However the issue that was there with the errors related to the new layout of the template files that was fixed for the normal documentation is still there for the HTML build.

enter image description here
POSTED BY: Martijn Froeling
POSTED BY: Connor Gray
POSTED BY: Martijn Froeling
POSTED BY: Martijn Froeling

Very much agree on the rebuilding of documentation when you only change some code files. PacletBuild should be smart enough to figure out that documentation notebooks don't need rebuilding in that situation. A simple file hash or something like that should take care of it. It should also have an option to wipe the old build and start from scratch (in case you want to be thorough).

Also: PacletTest is basically unusable in it's current form since it only works if you add the extension:

{"Tests", "Root" -> "./Tests", Method -> "Experimental-v1"}

No one is going to figure that out by themselves without expecting the source code.
POSTED BY: Sjoerd Smit
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard