Message Boards Message Boards

7
|
7575 Views
|
16 Replies
|
18 Total Likes
View groups...
Share
Share this post:

Cannot generate paclet documentation due to crash Mathematica 13.0

With the new paclet functions and documentation tools in Mathematica 13.0 and the problems with workbench, I finally came around to refactoring my paclet (qmritools) and attempted to do the full build via Mathematica. However, I did not succeed.

Some things have been discussed previously (here and here) and I am aware that the PacletTools are still experimental but i still wanted to share my experiences. Maybe someone has a solution or a workaround or a fix.

My paclet contains the main context QMRTools which loads all the sub packages.

In[9]:= << QMRITools`
SetDirectory[NotebookDirectory[]];
FileBaseName /@ FileNames["*.wl", "QMRITools\\kernel"]
QMRITools`$Contexts

Out[11]= {"CardiacTools", "CoilTools", "DenoiseTools", "DixonTools", \
"ElastixTools", "GeneralTools", "GradientTools", "ImportTools", \
"IVIMTools", "JcouplingTools", "MaskingTools", "NiftiTools", \
"PhysiologyTools", "PlottingTools", "ProcessingTools", "QMRITools", \
"ReconstructionTools", "RelaxometryTools", "SimulationTools", \
"SpectroTools", "TaggingTools", "TensorTools", "TractographyTools", \
"VisteTools"}

Out[12]= {"QMRITools`GeneralTools`", "QMRITools`MaskingTools`", \
"QMRITools`NiftiTools`", "QMRITools`ElastixTools`", \
"QMRITools`PlottingTools`", "QMRITools`DixonTools`", \
"QMRITools`IVIMTools`", "QMRITools`DenoiseTools`", \
"QMRITools`CardiacTools`", "QMRITools`RelaxometryTools`", \
"QMRITools`GradientTools`", "QMRITools`TensorTools`", \
"QMRITools`JcouplingTools`", "QMRITools`SpectroTools`", \
"QMRITools`ReconstructionTools`", "QMRITools`TractographyTools`", \
"QMRITools`VisteTools`", "QMRITools`ProcessingTools`", \
"QMRITools`SimulationTools`", "QMRITools`PhysiologyTools`", \
"QMRITools`CoilTools`", "QMRITools`TaggingTools`", \
"QMRITools`ImportTools`"}

In total it has over 600 functions (and options) wich all have ::usage defined. 

In[13]:= functions = 
  Sort@Flatten[Names[# <> "*"] & /@ QMRITools`$Contexts];
Length[functions]

Out[14]= 647

So to build the paclet the first step is to make all the fucntion and guide pages for which i used the new DocumentaionTools palette.

Notes on Documentation tools "Generate function pages":

  • usage text does not end in "." causes error messages during generation of the function pages.

solution: check usage texts for closing periods.

(*Check usage for end in "."*)
DeleteCases[
 If[StringTake[Information[#]["Usage"], -1] =!= ".", #] & /@ 
  functions, Null]
  • My paclet has over 600 usage texts however it consistantly crashes between 450 and 620 notebooks. The crash is such that mathematica completely crashes, frontend and kernel.

solution: only load half the sub contexts and generate the function pages, then load the other half of the sub contexts and generate those function pages.

  • If I add a single function for which I want to generate a function page I don't want to generate all 650 of them in the "Generate function pages" of the documentation palette it is possible to only generate selected pages. However this only allows for the function in the context QMRITools'* to be selected and not all the functions in the QMRITools'*'*. Also this menu automatically fills in the function found in the context. I can imaging this fails with 600+ function and options. In workbench there was the possibility to only select the missing functions.

partial solution: it is easy to find which function pages I am missing with the following code. However it wont allow me to fill in these functions in the menu since they are not in the top level context.

In[15]:= dir = 
  "QMRITools\\Documentation\\English\\ReferencePages\\Symbols";
missing = 
 If[! FileExistsQ[FileNameJoin[{dir, #}] <> ".nb"], #, Nothing] & /@ 
  functions
Sort[{#, Intersection[missing, Names[# <> "*"]]} & /@ 
  QMRITools`$Contexts]

Out[16]= {"ADCCalc"}

Out[17]= {{"QMRITools`CardiacTools`", {}}, {"QMRITools`CoilTools`", \
{}}, {"QMRITools`DenoiseTools`", {}}, {"QMRITools`DixonTools`", {}}, \
{"QMRITools`ElastixTools`", {}}, {"QMRITools`GeneralTools`", {}}, \
{"QMRITools`GradientTools`", {}}, {"QMRITools`ImportTools`", {}}, \
{"QMRITools`IVIMTools`", {}}, {"QMRITools`JcouplingTools`", {}}, \
{"QMRITools`MaskingTools`", {}}, {"QMRITools`NiftiTools`", {}}, \
{"QMRITools`PhysiologyTools`", {}}, {"QMRITools`PlottingTools`", {}}, \
{"QMRITools`ProcessingTools`", {}}, \
{"QMRITools`ReconstructionTools`", {}}, \
{"QMRITools`RelaxometryTools`", {}}, {"QMRITools`SimulationTools`", \
{}}, {"QMRITools`SpectroTools`", {}}, {"QMRITools`TaggingTools`", \
{}}, {"QMRITools`TensorTools`", {"ADCCalc"}}, \
{"QMRITools`TractographyTools`", {}}, {"QMRITools`VisteTools`", {}}}

enter image description here enter image description here

With some workarounds I eventually managed to generate all the function pages such that I can build my paclet using the PacletTools. However here things get messy and I am not able to build my packlet.

PacletTools:

  • PacletDocumentationBuild[] also consistently crashes between 400 and 500 function pages build.

  • PacletDocumentationBuild[ , , "HTML"] also consistently crashes Mathematica between 400 and 500 function pages build.

  • PacletBuild needs to always rebuild the full documentation. If I have done full documentation build already and only change some function definitions or Asset but no function or guide pages I still have to rebuild the entire documentation.

solution: "manually" copy the changed notebooks or Asset to the build directory and use CreatePacletArchive to pack the Build directory to a packet file. The code below copies all *.wl files and the paclet Extentions to the build folder. This way i can only call PacletDocumentationBuild when I think it is needed.

Options[ApplicationBuild] = {RebuildDoc -> False, 
   RebuildHTMLDoc -> False, MakeArchives -> True};
ApplicationBuild[location_?StringQ, opts : OptionsPattern[]] := 
 ApplicationBuild[LoadApplicationFolder[location], opts]
ApplicationBuild[pac_, OptionsPattern[]] := 
 Block[{location, name, filesApp, filesAppTar, buildDir},
  location = pac["Location"];
  name = pac["Name"];
  buildDir = FileNameJoin[{location, "build", name}];

  Print["Building " <> name <> " in location " <> location];

  Print["   - Building application"];
  filesApp = Prepend[
    Flatten[
     Select[Normal[
        PacletExtensionFiles[pac]], #[[1, 1]] =!= "Documentation" &][[
      All, 2]]],
    FileNameJoin[{location, "PacletInfo.wl"}]
    ];
  filesAppTar = StringReplace[#, location -> buildDir] & /@ filesApp;

  MapThread[(
     Quiet@CreateDirectory[DirectoryName[#2]];
     CopyFile[#1, #2, OverwriteTarget -> True]
     ) &, {filesApp, filesAppTar}];

  If[OptionValue[RebuildDoc],
   Print["   - Rebuilding documentation"];
   PacletDocumentationBuild[pac, DirectoryName[buildDir]]
   ,
   Print["   - Not building documentation"];
   ];

  If[OptionValue[RebuildHTMLDoc],
   Print["   - Rebuilding html documentation"];
   PacletDocumentationBuild[pac, buildDir <> "-html", "HTML"]
   ,
   Print["   - Not building html documentation"];
   ];

  If[OptionValue[MakeArchives],
   Print["   - Making paclet and archive"];
   CreatePacletArchive[buildDir];
   CreateArchive[buildDir, 
    FileNameJoin[
     Append[Drop[FileNameSplit[buildDir], -1], 
      name <> "-" <> pac["Version"] <> ".zip"]], 
    OverwriteTarget -> True];
   ,
   Print["   - Not making paclet and archive"];
   ]]

So in the end I can manage everything to work all the way I want. However, I cannot finish the documentation (and HTML) builds since the Mathematica FrontEnd crashes after processing ~400 files.

enter image description here
POSTED BY: Martijn Froeling
16 Replies
Sort By:

On the topic of PacletTools: I noticed that loading PacletTools and GeneralUtilities at the same time gives a shadowing issue with CatchAll:

<< GeneralUtilities`
<< PacletTools`
POSTED BY: Sjoerd Smit

Hi Martijn, just a quick update. I tried to reproduce the crash but was unable to. But I'm using v13.0 on macOS 10.15.7, and presuming based on your screenshots, you're using Windows, which could explain the different behavior.

A couple of notes on DocumentationBuildHTML:

  • It accepts the same predicate function as a third argument, so like with DocumentationBuildNotebooks, you can work around the crash by building specific notebooks for the time being.
  • You will probably want to specify the following options (PacletDocumentationBuild[.., "HTML"] does this for you):
DocumentationBuildHTML[
    ...,
    "UseLanguageExtension" -> False,  (* Output "file.html", not "file.html.en" *)
    "CompleteHTMLQ" -> True,          (* Include the JavaScript/CSS assets necessary to correctly render the HTML *)
]
POSTED BY: Connor Gray

Thanks! I will give the HTML build another try!

Yes, I'm on Windows:
Edition Windows 10 Enterprise
Version 21H2
OS build 19044.1466
Windows Feature Experience Pack 120.2212.3920.0
POSTED BY: Martijn Froeling

Hi Conor, today I have installed release 13.1.

It seems that the PacletDocumentationBuild is working now, and my Mathematica is no longer crashing. Not sure if anything has changed, but at least it works, which is great!

However, I do get some errors while running, although they don't seem to influence the build. For the documentation, I get this error, the referred file is indeed there but cannot be deleted I guess.

enter image description here

When I build the HTML I get the same but when the build starts with the symbol pages these errors pop up for each page, which is not the case for the guide pages.

enter image description here

And again there are some issues with the reference.nb file.

enter image description here

When installing the Paclet I often get this error. If it happens its always the same file that gives the issue and I have to quit the kernel to recover Mathematica, after that it always works immediately.

enter image description here

Happy to see everything working!
POSTED BY: Martijn Froeling
Updating Name
Updating Name
Posted 1 year ago

Hi Martijn,

I'm glad to see that the build experience has improved for you. There were a number of FrontEnd issue addressed in v13.1; it seems likely that one or more of them may have contributed to the crashes you experienced. The DeleteFile operations are unknown to me, but the Pick[..] messages are an issue I've run into before that occurs when the grid options in one of the 'Usage' sections at the top of a symbol reference page are not formed correctly. I don't have a correct example available off-hand, but you might try re-creating the notebook that is generating those messages; it's likely whatever is wrong was simply an accidentally deleted part of the typesetting of the usage message.

It's on my list to improve that error message in the future.
POSTED BY: Updating Name

I'll give that a try. I indeed remember there was an issue with the formatting of multi-line usage messages using the documentation palette. I'll see if that has changed in the new release and let you know if it will work.

Tnx!
POSTED BY: Martijn Froeling

I have rebuild de documentation pages using the palet, which now solves the errors and the issue I had before using that. This indeed removed the error I had before but now new weird things happen.

enter image description here

The errors keep coming and are not suppressed by the frontend (4 per page so over 2500), and they did break the frontend twice, but not every time. With Quiet@ it seems more stable.

If the build completes the pages that come out of it doesn't seem to be formatted correctly. Left is the page generated by the documentation palet, right is the page that comes from using PacletDocumentationBuild[paclet, buildDir].

enter image description here

The HTML build I haven't been able to complete now but will give it another try later.

I did commit the new doc pages to my git if you want to see what's going on.

POSTED BY: Martijn Froeling

After doing some comparison i noticed that for the old and new generated and template Guide and Function pages indeed the header of the documents has changed. The Meta information is now at the bottom but at the top of the pages the "New Modified etc." line is indeed missing.

enter image description here

enter image description here
POSTED BY: Martijn Froeling

Hi Martijn, I'm sorry you've run into this issue. A fix for this is now available in the form of PacletTools v0.0.12, which was released on Monday as an automatic paclet update.

You can check if v0.0.12 has already been installed automatically using PacletObject["PacletTools"]["Version"]. If the installed version is less than v0.0.12, then you can trigger the update mechanism using PacletInstall["PacletTools"].

v0.0.12 contains fixes for the GetNotebookHistoryData bug shown above, and an issue with building the new authoring notebook format. EDIT: To be clear, the build issue with the new authoring notebook is exactly the issue shown in your screenshot of AnisoFilterTensor — that is also fixed in PacletTools v0.0.12.

Please let me know if you run into further issues. I appreciate your patience.
POSTED BY: Connor Gray

Awesome! will give it a try!

Tnx for the swift update. Will let you know how things work out.
POSTED BY: Martijn Froeling

Everything seems to work as intended! It also seems a bit quicker, building 700 pages in ~10 min where it used to be almost 20.

Thanks for all the help

enter image description here
POSTED BY: Martijn Froeling

Hi Conner,

Today I tried to also rebuild my HTML documentation after the successful completion of the normal documentation build. However the issue that was there with the errors related to the new layout of the template files that was fixed for the normal documentation is still there for the HTML build.

enter image description here
POSTED BY: Martijn Froeling

Hi Martijn, I'm the primary developer of the PacletTools package. I'm sorry to hear that you've been having trouble with PacletDocumentationBuild. As you mentioned, it's still quite new and experimental, so there are some aspects of the UX that are not yet as polished as they could be — the lack of incremental rebuilds being a big one.

Incremental rebuilds are coming to PacletDocumentationBuild, likely for 13.1. However, for now, I'll describe a workaround that should mitigate both the slow builds and the crash.

PacletDocumentationBuild is partly a wrapper around our DocumentationBuild` package (which long-time users of the Workbench may recognize). If you use the lower-level DocumentationBuildNotebooks function directly, you can more precisely control which files get built:

Needs["DocumentationBuild`"]

DocumentationBuildNotebooks[
    (* Input directory. *)
    "~/path/to/QMRITools/Documentation",
    (* Output directory. Pick any location you'd prefer *)
    "~/Desktop/QMRIToolsDocumentation",
    (* Predicate function to filter which files to build.
        E.g., build only the CardiacTools and DenoiseTools guides *)
    StringContainsQ[#, "CardiacTools.nb" | "DenoiseTools.nb"] &
 ]

This should allow you to more quickly check changes to the documentation page you are currently working on, while avoiding re-building all the files you don't care about at that moment.

PacletDocumentationBuild[ , , "HTML"] also consistently crashes Mathematica between 400 and 500 function pages build.

The first step in building the HTML documentation is building the notebook documentation, so this likely crashes for the same reason.

I'm attempting to reproduce the crash locally, and I'll let you know what I'm able to find. For now, you may be able to narrow down which file is causing the crash by evaluating:

DocumentationBuildNotebooks[
    "~/path/to/QMRITools/Documentation",
    "~/Desktop/QMRIToolsDocumentation",
    Function[path,
        Print["Building: ", path];
        True
    ]
 ]

and making note of what the last printed file name is (and if it's even the same file every time).

EDIT: I just noticed that the QMRITools repository on GitHub doesn't appear to contain the full ~600 function pages mentioned in the description. Would it be possible for you to commit those to a branch that you could share with me? I've also reached out internally to our DocumentationTools team.
POSTED BY: Connor Gray

Hi Conor,

Awesome! thanks for the feedback, very helpfull. With this, I'm sure I can make it work!

I normally don't commit the function pages since they are automatically generated (either by workbench or the documentation pallette). I'm committing them now to the main branch so you should be able to find them now.

EDIT: additional note. not all pages look pretty since the function that generates them still had some issues and the workaround for that was to much work to fix .

If I find out where and for which file stuff crashes I will let you know.
POSTED BY: Martijn Froeling

Ok with these functions it seems to work. I saw that DocunmentationBuild also exposes DocumentationBuildHTML, so tried it as well.

Needs["DocumentationBuild`"]
dir = "D:\\werk\\workspace\\QMRITools\\QMRITools";

name = Last@FileNameSplit[dir];

guideDir = FileNameJoin[{"Documentation", "English", "Guides"}];
symDir = FileNameJoin[{"Documentation", "English", "ReferencePages", 
    "Symbols"}];

buildDirGuide = FileNameJoin[{"build", name, guideDir}];
buildDirSym = FileNameJoin[{"build", name, symDir}];

docDir = "Documentation";
buildDirHTML = FileNameJoin[{"build", name <> "-html"}];

The documentation notebooks build just fine did it 3 times to be sure.

DocumentationBuildNotebooks[FileNameJoin[{dir, guideDir}], 
  FileNameJoin[{dir, buildDirGuide}]];

DocumentationBuildNotebooks[FileNameJoin[{dir, symDir}], 
  FileNameJoin[{dir, buildDirSym}]];

enter image description here

The HTML build not so much.

DocumentationBuildHTML[FileNameJoin[{dir,docDir}],FileNameJoin[{dir,buildDirHTML}]];

enter image description here enter image description here

The same as I got with the Packet functions the crash is not predictable at a certain file but at random.

For now, this is great since I can rebuild my documentation again and with the upcoming updates, I'm sure the HTML will be there as well.

POSTED BY: Martijn Froeling

Very much agree on the rebuilding of documentation when you only change some code files. PacletBuild should be smart enough to figure out that documentation notebooks don't need rebuilding in that situation. A simple file hash or something like that should take care of it. It should also have an option to wipe the old build and start from scratch (in case you want to be thorough).

Also: PacletTest is basically unusable in it's current form since it only works if you add the extension:

{"Tests", "Root" -> "./Tests", Method -> "Experimental-v1"}

No one is going to figure that out by themselves without expecting the source code.
POSTED BY: Sjoerd Smit
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract