Message Boards Message Boards

GROUPS:

How to setup and organize larger paclets as of v13+

Posted 2 months ago
967 Views
|
6 Replies
|
18 Total Likes
|

Motivation

A typical advice given for writing larger software packages (cf. here) is to split your code into smaller components, which are easier to maintain and which are easier to delegate within a team of developers. Nowadays, we are also encouraged to make use of Paclets that nicely weave themselves into the installed system (cf. Paclets Overview). As (hopefully) the number of published paclets in the Paclet Repository grows, naming conflicts will become much more likely than before. That challenge was—according to my understanding—met with the introduction of PublisherIDs and constructs like Needs["context`"-> "alias`"] to avoid $ContextPath collisions (cf. Todd Gayley's WTC presentation)

In short, we want to:

  • organize larger paclets with a number of sub-packages that typically have interdependencies (often there is core functionality, that is used by more specialised packages)
  • to have a choice when loading a paclet of either using Needs["PublisherID`PacletName`"], putting contexts on the $ContextPath, or using Needs["PublisherID`PacletName`" -> "alias`"] as is recommended these days

How to do this properly?

In the examples I found on GitHub for the PacletCICD paclet (e.g., here, here, and here), there were no interdependencies, e.g., AddTwo does not call AddOne in the examples. Also, comparable to the deprecated use of an init.m file, Get is used to load sub-packages putting these contexts on ContextPath forfeiting the use of $ContextAliases?

Other examples, e.g., those given in the documentation or in the guidelines for creating paclets to be published, unfortunately are rather simple and do not touch multiple sub-packages with interdependencies.

A first solution and open questions

After some experimentation, I came up with the following solution, which I would like to discuss here.

Paclet Structure

Let's assume that our paclet structure on file looks like this:

PublisherID__PacletName/
    Kernel/
        Core.wl
        Special.wl
        PacletName.wl
    PacletInfo.wl

Where Special.wl contains expressions that call upon public expressions contained in Core.wl.

PacletInfo.wl

The PacletInfo can be kept as is after using Needs["PacletTools`"] and CreatePaclet["PublisherID/PacletName", $HomeDirectory ].

PacletObject[
    <|
        ...
        "Extensions" ->
            {
                "Kernel",
                "Root" -> "Kernel",
                "Context" -> 
                    {
                        { "PublisherID`PacletName`" , "PacletName.wl" },
                        { "PublisherID`PacletName`Core`", "Core.wl" },
                        { "PublisherID`PacletName`Special`", "Special.wl" }
                    }
            },
            ...
    |>
]

PacletName.wl

The main file loaded when we do Needs["PublisherID`PacletName`"] later on, will just make that sure the other packages contained in the paclet are loaded and that the public symbols are known. As we keep away from the context path, we will introduce aliases for the full-name references in the `Private` context of that package:

BeginPackage["PublisherID`PacletName`"]

(* load sub-packages for full-name reference only *)
(* evaluate Needs[ ] only when not run by Workbench/Eclipse *)
With[ { mainContext = "PublisherID`PacletName`" },
    If[ Not @ MemberQ[ $Packages, "MEET`" ],
        (* load sub-contexts *)
        Needs[ mainContext <> "Core`" -> None ];
        Needs[ mainContext <> "Special`" -> None ]
    ]
] 

(* introduce public symbols of the paclet *)
coreFunc::usage = " ... "
specialFunc::usage = " ... "

Begin["`Private`"]

(* define aliases to reference implementation *)
coreFunc = PublisherID`PacletName`Core`Private`coreFunc
specialFunc = PublisherID`PacletName`Special`Private`specialFunc

End[]

EndPackage[]

Core.wl

BeginPackage["PublisherID`PacletName`Core`"]
(* Exported symbols added here with SymbolName::usage *)

Begin["`Private`"]
(* Implementation of the package *)

coreFunc[ k_Integer ] := 2 * k (* simple example *)

End[]

EndPackage[]

Special.wl

BeginPackage["PublisherID`PacletName`Special`"]
(* Exported symbols added here with SymbolName::usage *)

Begin["`Private`"]
(* Implementation of the package *)

specialFunc[ arg_Integer ] := PublisherID`PacletName`coreFunc @ arg 

End[]

EndPackage[]

Using the installed paclet

With the implementation described above, we can use the deployed and installed paclet either doing:

Needs["PublisherID`PacletName`"]
coreFunc[4]
specialFunc[4]
(* 8 8 *)

or

Needs["PublisherID`PacletName`" -> "p`" ]
p`coreFunc[4]
p`specialFunc[4]
(* 8 8 *)

Note: During development of a paclet, we would need to do the following, before the paclet is loaded using Needs[]:

PacletDirectoryLoad[ paclet_directory ]
PacletDataRebuild[]

Needs[ ... ]

Open Questions

  1. Is what I described here the "proper" way to do this and if not, what is "best practice"?
  2. Is there a way to avoid the need of full-name references in a DRY (do not repeat yourself) kind of fashion?
  3. Is there something comparable to ParallelNeeds or do we have to DistributeContexts to support parallelization?

REVISIONS

I would like to keep this up to date, so that any improvement to THIS way of setting up larger paclets will be included in this post.

Context information should be provided for all sub-contexts as well

As Jason pointed out below, it seems that PacletInfo.wl should inform about all contexts. I have added this information now in the explicit form { "context`", filename } should the filename be obtainable by simply adding .wl or .m it may suffice to simply provide a list of contexts.

Usage messages need to be introduced in the PacletName` context

We need to have usage messages in the context PublisherID`PacletName` or they will not be shown. Since we are using full-name references we can immediately refer to `Private` context in PublisherID`PacletName`Private` or to refer to the public context of a sub-package, so that the symbol needs to be made public there.

Sub-packages do not need to load main context with Needs[ ]

Since the main package will load all sub-packages to allow full-name reference to implementations, there is no need to use Needs[ ] in the sub-packages.

Evaluate Needs[ ] only when not run by Workbench/Eclipse

The current plugin for Eclipse will automatically load the files in the Kernel/ directory (maybe this observation can be commented upon by WRI?) and thus the Needs[ ] statements for loading sub-packages should only be evaluated if the context Meet` is not listed among $Packages. Then everything works out nicely in Workbench/Eclipse. ;-)

6 Replies

I get an error when loading the paclet using the structure you've laid out, the line

Needs["PublisherID`PacletName`Core`" -> None]

because the package can't be found

In[3]:= FindFile["PublisherID`PacletName`Core`"]
Out[3]= $Failed

If I modify the kernel extension in the paclet info file to use

"Context" -> {
    {"PublisherID`PacletName`", "PacletName.wl"},
    {"PublisherID`PacletName`Core`", "Core.wl"},
    {"PublisherID`PacletName`Special`", "Special.wl"}
}

then it works as you describe.

I don't know if the method I lay out below is the 'best' practice but it works for me. I added another .wl file with package-internal functions that can be used by other subpackages that aren't exported to the user.

PacletName.wl

BeginPackage["PublisherID`PacletName`"]

(* implemented in Core.wl *)
coreFunc::usage = "..."

(* implemented in Special.wl *)
specialFunc::usage = "..."

Begin["`Private`"]

(* load all sub-packages here in the private context *)
Get["PublisherID`PacletName`Internal`"]
Get["PublisherID`PacletName`Core`"]
Get["PublisherID`PacletName`Special`"]

End[]

EndPackage[]

Core.wl

BeginPackage["PublisherID`PacletName`Core`"]
(* Exported symbols added here with SymbolName::usage *)

Begin["`Private`"]
Needs["PublisherID`PacletName`"] (* so that coreFunc is on the context path *)

coreFunc[k : (_Integer | _List)] := 2 * k (* simple example *)

End[]
EndPackage[]

Special.wl

BeginPackage["PublisherID`PacletName`Special`"]
Begin["`Private`"]

Needs["PublisherID`PacletName`"]
Needs["PublisherID`PacletName`Internal`"] (* for helperFunc *)

specialFunc[ arg_Integer ] := coreFunc @ helperFunc @ arg

End[]
EndPackage[]

Internal.wl

BeginPackage["PublisherID`PacletName`Internal`"]
(* introduce internal symbols of the paclet *)
helperFunc::usage = "..."

Begin["`Private`"]

helperFunc[x_] := {x, x^2}

End[]
EndPackage[]

Using this structure I get the same results,

In[5]:= Needs["PublisherID`PacletName`"]

In[6]:= coreFunc[4]
Out[6]= 8

In[7]:= specialFunc[4]
Out[7]= {8, 32}

But it avoids the function aliases you are using, which I find confusing

POSTED BY: Jason Biggs

Hi Jason,

my bad, of course, during development, one needs to start out with:

PacletDirectoryLoad[ paclet_directory ]
PacletDataRebuild[]

before Needs[...] is used to load the paclet. When I tested my setup, I did not have to add the other contexts, but I may have forgotten, that I in fact did. :)

EDIT

Using Get puts contexts on $ContextPath, which is what the recommendation given by Todd Gayley wants to avoid, e.g., we would like to refer to paclet functions using alias`funcname[ ].

EDIT2:

I just tested again on a MacBook Pro M1 Max (ARM64) using v13.1 and it should work as I described without having to provide additional "Context" information.

I tested using Eclipse and it does indeed not work without providing all context information.

Using Get puts contexts on $ContextPath

Using Get or Needs in the private portion of a package does not pollute the user's context path:

In[5]:= cpath = $ContextPath;
In[6]:= Needs["PublisherID`PacletName`"];
In[7]:= Complement[$ContextPath, cpath]

Out[7]= {"PublisherID`PacletName`"}
POSTED BY: Jason Biggs

Interim Conclusion

After having done a bit more testing and updating of my code (see Revisions section in my original post), the following picture emerges:

Using full-references with Needs["context' " -> None] as proposed by OP

  • aliases in main package serve as a “lookup-table”, i.e., it is immediately clear where a function is implemented and can thus serve as a central reference for development
  • full-name access to functions is possible, e.g., we may call PuiblisherID`PacletName`Special`Private`specialFunc directly from a notebook that has loaded the paclet
  • Using full-name external function calls in the sub-packages is a bit cumbersome, albeit (typically) just the main paclet context needs to be called
  • In Workbench/Eclipse we can use F3 from the main package to immediately jump to the code that is implementing the function in a sub-package
  • This implementation is completely compatible with Workbench/Eclipse, i.e., for development all you need to do is to comment out the Needs[ ] in the main package. Run as Wolfram will then work (if you have gotten rid of things like <<init.m and what have you in the run configurations).

Loading sub-packages into the main private context as proposed by Jason Briggs

  • it is not clear where a function has been implemented from looking at the main package
  • full-name access to functions is not possible, e.g., we cannot call PuiblisherID`PacletName`Special`Private`specialFunc from a notebook any more
  • short name references in the sub packages for external functions are more convenient
  • In Workbench/Eclipse it is not possible to use F3 to immediately jump to a functions implementation
  • The implementation is not compatible with Workbench/Eclipse (at least I have not managed to get it running with Run as Wolfram).

So far, these points make this a "matter of taste". Am I missing something? Is there something else that will clearly tilt the scales to one side?

EDIT

I added Workbench/Eclipse specific detail (F3) I addressed Workbench/Eclipse compatibility regarding runs during development

To wrap this up: Both approaches appear valid, but only the one proposed by the OP currently works out in Workbench/Eclipse, which is imo an important feature as larger projects call for the refactoring and integrative power of a full-fledged IDE.

What is further worth mentioning: The setup proposed by the OP can be used during development in Workbench/Eclipse and can then be converted into the form proposed by Jason Briggs for deployment:

  • Out-comment the „aliases“ in the Private context of the main package and add Needs or Get expressions to load the sub-packages
  • Add Needs statement to call the common core and de-endent packages in any subpackages (if deemed necessary)
  • Remove full context prefixes in the subpackages,e.g., most often PublisherID`PacletName`Core`

These changes can be easily and quickly done in an IDE. :)

I put this paclet in GitHub. https://github.com/bobohilario/SampleInterdependentPaclet

It's very similar to what Jason included in his comments.

I mostly created it so that we'd have an actual paclet that someone reading this post can clone and modify.

POSTED BY: Bob Sandheinrich
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract