Motivation

A typical advice given for writing larger software packages (cf. here) is to split your code into smaller components, which are easier to maintain and which are easier to delegate within a team of developers. Nowadays, we are also encouraged to make use of Paclets that nicely weave themselves into the installed system (cf. Paclets Overview). As (hopefully) the number of published paclets in the Paclet Repository grows, naming conflicts will become much more likely than before. That challenge was—according to my understanding—met with the introduction of PublisherIDs and constructs like Needs["context`"-> "alias`"] to avoid $ContextPath collisions (cf. Todd Gayley's WTC presentation)

In short, we want to:

• organize larger paclets with a number of sub-packages that typically have interdependencies (often there is core functionality, that is used by more specialised packages)
• to have a choice when loading a paclet of either using Needs["PublisherID`PacletName`"], putting contexts on the $ContextPath, or using Needs["PublisherID`PacletName`" -> "alias`"] as is recommended these days

How to do this properly?

In the examples I found on GitHub for the PacletCICD paclet (e.g., here, here, and here), there were no interdependencies, e.g., AddTwo does not call AddOne in the examples. Also, comparable to the deprecated use of an init.m file, Get is used to load sub-packages putting these contexts on ContextPath forfeiting the use of $ContextAliases?

Other examples, e.g., those given in the documentation or in the guidelines for creating paclets to be published, unfortunately are rather simple and do not touch multiple sub-packages with interdependencies.

A first solution and open questions

After some experimentation, I came up with the following solution, which I would like to discuss here.

Paclet Structure

Let's assume that our paclet structure on file looks like this:

PublisherID__PacletName/
    Kernel/
        Core.wl
        Special.wl
        PacletName.wl
    PacletInfo.wl

Where Special.wl contains expressions that call upon public expressions contained in Core.wl.

PacletInfo.wl

The PacletInfo can be kept as is after using Needs["PacletTools`"] and CreatePaclet["PublisherID/PacletName", $HomeDirectory ].

PacletObject[
    <|
        ...
        "Extensions" ->
            {
                "Kernel",
                "Root" -> "Kernel",
                "Context" -> 
                    {
                        { "PublisherID`PacletName`" , "PacletName.wl" },
                        { "PublisherID`PacletName`Core`", "Core.wl" },
                        { "PublisherID`PacletName`Special`", "Special.wl" }
                    }
            },
            ...
    |>
]

PacletName.wl

The main file loaded when we do Needs["PublisherID`PacletName`"] later on, will just make that sure the other packages contained in the paclet are loaded and that the public symbols are known. As we keep away from the context path, we will introduce aliases for the full-name references in the `Private` context of that package:

BeginPackage["PublisherID`PacletName`"]

(* load sub-packages for full-name reference only *)
(* evaluate Needs[ ] only when not run by Workbench/Eclipse *)
With[ { mainContext = "PublisherID`PacletName`" },
    If[ Not @ MemberQ[ $Packages, "MEET`" ],
        (* load sub-contexts *)
        Needs[ mainContext <> "Core`" -> None ];
        Needs[ mainContext <> "Special`" -> None ]
    ]
] 

(* introduce public symbols of the paclet *)
coreFunc::usage = " ... "
specialFunc::usage = " ... "

Begin["`Private`"]

(* define aliases to reference implementation *)
coreFunc = PublisherID`PacletName`Core`Private`coreFunc
specialFunc = PublisherID`PacletName`Special`Private`specialFunc

End[]

EndPackage[]

Core.wl

BeginPackage["PublisherID`PacletName`Core`"]
(* Exported symbols added here with SymbolName::usage *)

Begin["`Private`"]
(* Implementation of the package *)

coreFunc[ k_Integer ] := 2 * k (* simple example *)

End[]

EndPackage[]

Special.wl

BeginPackage["PublisherID`PacletName`Special`"]
(* Exported symbols added here with SymbolName::usage *)

Begin["`Private`"]
(* Implementation of the package *)

specialFunc[ arg_Integer ] := PublisherID`PacletName`coreFunc @ arg 

End[]

EndPackage[]

Using the installed paclet

With the implementation described above, we can use the deployed and installed paclet either doing:

Needs["PublisherID`PacletName`"]
coreFunc[4]
specialFunc[4]
(* 8 8 *)

or

Needs["PublisherID`PacletName`" -> "p`" ]
p`coreFunc[4]
p`specialFunc[4]
(* 8 8 *)

Note: During development of a paclet, we would need to do the following, before the paclet is loaded using Needs[]:

PacletDirectoryLoad[ paclet_directory ]
PacletDataRebuild[]

Needs[ ... ]

Open Questions

1. Is what I described here the "proper" way to do this and if not, what is "best practice"?
2. Is there a way to avoid the need of full-name references in a DRY (do not repeat yourself) kind of fashion?
3. Is there something comparable to ParallelNeeds or do we have to DistributeContexts to support parallelization?

REVISIONS

I would like to keep this up to date, so that any improvement to THIS way of setting up larger paclets will be included in this post.

Context information should be provided for all sub-contexts as well

As Jason pointed out below, it seems that PacletInfo.wl should inform about all contexts. I have added this information now in the explicit form { "context`", filename } should the filename be obtainable by simply adding .wl or .m it may suffice to simply provide a list of contexts.

Usage messages need to be introduced in the PacletName` context

We need to have usage messages in the context PublisherID`PacletName` or they will not be shown. Since we are using full-name references we can immediately refer to `Private` context in PublisherID`PacletName`Private` or to refer to the public context of a sub-package, so that the symbol needs to be made public there.

Sub-packages do not need to load main context with Needs[ ]

Since the main package will load all sub-packages to allow full-name reference to implementations, there is no need to use Needs[ ] in the sub-packages.

Evaluate Needs[ ] only when not run by Workbench/Eclipse

The current plugin for Eclipse will automatically load the files in the Kernel/ directory (maybe this observation can be commented upon by WRI?) and thus the Needs[ ] statements for loading sub-packages should only be evaluated if the context Meet` is not listed among $Packages. Then everything works out nicely in Workbench/Eclipse. ;-)