As the original poster, I would like to thank everyone for this excellent discussion. I have a lot of ideas to test out.

Clearly, this topic is complicated enough to have some 'official' tutorial from Wolfram Research. I am hopeful that with the release of Workbench 3, there will be a revision in the classes to go along with it. At one time, I was promised a second level course on Workbench and advanced programming topics, but this topic seems to have been lost in the ever lengthening list of things to do.

I can understand this, because this topic is of little interest or use to the vast majority of Wolfram language users -- those who use the tool as a big graphical calculator or rapid prototyping tool. However, the strength of any language's culture is the degree of support it provides to those people who want to push the envelope.

Would a user of this package "Main" normally only have ff as an available routine? Users would not normally dig into the tree structure using contexts or even know what the sublevel contexts are.

So the development of the application is like this: There is a master programmer. The master programmer has A level programmers who work for him and speak only to him. Each A level programmer may have B level programmers who work for them and speak only to them. In turn the B level programmers may have C level programmers, etc. The master programmer is the only one who speaks to the outside world.

This is an incredibly structured and organized system! Very authoritarian. Agreed that some projects may require it.

I would like to suggest that this is not the way that scientists or mathematicians work or the way research or study projects are organized. To begin with they don't know what the tree structure is in advance. If they knew they wouldn't be doing research or carrying out a study program. If they commit to a tree structure they will be limiting their flexibility and maybe biasing their thinking. Otherwise they must be continually revising the tree structure, which can get to be a messy chore.

These might be projects that involve just one or a small number of collaborators. Still, over time, they may assemble a very worthwhile application with multiple packages, documentation and accompanying notebooks. I believe these kind of users can be important to WRI, more so than very large scale industrial projects.

For these types of projects a looser parallel structure is needed. The folder structure for such an application might look like the following:

`$UserBaseDirectory`/Applications/
  ApplicationName/
   Documentation Folder,
   FrontEnd Folder
   Kernel/init.m
   PackageA.m
   PackageB.m
   PackageC.m

The three packages will all export routines to final users. In addition they may use routines exported by each other. Why might that happen? Because an organization by topic, or by individuals, may not correspond to a pure tree structure.

Here, I believe, is a way to fix it. Use a special form of init.m file for the application.

Declare the packages in the application.

ApplicationNamePackages = {"PackageA`", "PackageB`", "PackageC`"};

Context names to be added to $ContextPath in each package. Used in each package.

System`$ApplicationNameContextPaths = 
  "ApplicationName`" <> # & /@ ApplicationNamePackages;

Each package will then use the following statement after the BeginPackage statement. (As with Pieter only the first argument is used in the BeginPackage statement.)

$ContextPath=Union[$ContextPath,System`$ApplicationNameContextPaths];

Going back to the init.m file, the following routine will read the package Public symbols for each package, to establish contexts, but clear all the definitions including all definitions in the Private context. (It would be sort of nice if we had a command that would read only the Public part of a package and never touch the definitions in the Private section.)

ReadApplicationNamePackageNames[packageList_] :=
 Module[{fullPackageName, packageSymbolNames},
  Do[
    fullPackageName = "ApplicationName`" <> pname;
    Get[fullPackageName];
    packageSymbolNames = Names[fullPackageName <> "*"];
    Unprotect /@ packageSymbolNames;
    Clear /@ packageSymbolNames;
    Clear @@ {fullPackageName <> "Private`*"},
    {pname, packageList}];
  ]

The following does this for the packages.

ReadApplicationNamePackageNames[ApplicationNamePackages]; 

Finally, we reload all of the packages to obtain the definitions, now all placed in the correct context.

Get["ApplicationName`PackageA`"]
Get["ApplicationName`PackageB`"]
Get["ApplicationName`PackageC`"]

That is the end of the init.m file. The key requirement here is that all contexts for the application are known before any Private section code is established. It's possible that duplicate names might be used from different packages, which will produce shadowed warnings. But this can also occur in using routines from different applications or conflicting with WRI symbols. This is easily fixed and only a minor problem.

Perhaps this approach can be combined with the method described by Pieter. We would have more than one "Main" package, and these might or might not contain sub-trees. So in the above init.m code PackageA, PackageB and PackageC, would be three "Main" packages.

I find this rather difficult to understand Pieter. Can any package refer to an exported symbol from any other package, regardless of their relative positions in the tree structure? Can P1 make a hidden import of P2 and P2 make a hidden import of P1? Are the multiple packages no longer loaded by the init.m file?

If exported names are only known at the next higher level in a tree does that mean that the ultimate user can't use those lower level routines?

What does "He/She then subdivides his module in files using Get as a stub to a placeholder section." mean? Is a "module" a set of package files, each of which may have hidden imports? And what is GIT and GIThub?

This sounds like a method that is heavily dependent on the tree structure and not at all flexible.

A system that established the contexts of all exported symbols (to the ultimate user) first and only then established the definitions in the Private sections would be much more robust - and simpler.

This is an excellent question. First my own remarks on programming in the wolfram language. First I have a general idea on what to create. (the algorithm is my invention and the computer only executes it)

I create a few snippets of code for the crucial areas to check how complicated or easy the problem is (storyboard coding). Then I create a flowchart of input and output-data-formats and function names. This exercise creates more insight in the problem and in most of the times it simplifies the solution approach. (do I need global data objects or will I go for the non global symbols approach)

Then I create two notebooks. One to program the functions and one to test the functions. After programming the next function in the flowchart I test the function in the testnotebook (the programming notebook can be kept clean by doing all experiments in the testnotebook). When a few new functions have been added and tested it is time for an integration test by running the full testnotebook. So before I look at the Workbench all programming is done in 2 notebooks.

Every set of functions in my flowcharts are subdivided into more aggregated blocks of code. The idea is to put every aggregated block of code into a subpackage and then merge it all together into 1 Main[] function. So subpackages need to roll-up into 1 main package and my global variables need to be global to only the mainpackage context. Let's load subpackage P1 and P2 and the globalvars package.

BeginPackage["Main"] main::usage = "main[]..." Needs["Globalvars"] Needs["P1"] Needs["P2"] Begin["Private"] main := .... End[] EndPackage

The most important thing is not to include P1 and P2 in the second argument of BeginPackage. The reason is that they are Needs that are executed before the BeginPackage command itself. This puts P1 and P2 on the contextpath before Main is added. The problem arises when EndPackage is executed. This switches back to the contextpath status before Beginpackage. So P1 and P2 stay on the contextpath when leaving the package. The way Needs is used here is called Hidden import.

Now there is a new trick in M10. Usage exports a function from the private to the package context. But this now also works of a package in a package in a package. Every usage function is now visible to only the 1 level higher package. With this approach all your symbols are recognized in all your packages and you never have to use full context to refer to something. So you end up with only 1 context added to the ContextPath when you arrive back in the global context. You global symbols are now global to the context Main and lower so there is not issue if the use accidentally creates the same names in Global` context.

This is the moment when I start to use the workbench. I import my programming and testing notebooks and create .wl files for the package structure I need. This structure by the way depends very much on the number of programmers involved. Every programmer gets a part of the overall function flowchart. He/She then subdivides his module in files using Get as a stub to a placeholder section. The workbench has support for GIT and via GIThub you get version management control on the whole team. When you start to do some builds it is good to now you have all documents, notebooks, packages etc. in one place. In some occasions programmers de-rail into developing a solution and feel the needs to switch a few versions of their files back. This is easy when the package files are organized by programmer.

The new nested packages approach is a great addition to developing large programs in the Wolfram language since you organize by programmer and by aggregated package and execute version management on each of these files.

David, I have investigated your "Context Problem example". One of the issues (I think) is the use of Needs in the BeginPackage statement. Let me know what you think about my reply to George?

Yes, do some simple experiments to see if you have the general idea scoped out and let me know if you get to a point of confusion or contradiction...

One question that I'd ask first is why you want to do this. One reason may simply be organizational--that you want to create several different packages and have the code-base for each separate. An immediate question arises as to whether there are dependencies amongst the packages. Of course if there are dependencies they must be in a descending tree structure: no closed loops. Another question is whether you want all of the packages to be loaded at the same time. In the "old days" sometimes people wanted control over which packages in a group were loaded so as to conserve memory resources--that's rarely necessary these days though, unless a given package loads significant data resources, insanely large expressions, or needs to do some time -consuming calculations in the course of its loading.

I tend just to put everything into one package... and I generally program in a notebook rather than Workbench. But this is a matter of preference of course--there are advantages to each approach. If programming in a notebook, I just put the code for distinct areas in different section organizations. The size of the code is immaterial (subject to the caveats above): I have packages with on the order of 60k lines of code and they load quickly.

As for your question--there are several approaches. One general approach is to have a top-level context and to put all the subsidiary packages in their own contexts. Then in the BeginPackage call for the top level package, also include the other packages in the BeginPackage's second argument. Or the packages that are included in the BeginPackage's 2nd argument may not be all of the other packages' contexts--the contexts in that BeginPackage command as well as the other packages' BeginPackage commands can reflect the package dependency. Also the advantage of using BeginPackage for each sub package is that each will have a private sub-context and so one can have the public symbols properly separated from thePrivate stuff in each package's sub-context.

Another possibility that one may think of is to code the subsidiary packages' code in a way that does not declare contexts and just pull them in using Get inside of the main package's code as needed. This presents a problem since this either has to be done before the Private sub context or before it. Then it is a mess to get the subsidiary packages' public andPrivate symbols placed correctly: explicit full context names would need to be used. So this is a non-starter I think.

So, those are some thoughts...