Thanks for the reply.

For my immediate project, I suppose I could put al the code in one .m file There might be 10 or 20 pages of code when I am done.

However, my experience working with c (etc.) is that breaking up the code over multiple files is a good thing.

One idea that I had was to rewrite my c code (about 4000 pages) in Mathematica -- a significantly larger project than my current one, so my question really has to do with this larger project.

If I understand correctly, I could set up a master file that has BeginPackage["myBigProject", {"subProj1", "subProj2:, etc}] and then each sub project would start BeginPackage["subProj1], etc. (I see that the editor messed up what I typed, but I think that the meaning is clear.)

If one (or more) of the projects would have utilities used by other sub projects, could I then use the second argument to Need[] the lower level project into the sub project? It seems logical.

My main reason for doing this type of factoring is to keep the model code separate from the view. Even for a modest project, that seems to be desirable. The view would only need to know about the Public stuff from the model, and vice versa.

I am planning to use Workbench so that I can create the documentation. I know that there is (probably) a way to make the Symbol, Guide, and Tutorial pages without using Workbench, but there may be advantages in using the same tools that Wolfram developers use internally.

I guess I will have to do some experiments....

George, I have one reply and one posting on Wolfram Community that may be useful to you.

The first basically shows how to lay out an application with multiple packages. You essentially load all the packages through the init.m file. This reply is in:

http://community.wolfram.com/groups/-/m/t/393033?ppauth=L19VHYIc

The second is my posting on:

Large Multipackage Applications

This deals with the problem of not maintaining a strict tree order - which may be difficult or inconvenient to do. I have two small zip files, ContextProblem.zip and ContextFix.zip that illustrate the problem and a fix for it. The problem is that the BeginPackage statements read all the packages listed in the second argument and establishes definitions. If the context of some symbols is not known they get pushed into the Private context. A correct definition may later be established when the package itself is read in. But it's hit or miss which one will be used. The solution is to use the init.m file to first read in all the packages to establish contexts but Clear all the symbol and Private definitions when reading each package. Then, with all the contexts established, read them again. That appears to work but I haven't done large scale testing on it.

It really would be nice if a Workbench for documenting that was simple to use and clearly documented itself was provided. I view it as essential for serious Mathematica usage in science and mathematical applications.

(same post, something went wrong the first time...) This is an excellent question. First my own remarks on programming in the wolfram language. First I have a general idea on what to create. (the algorithm is my invention and the computer only executes it)

I create a few snippets of code for the crucial areas to check how complicated or easy the problem is (storyboard coding). Then I create a flowchart of input and output-data-formats and function names. This exercise creates more insight in the problem and in most of the times it simplifies the solution approach. (do I need global data objects or will I go for the non global symbols approach)

Then I create two notebooks. One to program the functions and one to test the functions. After programming the next function in the flowchart I test the function in the testnotebook (the programming notebook can be kept clean by doing all experiments in the testnotebook). When a few new functions have been added and tested it is time for an integration test by running the full testnotebook. So before I look at the Workbench all programming is done in 2 notebooks.

Every set of functions in my flowcharts are subdivided into more aggregated blocks of code. The idea is to put every aggregated block of code into a subpackage and then merge it all together into 1 Main[] function. So subpackages need to roll-up into 1 main package and my global variables need to be global to only the mainpackage context. Let's load subpackage P1 and P2 and the globalvars package.

BeginPackage["Main`"]
main::usage = "main[]..."
Needs["Globalvars`"]
Needs["P1`"]
Needs["P2`"]
Begin["`Private`"]
 main := ....
End[]
EndPackage

The most important thing is not to include P1 and P2 in the second argument of BeginPackage. The reason is that they are Needs that are executed before the BeginPackage command itself. This puts P1 and P2 on the contextpath before Main is added. The problem arises when EndPackage is executed. This switches back to the contextpath status before Beginpackage. So P1 and P2 stay on the contextpath when leaving the package. The way Needs is used here is called Hidden import.

Now there is a new trick in M10. Usage exports a function from the private to the package context. But this now also works of a package in a package in a package. Every usage function is now visible to only the 1 level higher package. With this approach all your symbols are recognized in all your packages and you never have to use full context to refer to something. So you end up with only 1 context added to the ContextPath when you arrive back in the global context. You global symbols are now global to the context Main and lower so there is not issue if the use accidentally creates the same names in Global` context.

This is the moment when I start to use the workbench. I import my programming and testing notebooks and create .wl files for the package structure I need. This structure by the way depends very much on the number of programmers involved. Every programmer gets a part of the overall function flowchart. He/She then subdivides his module in files using Get as a stub to a placeholder section. The workbench has support for GIT and via GIThub you get version management control on the whole team. When you start to do some builds it is good to now you have all documents, notebooks, packages etc. in one place. In some occasions programmers de-rail into developing a solution and feel the needs to switch a few versions of their files back. This is easy when the package files are organized by programmer.

The new nested packages approach is a great addition to developing large programs in the Wolfram language since you organize by programmer and by aggregated package and execute version management on each of these files.

Pieter, yes, let me read your comments more carefully later on and experiment a bit. I didn't know about the version 10 behavior of usage messages that you describe (is it documented somewhere that you can point me to?). The only immediate issue with that of course is if the package is to be used for an earlier version of Mathematica. But let me look over your comments later on when I have some time and I will respond.

David (Park), First my answer to your reply.

"Can any package refer to an exported symbol from any other package, regardless of their relative positions in the tree structure?" With BeginPackage["pack",{"needs1","needs2"}] all context will be left on the Contextpath after the Endpackage statement. So yes when the user is in Global all contexts will be check in sequence for a particular symbol. This is not what I intent to create. I want to create a rollup of packages and make only 1 symbol available to the global context.

"What does "He/She then subdivides his module in files using Get as a stub to a placeholder section." mean?" This is a package with two placeholders in it. The ProgA and ProgB are .wl that are no packages but do export and have their own private section:

BeginPackage["PPackA`"]
Get["ProgA`"]; 
Get["ProgB`"];
EndPackage[]

Aff::usage="Aff[] is exported outside the next begin end statement"
Begin["`Private1`"]
Aff[x_]:=x^2
End[]

Agg::usage="Agg[] is exported outside the next begin end statement"
Begin["`Private2`"]
Agg[x_]:=x^2
End[]

"And what is GIT and GIThub?" GIT create local version management in a folder on your disk (workbench workspace). GIT takes snapshots of all folders and files in that directory. You are able to switch back in versions per folder, or file. You are also able to compare files in different versions. Very important functionality when you have an existing application and want to change some things to improve. If it does not work you need to be able to go back to the "production" code. GIThub is more of sharing mechanism. The main "build" officer needs to be able to see all the "official" work of the programmers (in their workbench folder) in its team to create a team.

The remark in "a tree structure and not at all flexible" is not understood. I am after a bullet proof piece of software that a user can not by accident change disabling it functionality.

Would a user of this package "Main" normally only have ff as an available routine? Users would not normally dig into the tree structure using contexts or even know what the sublevel contexts are.

So the development of the application is like this: There is a master programmer. The master programmer has A level programmers who work for him and speak only to him. Each A level programmer may have B level programmers who work for them and speak only to them. In turn the B level programmers may have C level programmers, etc. The master programmer is the only one who speaks to the outside world.

This is an incredibly structured and organized system! Very authoritarian. Agreed that some projects may require it.

I would like to suggest that this is not the way that scientists or mathematicians work or the way research or study projects are organized. To begin with they don't know what the tree structure is in advance. If they knew they wouldn't be doing research or carrying out a study program. If they commit to a tree structure they will be limiting their flexibility and maybe biasing their thinking. Otherwise they must be continually revising the tree structure, which can get to be a messy chore.

These might be projects that involve just one or a small number of collaborators. Still, over time, they may assemble a very worthwhile application with multiple packages, documentation and accompanying notebooks. I believe these kind of users can be important to WRI, more so than very large scale industrial projects.

For these types of projects a looser parallel structure is needed. The folder structure for such an application might look like the following:

`$UserBaseDirectory`/Applications/
  ApplicationName/
   Documentation Folder,
   FrontEnd Folder
   Kernel/init.m
   PackageA.m
   PackageB.m
   PackageC.m

The three packages will all export routines to final users. In addition they may use routines exported by each other. Why might that happen? Because an organization by topic, or by individuals, may not correspond to a pure tree structure.

Here, I believe, is a way to fix it. Use a special form of init.m file for the application.

Declare the packages in the application.

ApplicationNamePackages = {"PackageA`", "PackageB`", "PackageC`"};

Context names to be added to $ContextPath in each package. Used in each package.

System`$ApplicationNameContextPaths = 
  "ApplicationName`" <> # & /@ ApplicationNamePackages;

Each package will then use the following statement after the BeginPackage statement. (As with Pieter only the first argument is used in the BeginPackage statement.)

$ContextPath=Union[$ContextPath,System`$ApplicationNameContextPaths];

Going back to the init.m file, the following routine will read the package Public symbols for each package, to establish contexts, but clear all the definitions including all definitions in the Private context. (It would be sort of nice if we had a command that would read only the Public part of a package and never touch the definitions in the Private section.)

ReadApplicationNamePackageNames[packageList_] :=
 Module[{fullPackageName, packageSymbolNames},
  Do[
    fullPackageName = "ApplicationName`" <> pname;
    Get[fullPackageName];
    packageSymbolNames = Names[fullPackageName <> "*"];
    Unprotect /@ packageSymbolNames;
    Clear /@ packageSymbolNames;
    Clear @@ {fullPackageName <> "Private`*"},
    {pname, packageList}];
  ]

The following does this for the packages.

ReadApplicationNamePackageNames[ApplicationNamePackages]; 

Finally, we reload all of the packages to obtain the definitions, now all placed in the correct context.

Get["ApplicationName`PackageA`"]
Get["ApplicationName`PackageB`"]
Get["ApplicationName`PackageC`"]

That is the end of the init.m file. The key requirement here is that all contexts for the application are known before any Private section code is established. It's possible that duplicate names might be used from different packages, which will produce shadowed warnings. But this can also occur in using routines from different applications or conflicting with WRI symbols. This is easily fixed and only a minor problem.

Perhaps this approach can be combined with the method described by Pieter. We would have more than one "Main" package, and these might or might not contain sub-trees. So in the above init.m code PackageA, PackageB and PackageC, would be three "Main" packages.