What is the best way to manage all the small programs you create in MMA?

I have finally reached the stage where I am writing useful tools in Mathematica. In general these are small text manipulation programs that clean up log files etc. However, I now have a new problem, how best to manage these ‘snippets’.

I am interested to know what more experienced developers do. For instance, do you create a “useful functions” module that you then include in new projects, or do you save them as the equivalent of Gists and just copy and paste them whenever you need them?

I am after something better than just saving them as notebooks, because I tend to use the notebooks in specific projects, and that leaves me with the problem of remembering where I last used a specific tool, and hence finding it.

I was hopeful that the Wolfram function repository might help me, but that is clearly aimed at more sophisticated functions. Is it possible to have a personal function repository?

Cheers
Andy

Creating a package in WorkBench is reasonably straightforward and has the attraction that you can create associated help documentation pages so you can reference options as well as include examples illustrating the use.

You can then install it as a package and Needs[“MyUtilities`”] will ensure that your favorite functionality is available.

Thanks Mark, I had never heard of WorkBench. I will go and take a look.

Cheers
Andy

Andy,

Create a package. Workbench is good for debugging but is likely overkill to get started. You call the package by typing Needs[].

The easiest way is to put all the functions in a single notebook (or call one notebook from another). In options inspector check off to save a package when you save the notebook (I’m not on my computer so I can’t look up the option’s exact title)

Regards

Neil

A low-maintenance alternative to managing packages is to simply run NotebookEvaluate[“~/path/to/myUtilities.nb”] in whatever notebook you want to use myUtilities functions and symbols.

I use this approach in my forthcoming book Functional Dataflow, and was happy to hear WRI’s publishing unit is adopting this method of separating utility functions to such Appendix notebooks.

Advantages:

  • No need to modify anything in the source notebook

  • Can be chained so that a target notebook need only call NotebookEvaluate once, and not the entire set of dependencies

  • If the source notebooks are managed eg Google Cloud or similar cloud filesystems, then you need only update at a single location. These updates will be auto-sync’d to other device filesystems.

Disadvantages:

  • No namespace control
  • No private/public distinction

Thank you Neil, and Alan. Both those suggestions sound great. I think I will try both approaches and see if one seems to fit my natural workflow better.

Cheers
Andy

Mathematica has a standard method, quite simple, to put together material in an Application, which is convenient for your own usage and easily transmitted to other users. I describe this in my essay at A Mathematica Style at my web site.

An Application is placed in your $UserBaseDirectory//Applications folder. It can contain notebooks, packages, palettes, style sheets, documentation, other types of files, and folders. The packages can be loaded with a simple statement, and Mathematica automatically knows how to access all the material. You can have just the items you want. You can just zip it up and transmit it to another person. You can have private development folders that you delete from the public distribution. If you start out with this format you won’t have to reorganize or jerry-build when you add new things.

Workbench, Wolfram Workbench: Eclipse-based IDE for Wolfram Language, is a standard-issue Wolfram product and, as near as I can tell, the only officially endorsed mechanism to create documentation which is integrated into the help system.

It is pretty simple once you get it set up.

Hi David,
Thank you for the link to your essay, that looks like a really useful approach. I will give that a try too.

Cheers
Andy

Dear Andy,

I really appreciate you asking this question as I have also run into the same issues at times so I assume there are other users of the Wolfram Language who have advanced to the point that just developing in notebooks is no longer the ideal solution. I agree with many commenters that creating a package file is likely the easiest solution for you current problem but I thought I would add some additional food for thought as you continue to progress as a user of the Wolfram Language.

If you are interested in creating code that will be used again and again, I would recommend using and an Integrated Development Environment (or IDE) as many have suggested. While I think Wolfram Workbench by Eclipse is the only officially supported IDE for Wolfram Language, there are now several plug-ins for other IDEs that add support for Wolfram Language so Workbench is not your only option. I personally use IntelliJ IDEA with the Wolfram Language plug-in and I really like it as IntelliJ has a lot of great features for developers. Here’s a link to the available Wolfram Language IDE options: Wolfram Resources for Software Developers

I would also recommend setting up a git repository (rep) at places such as GitHub or or BitBucket so that managing your code becomes much easier than keeping code in various notebooks. Creating a git repo also allows you to retrieve previous versions of code easily and have friends and colleagues review and test your code.

My last recommendation would be to set up a paclet for your various package files so all of your functions could be accessible in the same context (which is easier than calling Get or Needs on individual package files) and can include test files too. Unfortunately, there are not too many examples of Wolfram Language paclets on the web nor detailed documentation on setting up paclets but I would be happy to send you a notebook with instructions and code templates for setting up a paclet if you are interested. I hope this was helpful.

Cheers,
Micah

Actually, I think the Resource System (as related to, but distinct from the public Wofram function repository) is a great solution for “snippets” like this as you call them. Simply create a Resource Function! Resource Functions do NOT have to be pushed to the WFR, but can be deployed locally on your machine, or sharably in the cloud. For local publication, you don’t even need to write documentation (tho, even for snippets, I think it is usually worth it to write down a basic usage).

Anyway just an idea. The same menu that gets you “Submit to FunctionRepository” will also get you options for “deploy in this session”, “deploy in the cloud”, and “deploy on this computer”. If you choose the last of these, the function will always be available (without any pre-loading) via ResourceFunction[“MyNeatFunction”].

Hope this helps!

Hi Micah,
Thanks very much for your thoughts. I would be very interested to look at the paclets notebook, it sounds very encouraging.

I am also really interested in the various IDEs that people are using. I am gradually understanding the different ways in which I could use the Wolfram language. There seems to be a breakout when one moves from exploring a problem - notebooks seem to work well for that - and actually coding a solution, where perhaps the IDEs make more sense.

I am also conscious that my thinking has been coloured by the fact that I usually use Smalltalk, and therefore I am bringing a whole set of assumptions to this particular problem.

Cheers
Andy

Hi Paco,
I am definitely going to explore that option. I certainly like the idea of being able to instantly access any functions I have created. Out of curiosity, is there a simple way to list the functions? I realise I am trying to convert MMA into Smalltalk! but it would be useful to have some sort of function browser. I am sure my list of functions will grow over time. Actually, is there a way to add them to the help system? Perhaps that would be useful too.

By the way, for the future reference of anyone reading this thread, the menu that Paco is referencing is not in the Mathematica program, but rather in a notebook that one downloads from WFR - it took me a little while to work that out.

Cheers
Andy

Andy, yes, kind of. Your locally cached resource functions can be listed with ResourceFunction["LocalResourceObjects"]["Function"]. This includes both ones you’ve deployed locally, and ones you’ve previously downloaded from the WFR. Another resource that might be useful to you for maintaining a list of your frequently used resources is ResourceFunction["CreateResourceObjectGallery"].

Note:
You can also get the notebook for starting a new Resource Function submission through the File menu of the desktop product in versions 11.3+ :slight_smile:

I may be biased, but in my opinion, one of the most useful features of resource functions is the ability to easily save a function (including necessary dependent functions) for later use. For example, I just recently wrote this function AssociateToNested:

In[1]:= AssociateToNested[a, {"a", "b"} -> 1]
In[2]:= a
Out[2]= <|"a" -> <|"b" -> 1|>|>

In[3]:= AssociateToNested[a, {{"a", "c", "d"} -> f[x], "b" :> 1 + 1}]
In[4]:= a
Out[4]= <|"a" -> <|"b" -> 1, "c" -> <|"d" -> f[x]|>|>, "b" :> 1 + 1|>

I used to save my personal functions in an auto loading paclet on my machine, but these days I’ve found it to be much easier to just store them as resource functions. For example, this deploys it on my local machine:

In[5]:= LocalCache @ DefineResourceFunction @ AssociateToNested
Out[5]= LocalObject["file:///C:/Users/rhennigan/AppData/Roaming/Wolfram/Objects/49c7018f-75f7-4e51-b37b-ca75bced73bd"]

I didn’t have to track down all the code I wrote in my notebook and copy and paste it into a separate package. The function is now usable any time on my machine automagically even after quitting the kernel:

In[6]:= Quit[]

In[1]:= ResourceFunction["AssociateToNested"][a, {"a", "b"} -> x]
In[2]:= a
Out[2]= <|"a" -> <|"b" -> x|>|>

Similarly, I can save functions to my cloud account with:

In[1]:= CloudDeploy[ DefineResourceFunction @ AssociateToNested, 
                     "ResourceFunctions/AssociateToNested",
                     Permissions -> "Public"
        ]

Out[1]= CloudObject["https://www.wolframcloud.com/obj/rhennigan/ResourceFunctions/AssociateToNested"]

Since I gave it public permissions, now anyone can access and use it with:

ResourceFunction["user:rhennigan/ResourceFunctions/AssociateToNested"]

I think some people might look at some of the well-documented functions on the public Wolfram Function Repository and think the process of creating a ResourceFunction might be a daunting task, but for your personal function deployments you don’t need any documentation at all.

As a bonus example, one of the best resource functions ever created can’t be found on the Wolfram Function Repository: https://www.wolframcloud.com/obj/rhennigan/DeployedResources/Function/BirdPeek

Making user-defined functions 2nd-class citizens is the same uglyness as having to import packages - something that WRI explicitly tries to automate - yet we have this ugly syntax:

ResourceFunction[“AssociateToNested”][a, {“a”, “b”} → x]

versus 1-st class functions:

AssociateToNested[a, {“a”, “b”} → x]

Even Python’s syntax is preferable.

Long function names are preferred to short for exactly this reason.

You can dump and restore all Your kernel expressions to file to “quickly” restore a session having “all your stuff ready”. here’s how to do one expression only:

x>>filename;  (* see Put and moreso DumpSave[] in Help *)
y<<filename;

So you don’t have to store them all. If you did you’d use a library system (library science is a college course).

Back to long names: you may need to replace some names and get a compatibility problem. the answer is to use fully long names and append more to the name if there is a compatibility problem.

You should avoid using nick names and short names because you’ll run into naming collisions or forget what the nick names are in some years.

Long names are best.

use Context[] to your best advantage. you don’t have to be making a Package to use Contexts

Personally, I don’t have a huge collection so i have a local package that loads Wolfram packages that i use in “the average full session”, including a few of my own.

But I often “need a blank slate” - so unless i have two kernels two front ends I don’t load anything. Also I am susceptible to power outage and forced reboots from automatic system updates. So I don’t want the extra wait of loading libraries unless: I call out to use them.

Andy, I think you’ve started a very interesting and useful discussion.

However, I think good answers depend a lot on what your objective was in turning to Mathematica in the first place. Mathematica is a nice application to use for many reasons - but I just wonder what your specific reason is?

Hi Richard,
Thank you for your very detailed explanation. What you are proposing is very close to what I was looking for. However, I agree with Alan that the need to wrap my function in a ResourceFunction[] call adds to the cognitive overhead, i.e. I have to remember to treat one function differently from another. Having said that, I shall definitely experiment, because the overhead may prove to be insignificant.

By the way, I tried loading your published function, but couldn’t get it to work. My input/output looks like the below. Am I doing something wrong?

In[202]:= ResourceFunction[“user:rhennigan/ResourceFunctions/AssociateToNested”]
Out[202]= AssociateToNested
In[203]:= ResourceFunction[“AssociateToNested”][a,{“a”,“b”}->x]
During evaluation of In[203]:= ResourceAcquire::apierr: The specified resource could not be found.
Out[203]= $Failed[a,{a,b}->x]