Message Boards Message Boards

What is the best way to manage all the small programs you create in MMA?

Posted 5 years ago

I have finally reached the stage where I am writing useful tools in Mathematica. In general these are small text manipulation programs that clean up log files etc. However, I now have a new problem, how best to manage these 'snippets'.

I am interested to know what more experienced developers do. For instance, do you create a "useful functions" module that you then include in new projects, or do you save them as the equivalent of Gists and just copy and paste them whenever you need them?

I am after something better than just saving them as notebooks, because I tend to use the notebooks in specific projects, and that leaves me with the problem of remembering where I last used a specific tool, and hence finding it.

I was hopeful that the Wolfram function repository might help me, but that is clearly aimed at more sophisticated functions. Is it possible to have a personal function repository?

Cheers Andy

POSTED BY: Andrew Burnett
27 Replies
Posted 5 years ago

Thank you to everyone who took the time to respond to my question. I think I have plenty to experiment with. I will update this thread, in a few months, when I have had a chance to try out the various options. Hopefully, the update will be useful for anyone stumbling across this question in the future.

POSTED BY: Andrew Burnett

Hi Andy,

First you should become reasonably familiar with Mathematica. The best way to do this is to read the 'Introduction for Programmers' and 'An Elementary Introduction to the Wolfram Language'. by Stephen Wolfram, both available at the bottom of the Main Documentation page in Help. You may already have read these.

Mathematica is not Smalltalk or any other computer language. I think it pretty much transcends these other languages and it's not too useful to try to convert it to one of them. The time you might spend in doing that is lost to learning concepts and methods of machine learning.

I don't know why you should get immersed in an IDE. (Except possibly Workbench - but only for the purpose of documentation.) An external IDE is not Mathematica. It has nothing to do with Mathematica. In a very practical sense Mathematica is its own IDE. All the time you spend learning an IDE is time not spent on studying machine learning.

You don't need an IDE for debugging. Just put temporary Print statements in your code (usually inside a Module) to display intermediate results. I've found that always gets me to the problem and the solution.

I've been using Mathematica since 1996, just before Version 3 came out. I wrote most of the code and did much of the design of a Tensorial application (with Renan Cabrera), and a UnitsHelper package. And wrote and sold hundreds of copies of a Presentations package that started out just the way you are accumulating routines. I've been working for some time on a quite large Grassmann Algebra/Calculus application with John Browne. So my essay advice came from quite a bit of experience.

Package routines often grow out of notebook work. I put a Routines Section at the top of the notebook (also known as Package Purgatory) and put initial routines there. I always include a usage statement, a SyntaxInformation statement, and the routine itself. (Sometimes there are also Options, Attributes and error messages.) I make the routine an Initialization cell and use it for a while in the notebook. Then only after testing and using the routine for some time do I copy it into a package file (package Heaven). A package file in appearance is very much like a regular notebook. It can have Section headers and Text cells as well as the Initialization cells for the code. The entire process is easy and smooth. And the result meshes with the normal use of Mathematica.

Andy, I think you've started a very interesting and useful discussion.

However, I think good answers depend a lot on what your objective was in turning to Mathematica in the first place. Mathematica is a nice application to use for many reasons - but I just wonder what your specific reason is?

Posted 5 years ago

Hi David, Ah yes, clarification - always a good idea! The back story is that I decided to spend a year using MMA exclusively (as far as possible). I really want to understand what I can and can't do in the system. Or, perhaps to put it more accurately, what I should and shouldn't use it for. I am particularly interested in exploring how machine learning might change the way I solve problems.

So far, I have found myself using MMA in two distinct modes: 1) exploring a problem, for which the notebook is a good model 2) building small utility applications, which I will want to reuse in whole, or part.

Because I come from a purely Object Oriented background (Smalltalk), I find myself thinking of functions as if they were objects, and that leads me to want to create instances of functions, hence the need to understand how to save, and reuse, with the minimum effort.

I went back and reread your essay on how to structure applications, and I found that very useful. I am going to adopt that structure when building this little utilities and see if it helps. I think I will also try Alan's suggestion about capturing my functions in a notebook, and then evaluating it at the start of each project. If I understand correctly, that will allow me to access the functions without wrapping them in any other function call.

In addition to the above, I am going to try out the IDE experiments. I am not sure at what point, in the development of a solution, one switches from the notebook to the IDE. Time will tell.

POSTED BY: Andrew Burnett
Anonymous User
Anonymous User
Posted 5 years ago

use Context[] to your best advantage. you don't have to be making a Package to use Contexts

Personally, I don't have a huge collection so i have a local package that loads Wolfram packages that i use in "the average full session", including a few of my own.

But I often "need a blank slate" - so unless i have two kernels two front ends I don't load anything. Also I am susceptible to power outage and forced reboots from automatic system updates. So I don't want the extra wait of loading libraries unless: I call out to use them.

POSTED BY: Anonymous User
Posted 5 years ago

Hi John, Could you explain a little more about how I use Context[]. I looked at the help, but it wasn't obvious to me how I would set things up to load my utility functions.

Thanks Andy

POSTED BY: Andrew Burnett
Anonymous User
Anonymous User
Posted 5 years ago

Long function names are preferred to short for exactly this reason.

You can dump and restore all Your kernel expressions to file to "quickly" restore a session having "all your stuff ready". here's how to do one expression only:

x>>filename;  (* see Put and moreso DumpSave[] in Help *)
y<<filename;

So you don't have to store them all. If you did you'd use a library system (library science is a college course).

Back to long names: you may need to replace some names and get a compatibility problem. the answer is to use fully long names and append more to the name if there is a compatibility problem.

You should avoid using nick names and short names because you'll run into naming collisions or forget what the nick names are in some years.

Long names are best.

POSTED BY: Anonymous User

Making user-defined functions 2nd-class citizens is the same uglyness as having to import packages - something that WRI explicitly tries to automate - yet we have this ugly syntax:

ResourceFunction["AssociateToNested"][a, {"a", "b"} -> x]

versus 1-st class functions:

AssociateToNested[a, {"a", "b"} -> x]

Even Python's syntax is preferable.

POSTED BY: Alan Calvitti

You could just do:

 AssociateToNested = ResourceFunction["AssociateToNested"];

And then it is 1st-class in your view (-:

POSTED BY: Sander Huisman

Can you please submit to WFR a function that auto-converts f = ResourceFunction["f"] - I have hundreds in just my data toolkit - and resolves potential name-clashes? - The latter should be done on Wolfram's side, since they are in a position to view all definitions and can flag them.

POSTED BY: Alan Calvitti

I may be biased, but in my opinion, one of the most useful features of resource functions is the ability to easily save a function (including necessary dependent functions) for later use. For example, I just recently wrote this function AssociateToNested:

In[1]:= AssociateToNested[a, {"a", "b"} -> 1]
In[2]:= a
Out[2]= <|"a" -> <|"b" -> 1|>|>

In[3]:= AssociateToNested[a, {{"a", "c", "d"} -> f[x], "b" :> 1 + 1}]
In[4]:= a
Out[4]= <|"a" -> <|"b" -> 1, "c" -> <|"d" -> f[x]|>|>, "b" :> 1 + 1|>

I used to save my personal functions in an auto loading paclet on my machine, but these days I've found it to be much easier to just store them as resource functions. For example, this deploys it on my local machine:

In[5]:= LocalCache @ DefineResourceFunction @ AssociateToNested
Out[5]= LocalObject["file:///C:/Users/rhennigan/AppData/Roaming/Wolfram/Objects/49c7018f-75f7-4e51-b37b-ca75bced73bd"]

I didn't have to track down all the code I wrote in my notebook and copy and paste it into a separate package. The function is now usable any time on my machine automagically even after quitting the kernel:

In[6]:= Quit[]

In[1]:= ResourceFunction["AssociateToNested"][a, {"a", "b"} -> x]
In[2]:= a
Out[2]= <|"a" -> <|"b" -> x|>|>

Similarly, I can save functions to my cloud account with:

In[1]:= CloudDeploy[ DefineResourceFunction @ AssociateToNested, 
                     "ResourceFunctions/AssociateToNested",
                     Permissions -> "Public"
        ]

Out[1]= CloudObject["https://www.wolframcloud.com/obj/rhennigan/ResourceFunctions/AssociateToNested"]

Since I gave it public permissions, now anyone can access and use it with:

ResourceFunction["user:rhennigan/ResourceFunctions/AssociateToNested"]

I think some people might look at some of the well-documented functions on the public Wolfram Function Repository and think the process of creating a ResourceFunction might be a daunting task, but for your personal function deployments you don't need any documentation at all.

As a bonus example, one of the best resource functions ever created can't be found on the Wolfram Function Repository: https://www.wolframcloud.com/obj/rhennigan/DeployedResources/Function/BirdPeek

POSTED BY: Richard Hennigan
Posted 5 years ago

Hi Richard, Thank you for your very detailed explanation. What you are proposing is very close to what I was looking for. However, I agree with Alan that the need to wrap my function in a ResourceFunction[] call adds to the cognitive overhead, i.e. I have to remember to treat one function differently from another. Having said that, I shall definitely experiment, because the overhead may prove to be insignificant.

By the way, I tried loading your published function, but couldn't get it to work. My input/output looks like the below. Am I doing something wrong?

In[202]:= ResourceFunction["user:rhennigan/ResourceFunctions/AssociateToNested"] Out[202]= AssociateToNested In[203]:= ResourceFunction["AssociateToNested"][a,{"a","b"}->x] During evaluation of In[203]:= ResourceAcquire::apierr: The specified resource could not be found. Out[203]= $Failed[a,{a,b}->x]

POSTED BY: Andrew Burnett
Posted 5 years ago

Hi Paco, I am definitely going to explore that option. I certainly like the idea of being able to instantly access any functions I have created. Out of curiosity, is there a simple way to list the functions? I realise I am trying to convert MMA into Smalltalk! but it would be useful to have some sort of function browser. I am sure my list of functions will grow over time. Actually, is there a way to add them to the help system? Perhaps that would be useful too.

By the way, for the future reference of anyone reading this thread, the menu that Paco is referencing is not in the Mathematica program, but rather in a notebook that one downloads from WFR - it took me a little while to work that out.

Cheers Andy

POSTED BY: Andrew Burnett

Andy, yes, kind of. Your locally cached resource functions can be listed with ResourceFunction["LocalResourceObjects"]["Function"]. This includes both ones you've deployed locally, and ones you've previously downloaded from the WFR. Another resource that might be useful to you for maintaining a list of your frequently used resources is ResourceFunction["CreateResourceObjectGallery"].

Note: You can also get the notebook for starting a new Resource Function submission through the File menu of the desktop product in versions 11.3+ :)enter image description here

POSTED BY: Paco Jain

Actually, I think the Resource System (as related to, but distinct from the public Wofram function repository) is a great solution for "snippets" like this as you call them. Simply create a Resource Function! Resource Functions do NOT have to be pushed to the WFR, but can be deployed locally on your machine, or sharably in the cloud. For local publication, you don't even need to write documentation (tho, even for snippets, I think it is usually worth it to write down a basic usage).

Anyway just an idea. The same menu that gets you "Submit to FunctionRepository" will also get you options for "deploy in this session", "deploy in the cloud", and "deploy on this computer". If you choose the last of these, the function will always be available (without any pre-loading) via ResourceFunction["MyNeatFunction"].

Hope this helps!

POSTED BY: Paco Jain

Dear Andy,

I really appreciate you asking this question as I have also run into the same issues at times so I assume there are other users of the Wolfram Language who have advanced to the point that just developing in notebooks is no longer the ideal solution. I agree with many commenters that creating a package file is likely the easiest solution for you current problem but I thought I would add some additional food for thought as you continue to progress as a user of the Wolfram Language.

If you are interested in creating code that will be used again and again, I would recommend using and an Integrated Development Environment (or IDE) as many have suggested. While I think Wolfram Workbench by Eclipse is the only officially supported IDE for Wolfram Language, there are now several plug-ins for other IDEs that add support for Wolfram Language so Workbench is not your only option. I personally use IntelliJ IDEA with the Wolfram Language plug-in and I really like it as IntelliJ has a lot of great features for developers. Here's a link to the available Wolfram Language IDE options: http://www.wolfram.com/developer/?source=frontpage-computation

I would also recommend setting up a git repository (rep) at places such as GitHub or or BitBucket so that managing your code becomes much easier than keeping code in various notebooks. Creating a git repo also allows you to retrieve previous versions of code easily and have friends and colleagues review and test your code.

My last recommendation would be to set up a paclet for your various package files so all of your functions could be accessible in the same context (which is easier than calling Get or Needs on individual package files) and can include test files too. Unfortunately, there are not too many examples of Wolfram Language paclets on the web nor detailed documentation on setting up paclets but I would be happy to send you a notebook with instructions and code templates for setting up a paclet if you are interested. I hope this was helpful.

Cheers, Micah

POSTED BY: Micah Lindley
Posted 5 years ago

Hi Micah, Thanks very much for your thoughts. I would be very interested to look at the paclets notebook, it sounds very encouraging.

I am also really interested in the various IDEs that people are using. I am gradually understanding the different ways in which I could use the Wolfram language. There seems to be a breakout when one moves from exploring a problem - notebooks seem to work well for that - and actually coding a solution, where perhaps the IDEs make more sense.

I am also conscious that my thinking has been coloured by the fact that I usually use Smalltalk, and therefore I am bringing a whole set of assumptions to this particular problem.

Cheers Andy

POSTED BY: Andrew Burnett

Dear Andy,

How are you today? Here's the notebook I promised you a couple of weeks ago. Let me know if you have any questions.

Cheers, Micah

Attachments:
POSTED BY: Micah Lindley
Posted 5 years ago

Hi Micha, Many thanks for this. I shall enjoy digging into it

Cheers Andy

POSTED BY: Andrew Burnett
Posted 5 years ago

Hi David, Thank you for the link to your essay, that looks like a really useful approach. I will give that a try too.

Cheers Andy

POSTED BY: Andrew Burnett

Mathematica has a standard method, quite simple, to put together material in an Application, which is convenient for your own usage and easily transmitted to other users. I describe this in my essay at A Mathematica Style at my web site.

An Application is placed in your $UserBaseDirectory//Applications folder. It can contain notebooks, packages, palettes, style sheets, documentation, other types of files, and folders. The packages can be loaded with a simple statement, and Mathematica automatically knows how to access all the material. You can have just the items you want. You can just zip it up and transmit it to another person. You can have private development folders that you delete from the public distribution. If you start out with this format you won't have to reorganize or jerry-build when you add new things.

Posted 5 years ago

Thank you Neil, and Alan. Both those suggestions sound great. I think I will try both approaches and see if one seems to fit my natural workflow better.

Cheers Andy

POSTED BY: Andrew Burnett

A low-maintenance alternative to managing packages is to simply run NotebookEvaluate["~/path/to/myUtilities.nb"] in whatever notebook you want to use myUtilities functions and symbols.

I use this approach in my forthcoming book Functional Dataflow, and was happy to hear WRI's publishing unit is adopting this method of separating utility functions to such Appendix notebooks.

Advantages:

  • No need to modify anything in the source notebook

  • Can be chained so that a target notebook need only call NotebookEvaluate once, and not the entire set of dependencies

  • If the source notebooks are managed eg Google Cloud or similar cloud filesystems, then you need only update at a single location. These updates will be auto-sync'd to other device filesystems.

Disadvantages:

  • No namespace control
  • No private/public distinction
POSTED BY: Alan Calvitti

Andy,

Create a package. Workbench is good for debugging but is likely overkill to get started. You call the package by typing Needs[].

The easiest way is to put all the functions in a single notebook (or call one notebook from another). In options inspector check off to save a package when you save the notebook (I’m not on my computer so I can’t look up the option’s exact title)

Regards

Neil

POSTED BY: Neil Singer
Posted 5 years ago

Thanks Mark, I had never heard of WorkBench. I will go and take a look.

Cheers Andy

POSTED BY: Andrew Burnett

Workbench, http://www.wolfram.com/workbench/, is a standard-issue Wolfram product and, as near as I can tell, the only officially endorsed mechanism to create documentation which is integrated into the help system.

It is pretty simple once you get it set up.

POSTED BY: Mark Kotanchek

Creating a package in WorkBench is reasonably straightforward and has the attraction that you can create associated help documentation pages so you can reference options as well as include examples illustrating the use.

You can then install it as a package and Needs["MyUtilities`"] will ensure that your favorite functionality is available.

POSTED BY: Mark Kotanchek
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract