Group Abstract Group Abstract

Message Boards Message Boards

What is the best way to manage all the small programs you create in MMA?

Posted 6 years ago

I have finally reached the stage where I am writing useful tools in Mathematica. In general these are small text manipulation programs that clean up log files etc. However, I now have a new problem, how best to manage these 'snippets'.

I am interested to know what more experienced developers do. For instance, do you create a "useful functions" module that you then include in new projects, or do you save them as the equivalent of Gists and just copy and paste them whenever you need them?

I am after something better than just saving them as notebooks, because I tend to use the notebooks in specific projects, and that leaves me with the problem of remembering where I last used a specific tool, and hence finding it.

I was hopeful that the Wolfram function repository might help me, but that is clearly aimed at more sophisticated functions. Is it possible to have a personal function repository?

Cheers Andy

POSTED BY: Andrew Burnett
27 Replies
Posted 6 years ago

Thank you to everyone who took the time to respond to my question. I think I have plenty to experiment with. I will update this thread, in a few months, when I have had a chance to try out the various options. Hopefully, the update will be useful for anyone stumbling across this question in the future.

POSTED BY: Andrew Burnett

Hi Andy,

First you should become reasonably familiar with Mathematica. The best way to do this is to read the 'Introduction for Programmers' and 'An Elementary Introduction to the Wolfram Language'. by Stephen Wolfram, both available at the bottom of the Main Documentation page in Help. You may already have read these.

Mathematica is not Smalltalk or any other computer language. I think it pretty much transcends these other languages and it's not too useful to try to convert it to one of them. The time you might spend in doing that is lost to learning concepts and methods of machine learning.

I don't know why you should get immersed in an IDE. (Except possibly Workbench - but only for the purpose of documentation.) An external IDE is not Mathematica. It has nothing to do with Mathematica. In a very practical sense Mathematica is its own IDE. All the time you spend learning an IDE is time not spent on studying machine learning.

You don't need an IDE for debugging. Just put temporary Print statements in your code (usually inside a Module) to display intermediate results. I've found that always gets me to the problem and the solution.

I've been using Mathematica since 1996, just before Version 3 came out. I wrote most of the code and did much of the design of a Tensorial application (with Renan Cabrera), and a UnitsHelper package. And wrote and sold hundreds of copies of a Presentations package that started out just the way you are accumulating routines. I've been working for some time on a quite large Grassmann Algebra/Calculus application with John Browne. So my essay advice came from quite a bit of experience.

Package routines often grow out of notebook work. I put a Routines Section at the top of the notebook (also known as Package Purgatory) and put initial routines there. I always include a usage statement, a SyntaxInformation statement, and the routine itself. (Sometimes there are also Options, Attributes and error messages.) I make the routine an Initialization cell and use it for a while in the notebook. Then only after testing and using the routine for some time do I copy it into a package file (package Heaven). A package file in appearance is very much like a regular notebook. It can have Section headers and Text cells as well as the Initialization cells for the code. The entire process is easy and smooth. And the result meshes with the normal use of Mathematica.

Posted 6 years ago

Hi David, Ah yes, clarification - always a good idea! The back story is that I decided to spend a year using MMA exclusively (as far as possible). I really want to understand what I can and can't do in the system. Or, perhaps to put it more accurately, what I should and shouldn't use it for. I am particularly interested in exploring how machine learning might change the way I solve problems.

So far, I have found myself using MMA in two distinct modes: 1) exploring a problem, for which the notebook is a good model 2) building small utility applications, which I will want to reuse in whole, or part.

Because I come from a purely Object Oriented background (Smalltalk), I find myself thinking of functions as if they were objects, and that leads me to want to create instances of functions, hence the need to understand how to save, and reuse, with the minimum effort.

I went back and reread your essay on how to structure applications, and I found that very useful. I am going to adopt that structure when building this little utilities and see if it helps. I think I will also try Alan's suggestion about capturing my functions in a notebook, and then evaluating it at the start of each project. If I understand correctly, that will allow me to access the functions without wrapping them in any other function call.

In addition to the above, I am going to try out the IDE experiments. I am not sure at what point, in the development of a solution, one switches from the notebook to the IDE. Time will tell.

POSTED BY: Andrew Burnett
Anonymous User
Anonymous User
Posted 6 years ago

use Context[] to your best advantage. you don't have to be making a Package to use Contexts

Personally, I don't have a huge collection so i have a local package that loads Wolfram packages that i use in "the average full session", including a few of my own.

But I often "need a blank slate" - so unless i have two kernels two front ends I don't load anything. Also I am susceptible to power outage and forced reboots from automatic system updates. So I don't want the extra wait of loading libraries unless: I call out to use them.

POSTED BY: Anonymous User
Posted 6 years ago
POSTED BY: Andrew Burnett
Anonymous User
Anonymous User
Posted 6 years ago

Long function names are preferred to short for exactly this reason.

You can dump and restore all Your kernel expressions to file to "quickly" restore a session having "all your stuff ready". here's how to do one expression only:

x>>filename;  (* see Put and moreso DumpSave[] in Help *)
y<<filename;

So you don't have to store them all. If you did you'd use a library system (library science is a college course).

Back to long names: you may need to replace some names and get a compatibility problem. the answer is to use fully long names and append more to the name if there is a compatibility problem.

You should avoid using nick names and short names because you'll run into naming collisions or forget what the nick names are in some years.

Long names are best.

POSTED BY: Anonymous User

Making user-defined functions 2nd-class citizens is the same uglyness as having to import packages - something that WRI explicitly tries to automate - yet we have this ugly syntax:

ResourceFunction["AssociateToNested"][a, {"a", "b"} -> x]

versus 1-st class functions:

AssociateToNested[a, {"a", "b"} -> x]

Even Python's syntax is preferable.

POSTED BY: Alan Calvitti

You could just do:

 AssociateToNested = ResourceFunction["AssociateToNested"];

And then it is 1st-class in your view (-:

POSTED BY: Sander Huisman

Can you please submit to WFR a function that auto-converts f = ResourceFunction["f"] - I have hundreds in just my data toolkit - and resolves potential name-clashes? - The latter should be done on Wolfram's side, since they are in a position to view all definitions and can flag them.

POSTED BY: Alan Calvitti
POSTED BY: Richard Hennigan
Posted 6 years ago
POSTED BY: Andrew Burnett
Posted 6 years ago
POSTED BY: Andrew Burnett

Andy, yes, kind of. Your locally cached resource functions can be listed with ResourceFunction["LocalResourceObjects"]["Function"]. This includes both ones you've deployed locally, and ones you've previously downloaded from the WFR. Another resource that might be useful to you for maintaining a list of your frequently used resources is ResourceFunction["CreateResourceObjectGallery"].

Note: You can also get the notebook for starting a new Resource Function submission through the File menu of the desktop product in versions 11.3+ :)enter image description here

POSTED BY: Paco Jain
POSTED BY: Paco Jain
POSTED BY: Micah Lindley
Posted 6 years ago
POSTED BY: Andrew Burnett

Dear Andy,

How are you today? Here's the notebook I promised you a couple of weeks ago. Let me know if you have any questions.

Cheers, Micah

Attachments:
POSTED BY: Micah Lindley
Posted 6 years ago

Hi Micha, Many thanks for this. I shall enjoy digging into it

Cheers Andy

POSTED BY: Andrew Burnett
Posted 6 years ago

Hi David, Thank you for the link to your essay, that looks like a really useful approach. I will give that a try too.

Cheers Andy

POSTED BY: Andrew Burnett
Posted 6 years ago

Thank you Neil, and Alan. Both those suggestions sound great. I think I will try both approaches and see if one seems to fit my natural workflow better.

Cheers Andy

POSTED BY: Andrew Burnett

A low-maintenance alternative to managing packages is to simply run NotebookEvaluate["~/path/to/myUtilities.nb"] in whatever notebook you want to use myUtilities functions and symbols.

I use this approach in my forthcoming book Functional Dataflow, and was happy to hear WRI's publishing unit is adopting this method of separating utility functions to such Appendix notebooks.

Advantages:

  • No need to modify anything in the source notebook

  • Can be chained so that a target notebook need only call NotebookEvaluate once, and not the entire set of dependencies

  • If the source notebooks are managed eg Google Cloud or similar cloud filesystems, then you need only update at a single location. These updates will be auto-sync'd to other device filesystems.

Disadvantages:

  • No namespace control
  • No private/public distinction
POSTED BY: Alan Calvitti

Andy,

Create a package. Workbench is good for debugging but is likely overkill to get started. You call the package by typing Needs[].

The easiest way is to put all the functions in a single notebook (or call one notebook from another). In options inspector check off to save a package when you save the notebook (I’m not on my computer so I can’t look up the option’s exact title)

Regards

Neil

POSTED BY: Neil Singer
Posted 6 years ago

Thanks Mark, I had never heard of WorkBench. I will go and take a look.

Cheers Andy

POSTED BY: Andrew Burnett

Workbench, http://www.wolfram.com/workbench/, is a standard-issue Wolfram product and, as near as I can tell, the only officially endorsed mechanism to create documentation which is integrated into the help system.

It is pretty simple once you get it set up.

POSTED BY: Mark Kotanchek

Creating a package in WorkBench is reasonably straightforward and has the attraction that you can create associated help documentation pages so you can reference options as well as include examples illustrating the use.

You can then install it as a package and Needs["MyUtilities`"] will ensure that your favorite functionality is available.

POSTED BY: Mark Kotanchek
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard