Community RSS Feed
http://community.wolfram.com
RSS Feed for Wolfram Community showing any discussions in tag Packages sorted by activePJLink: Hooking up Mathematica and Python
http://community.wolfram.com/groups/-/m/t/1468475
Here's a cross-post of something I originally wrote [here](https://www.wolframcloud.com/objects/b3m2a1/home/pjlink-hooking-up-mathematica-and-python.html) about how to get python and Mathematica to work together like JLink.
I thought this might have broad appeal here.
PJLink: Hooking up Mathematica and Python
---
Mathematica is an incredibly powerful platform with a fun and intellectually pleasing language, but is incredibly expensive and closed source. Python is a convenient, pretty powerful language with a lot of support from the developer community. For as long as the two have existed people have been trying to tie them together, but very little has been done to do so at the native level with efficient, convenient exchange between the two. That's why over the past few weeks I took the time to build a clean, convenient link between the two. This post will go into how the link was built and some of its features, but first I think a little demo is appropriate.
## A Quick Demo
### Installing PJLink
The link is based off of the [J/Link](https://reference.wolfram.com/language/JLink/tutorial/Overview.html) interface built into Mathematica for hooking up Java and Mathematica. To wit, I called it [PJ/Link](https://github.com/b3m2a1/PJLink) . It lives on my paclet server as well as GitHub, so we can easily install it from there:
PacletInstall["PJLink", "Site"->"http://www.wolframcloud.com/objects/b3m2a1.paclets/PacletServer"]
(*Out:*)
![hookingupmathematicaandpython-274752603667507597](https://www.wolframcloud.com/objects/b3m2a1/home/img/hookingupmathematicaandpython-274752603667507597.png)
### Loading PJLink in Jupyter
For this demo we'll need the path to this thing as well (note that the version might change in the future):
%["Location"]
(*Out:*)
"~/Library/Mathematica/Paclets/Repository/PJLink-1.0.0"
Now we'll leave Mathematica and open up a Jupyter notebook:
![hookingupmathematicaandpython-921972091567300718](https://www.wolframcloud.com/objects/b3m2a1/home/img/hookingupmathematicaandpython-921972091567300718.png)
Next we'll get that path available so we can actually make use of the package. Then we'll load things from the subsidiary ```SubprocessKernel``` package which is included in the paclet and makes use of PJLink:
<pre>import os, sys
pjlink_path = &quot;~/Library/Mathematica/Paclets/Repository/PJLink-1.0.0&quot; #this is whatever path was extracted before
sys.path.insert(0, os.path.expanduser (pjlink_path))
from SubprocessKernel import SubprocessKernel
from SubprocessKernel import MathematicaBlock, LinkEnvironment
## these are helpers I&apos; ll use in the demo</pre>
![hookingupmathematicaandpython-695494794636171070](https://www.wolframcloud.com/objects/b3m2a1/home/img/hookingupmathematicaandpython-695494794636171070.png)
### Bidirectional Communication
Once we have this we can start a subprocess kernel which will open a Mathematica front-end to interact with. We'll also start and evaluator Mathematica can use to call back into python.
You may see a long string of output from your C compiler as the setup.py file builds out the native library that PJLink uses. Don't worry, this should only happen once. If it fails, raise an issue on [GitHub](https://github.com/b3m2a1/PJLink/issues) so I can deal with it.
Once Mathematica has loaded, we'll use the ```MathematicaBlock``` context manager so we can write something that looks a lot like Mathematica code and use the ```MEval``` function we'll define to run the code. That code for all this looks like:
<pre>ker = SubprocessKernel()
def MEval (expr, wait = True, kernel = ker) :
&quot;&quot; &quot;MEval evaluates a Mathematica expression in the Mathematica kernel
&quot; &quot;&quot;
kernel.drain() # just to make sure things are clen
return kernel.evaluate (expr, wait = wait)
ker.start()
ker.start_evaluator()</pre>
After that we can simply call into Mathematica:
<pre>with MathematicaBlock():
res = MEval (Set (M.hi, &quot;Hello from python!&quot;))
res</pre>
![hookingupmathematicaandpython-4447232891793681130](https://www.wolframcloud.com/objects/b3m2a1/home/img/hookingupmathematicaandpython-4447232891793681130.png)
We can see string ```"Hello from python!"``` was set to the symbol ```hi``` on the Mathematica side and was returned back by ```MEval``` . Symbols that aren't in the ```"System`"``` context need to be prefaced by an ```M.``` as that's a special class that can resolve symbol names like that.
We can also get efficient data transfer of arrays from either side. Here we'll take some Mathematica data and get it back out on the python side. The first thing we need to do is go to the Mathematica notebook that opened and load the ```"PJLink`"``` context. Then we'll install the python runtime that the ```SubprocessKernel``` object configured. This looks like:
<<PJLink`
InstallPython[ LinkObject->SubprocessKernel`$PyEvaluateLink, ProcessObject->None];
Once it's installed, we'll use it directly via ```PyEvaluate``` :
With[{arr= RandomReal[{-1, 1}, {50, 50, 50}]},
PyEvaluate[dat=arr]
]
Calls into python are done in an environment held only by the link, so to access that we need to wrap the evaluator we started ( ```ker.evaluator``` ) in a ```LinkEnvironment``` context manager:
<pre>with LinkEnvironment(ker.evaluator):
res = dat.shape
res</pre>
![hookingupmathematicaandpython-8804306407173974153](https://www.wolframcloud.com/objects/b3m2a1/home/img/hookingupmathematicaandpython-8804306407173974153.png)
Arrays are held as NumPy arrays by default on the python side, although this may be disabled. If disabled, they're held as a data type called ```BufferedNDArray``` which holds the data as a single C-contiguous array and allows slicing and viewing into it (although no efficient math or manipulation of any sort).
Finally, to close out the demo, we'll plot something on the Mathematica side and watch it come back on the python side. The code for this should be pretty self-explanatory by this point, but there is one cute feature to note:
<pre>with MathematicaBlock():
res = MEval(
Rasterize(
Plot(Sin (M.x), List (M.x, 0, Times (2, Pi)),
ImageSize = [250, 250],
PlotLabel = &quot;sin(x) as plotted in Mathematica&quot;
)
)
)
res</pre>
Unfortunately it really does matter that we pass a ```List``` expression instead of a python list for the second argument to ```Plot``` as otherwise the system hangs for reasons that aren't totally clear. On the other hand, we can see how nice options passing is in the interface. We make use of the python ```**kwargs``` setup and that ```ImageSize= ...``` and ```PlotLabel= ...``` both get automatically converted into rules (albeit with a ```String``` key instead of a ```Symbol``` ). The ```Rasterize``` is, sadly, similarly necessary as there is currently no logic in the package to automatically convert ```Graphics``` expression into their rasterized forms.
![hookingupmathematicaandpython-6192533434254394386](https://www.wolframcloud.com/objects/b3m2a1/home/img/hookingupmathematicaandpython-6192533434254394386.png)
I think we'll close out the demo here, though, and move onto a description of how this works.
## PJLink Native Library
The heart of PJLink is the C library that connects a python runtime to MathLink. The source for this can be found [here](https://github.com/b3m2a1/PJLink/blob/master/PJLink/PJLinkNativeLibrary/src/PJLinkNativeLibrary.cpp) . This library, once compiled by the setup.py file packaged with it, implements the basic MathLink calls in a way that python can use them and attempts to do so with efficient memory usage and data transfer.
### Data Sharing in the Native Library
The heart of the native library is the set of ```PutArray``` and ```GetArray``` functions it implements. Beyond anything else, it is the fast transfer of large arrays of data that makes a C-level connection so appealing. The way we handle this on the python side is via the python [buffer protocol](https://docs.python.org/3/c-api/buffer.html) . We enforce the condition that all data sent and received on the python side must be handled by an object that can work with a C-contiguous buffer of data. By default this is done with [NumPy](http://www.numpy.org/) if it is installed, but if not there is a custom object called ```BufferedNDArray``` in the [HelperClasses](https://github.com/b3m2a1/PJLink/blob/master/PJLink/HelperClasses.py) package that deals with this.
### Threading in the Native Library
Python has something called the [Global Interpreter Lock (GIL)](https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock) which is a method for synchronizing python state. Unfortunately for us, the presence of the GIL means that standard C calls of the kind we'll be using will cause all threads to lock. To get around this, every call into the MathLink library in the native library is wrapped in the ```MLTHREADED``` macro which handles the releasing and reacquiring of the lock. This allows our threads to work once more. Any extensions to the library should keep this in mind.
## Class Structures
PJLink provides a glut of classes that handle the details communication, so we will quickly detail what the important ones do. More information is always available upon request.
### The *Link classes
PJLink is based off of JLink and so it makes use of the same kind of class structure that JLink does. This means that it has a ```MathLink``` class that provides a template for the kind of link we'll work with and a ```KernelLink``` class that works specifically with Mathematica kernels. In general, we will only really work with a subclass of a ```KernelLink``` called a ```WrappedKernelLink``` that implements the ```KernelLink``` interface by calling into a ```NativeLink``` which is the only class which actually touches the native library at all.
If one is controlling a Mathematica kernel from python, it will be handled by a ```WrappedKernelLink``` .
### Reader class
The ```Reader``` class handles the other half of the communication. It waits for calls from Mathematica and processes them via the ```KernelLink._handlePacket``` function. Most commonly these calls in turn call ```KernelLink.__callPython``` which builds a python call from the symbolic python packet that ```PyEvaluate``` sends over the link. A ```Reader``` does its best not to completely prevent its link from passing data *to* Mathematica, but in general it is best not to depend on this as the ```NativeLink``` interface allows only a single thread to access the library at once for reasons of safety and stability.
### MathLinkEnvironment and MathLinkException
The ```MathLinkEnvironment``` is a standalone class that handles all of the various flags and state that the links need. It centralizes all information about what a given token or flag from MathLink means and provides utility functions for working with this. ```MathLinkException``` is a subclass of the standard python ```Exception``` class that handles the MathLink-specific exceptions that are returned. It in turn calls into ```MathLinkEnvironment``` to learn what various exceptions mean.
### MPackage, MLSym, and MLExpr
The HelperClasses package provides a large number of (generally) smaller classes that serve to make code cleaner in its implementation. A big part of this is done by the ```MPackage``` , ```MLSym``` , and ```MLExpr``` classes, which allow for a way to create packets with a syntax that looks more like standard Mathematica code. ```MLSym``` and ```MLExpr``` are types that a ```KernelLink``` knows how to put onto a link and ```MPackage``` provides utilities and a custom ```__getattr__``` so that the packet code can look like Mathematica code.
### MathematicaBlock and LinkEnvironment
Both ```MathematicaBlock``` and ```LinkEnvironment``` are also in the HelperClasses. They both edit the current evaluation state as [context managers](https://docs.python.org/3/reference/datamodel.html#context-managers) so that explicit references to ```MPackage``` can be dropped and variables held by a given link can be easily accessed. Being context managers, they are both used via ```with``` statements and change the execution environment of the enclosing block.
### BufferedNDArray, ImageData, and SparseArrayData
These are all data classes that allow for more efficient and convenient data transfer. The ```ImageData``` and ```SparseArrayData``` classes hold data coming in from Mathematica as put using ```PJLink`Package`AddTypeHints``` . They have methods to efficiently transform to more standard formats like ```PIL.Image``` and ```scipy.sparse.csr_matrix``` . As more data types are handled by ```AddTypeHints``` it can be assumed that more classes like these will be written.
## Mathematica-side Package
That was all to do with the python side of things, which is where most of the development work had to go. On the other hand, the Mathematica side of the equation still requires some explanation. The package itself is really quite simple, so please feel free to [peruse the source](https://github.com/b3m2a1/PJLink/blob/master/Mathematica/PJLink.wl) .
### InstallPython
```InstallPython``` is the most basic function in the package. It either finds or is given a python version or executable, attempts to open it via ```StartProcess``` , then links to it via ```LinkCreate``` and the ```start_kernel.py``` script provided in the PJLink python package.
Notably, all it really requires is a ```LinkObject``` , so you can pass one directly via the ```LinkObject``` option. It will also by default try to make a python ```ProcessObject``` but you can pass that via the ```ProcessObject``` option or you can pass ```None``` in which case it won't attach to a Mathematica controlled process.
### ClosePython
```ClosePython``` is the counterpart to ```InstallPython``` . It closes an opened python runtime by version or executable. When a new kernel is installed it is added to ```PJLink`Package`$PythonKernels``` and this is what ```ClosePython``` looks for to close.
### PyEvaluate / PyEvaluateString
This is the heart of the package. It takes Mathematica-esque code, converts it into a structure that can be processed by ```KernelLink.__callPython()``` and sends it over and waits for a response. The conversion is handled by ```PJLink`SymbolicPython`ToSymbolicPython``` which was originally written for the [PyTools package](https://github.com/b3m2a1/mathematica-PyTools) . This is the best way to move data to python as things like ```Image``` objects, packable arrays, and ```SparseArray``` objects will be moved over intelligently.
```PyEvaluateString``` is like ```PyEvaluate``` , but with the recognition that ```ToSymbolicPython``` will always be a little bit lacking. It allows one to simply call a string of python code on the link and get the results back.
### PyWrite / PyWriteString / PyRead / PyReadErr
These are all functions that make use of the fact that when the ```Reader``` object started it allowed an interactive session to keep running and reading / writing on stdin, stdout, and stderr. The ```Read``` functions read from stdout and stderr and the write functions write to stdin. The former takes Mathematica code and auto-converts it into a string. The latter simply passes in the given string.
## Future Work
PJLink 1.0.0, beefy as it already is, should only really be seen as the beginning. My hope is that much more can be done to allow for more native data type transfer and for intelligent communication between the two systems.
In my demo I tried to show some of the things that make the interoperation of the two so nice, but I obviously don't have the breadth of knowledge to know all of the many applications this can be put to. Applications built off of PJLink are always welcome and I'm happy to provide any requisite information and extensions to PJLink to get them built.
Alongside that, I think better integration on the Mathematica side is necessary. There is a partial interface for allowing a ```PythonObject``` structure to hide the details of ```PyEvaluate``` on the Mathematica side, but this needs work from both ends, first hooking up the ```Language`MutatationHandler``` interface and then extending the same on the python side. After that, a ```JavaBlock``` -like setup that allows a link to be opened, used, and cleaned up would be highly useful for sandboxing.
Finally, I'm sure there are numerous bugs hiding in the package as it stands. Please find them and let me know about them so they can be worked out.
In the meantime, I hope you enjoy PJLink and being able to use my two favorite languages symbiotically.b3m2a1 2018-09-20T09:18:51ZLazy lists in Mathematica
http://community.wolfram.com/groups/-/m/t/1467915
Hi all. In this post I want to demonstrate a package I wrote (and still tweak here and there), which implements Haskell-style lazy lists in Mathematica. Anyone who wants to play around with it can find here on Github:
https://github.com/ssmit1986/lazyLists
## What are lazy lists? ##
Before diving into implementation details, let's give some motivation for the use of lazy lists. A lazy list is a method to implicitly represent a long (possibly infinite) list. Of course, you cannot truly have an infinite list in your computer, so the central idea is to format the list as a linked list consisting of 2 parts. This means that a `lazyList` will always look like this:
lazyList[first, tail]
Here, `first` is the first element of the list and `tail` is a held expression that, when evaluated, will give you the rest of the list, which is again a `lazyList` with a first element and a tail. So in other words: elements of a `lazyList` are only generated when they are needed and not before. This makes it possible to represent infinite lists and perform list operations on them. To get the elements of the list, one can simply evaluate the tail as often as needed to progress through the list.
For example, let's define `lazyRange[]` as the lazy list of all positive integers. Then
Map[#^2&, lazyRange[]]
becomes the infinite list of all squares, again represented as a lazy list. You can go even further, though. For example, you can generate the triangular numbers by doing a `FoldList` over the integers and then select the odd ones with a `Select`:
Select[FoldList[Plus, 0, lazyRange[]], OddQ]
which is yet another lazy list. So if we want the first 100 odd triangular numbers, we simply evaluate the tail of this lazy list 99 times to get them. In contrast, if you'd try to do this with a normal list, you could do something like this:
Select[FoldList[Plus, 0, Range[n]], OddQ]
However, what value should you pick for `n`? If you pick it too low, you won't get your 100 numbers. If it's too high, you're doing too much work. Of course you could write some sort of `While` loop, but the code for that would be less concise and doesn't really play into the strengths of Wolfram Language.
## Implementation ##
To illustrate how my code works, I will reproduce some of the code in this blog post, though the package code is different in some respects for efficiency reasons.
The easiest way to prevent the tail from evaluating is to give `lazyList` the `HoldRest` attribute, which is how I implemented them:
Attributes[lazyList] = {HoldRest}
Next, we need some way to construct basic infinite lists like the positive integers. This is generally done recursively. My `lazyRange[]` function takes up to 2 arguments: a starting value (1 by default) and an increment value (also 1 by default):
lazyRange[start : _ : 1, step : _ : 1] := lazyList[start, lazyRange[start + step, step]]
We can extract the first element with `First` and advance through the list with `Last`:
First@lazyRange[]
First@Last@lazyRange[]
First@Last@Last@lazyRange[]
Out[100]= 1
Out[101]= 2
Out[102]= 3
We can also check that the tail of `lazyRange[]` is equal to the list of integers starting from 2:
In[103]:= Last@lazyRange[] === lazyRange[2]
Out[103]= True
Of course, iterating `Last` can be done with `NestList`, so if we want to get the first `n` elements of the lazy list, we can define the following special functionality for `Take` by setting an `UpValue` for `lazyList`:
lazyList /: Take[l_lazyList, n_Integer] := NestList[Last, l, n - 1][[All, 1]]
Take[lazyRange[], 10]
Out[105]= {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
As it turns out, nesting `Last` isn't actually the most efficient way to do this, so I ended up implementing `Take` with `ReplaceRepeated` and `Sow`/`Reap` to make the best use of the pattern matching capabilities of WL.
Next, we want to be able to do transformations on `lazyList`s. The simplest one is `Map`: you simply create a `lazyList` with the function applied to the first element and then `Map` the function over the tail:
lazyList /: Map[f_, lazyList[first_, tail_]] := lazyList[
f[first],
Map[f, tail]
];
Map[#^2 &, lazyRange[]]
Take[%, 10]
Out[117]= lazyList[1, (#1^2 &) /@ lazyRange[1 + 1, 1]]
Out[118]= {1, 4, 9, 16, 25, 36, 49, 64, 81, 100}
Similarly, `Select` is easily implemented by repeatedly evaluating the tail until we find an element that satisfied the selector function `f`. At that point we found our element and return a `lazyList`:
lazyList /: Select[lazyList[first_, tail_], f_] /; f[first] := lazyList[first, Select[tail, f]];
lazyList /: Select[lazyList[first_, tail_], f_] := Select[tail, f];
As an example, we can now find the first 10 numbers that are co prime to 12 and the first 10 squares that are 1 more than a multiple of 3:
Take[Select[lazyRange[], CoprimeQ[#, 12] &], 10]
Take[Select[Map[#^2 &, lazyRange[]], Mod[#, 3] === 1 &], 10]
Out[128]= {1, 5, 7, 11, 13, 17, 19, 23, 25, 29}
Out[129]= {1, 4, 16, 25, 49, 64, 100, 121, 169, 196}
I hope this gives a good enough overview of the benefits of `lazyLists` as well as giving you an idea of how to use them. In the package I tried to implement other list computational functionality (such as `MapIndexed`, `MapThread`, `FoldList`, `Transpose`, `Cases`, and `Pick`) for lazy lists as efficiently as possible.
Please let me know if you have further suggestions!Sjoerd Smit2018-09-19T22:04:27ZHow to enforce priority on custom directory over paclets default repository
http://community.wolfram.com/groups/-/m/t/1471115
[**Topic cross posted in mathematica.stackexchange.com**](https://mathematica.stackexchange.com/q/182287/5478)
As explained in https://mathematica.stackexchange.com/q/59127/5478 loading packages from custom directories may not be so intuitive. Not to mention absolute lack of documentation of interaction between `$Path`, `PacletDirectoryAdd`, paclet's version etc.
Recent update to linked question explains something that hit me hard lately:
## The problem
If you have ``MyPackage` `` (V2) paclet installed in default repo and ``MyPackage` `` (V1) in customDir. Then:
PrependTo[ $Path, customDir];
PacletManager`PacletDirectoryAdd @ customDir;
<< MyPackage`
will load ``MyPackage` `` V2 from default repo.
I find this unexpected and really problematic
## Real world use case
You are working on dev branch on V2, part of testing involves packing it and installing locally, which means there will be V2 in default repo.
Before finishing V2 you need to push a fix to V1 so you create another fix branch from master (V1). You would like to load your code, **surprise** it loads V2 from default repo despite your changes to `$Path` and paclet directories.
## Questions
How to make sure that ``MyPackage` `` will be loaded from `customDir`?
``MyPackage` `` may contain 3rd part packages inside, e.g. `MyPackage/WL/GitLink` which it loads internally e.g. by temporarily blocking `$path` and prepending `MyPackage/WL`. However, `GitLink` may be already installed locally, newer or older, does not matter, the one from `customDir/MyPakcage/WL` should be loaded. Which means the solution can be based on fixing the issue for `MyPackage` only.
p.s. will try to provide a code to play with later.Kuba Podkalicki2018-09-21T07:00:42ZWork with functions defined inside a Package?
http://community.wolfram.com/groups/-/m/t/1150206
I have functions defined inside a Begin Block of a Package in the normal manner (BeginPackage ... foo::usage="bar" ... Begin ... DegreesPerMeterAtmosphere[targelevdeg_,startalt_]: = targelevdeg+startalt ... etc.)
I tried to define variables that would be exposed by the packaged (as below) - but it does not work and I can't find the correct comments ion the manuals. This is the simplified code:
BeginPackage[ "FoundationFunctions`"]
speedlight::usage = "The speed of light in meters/sec";
DegreesPerMeterAtmosphere::usage = "DegreesPerMeterAtmosphere[targelevdeg,startalt] stuff"
Begin[ "Private`"]
speedlight:=299792458 (* m/s *);
DegreesPerMeterAtmosphere[targelevdeg_,startalt_]: = targelevdeg+startalt
End[]
EndPackage[]
But when I Get[] the package from a Notebook the function is pulled across but not the variable. Specifically Names["FoundationFunctions`*"] yields {"DegreesPerMeterAtmosphere"}.
Am I trying to do something that is silly (perhaps one cannot use packages to define/ encapsulate variables)? Or have I done the right thing in the wrong way?Andrew Macafee2017-07-20T13:01:51Z