Has anyone gotten ExternalEvaluate to work with Anaconda Python on OS X?

Questions
, ,
1

Has anyone gotten ExternalEvaluate to work with Anaconda Python on OS X? It used to work for me in the prerelease, but it’s not working in the final release.

In[2]:= py = StartExternalSession["Python"]
Out[2]= ExternalSessionObject["3dac9d41-1400-48be-83ea-9b6012316ff4"]

In[3]:= ExternalEvaluate[py, "1+1"]

During evaluation of In[3]:= Import::nopythonevals: No Python external evaluator found. Use RegisterExternalEvaluator to register an external evaluator.
During evaluation of In[3]:= ExternalEvaluate::interpFail: The result 2 failed to be interpreted as a WL expression. Use "ReturnType"->"String" instead.

Out[3]= $Failed

Screenshot2017-09-1511.01.00.png
Screenshot2017-09-1511.01.00.png1692×670 68.6 KB

Notice that Python did return the result 2. What fails is only the translation of this result to Mathematica.

I would like to know if this is broken for everyone or if my system is misconfigured.

After this problem is solved, I would also like to know if it is possible to use different virtualenvs. Can I have multiple external evaluators registered, each corresponding to a specific virtualenv? Update: It seems that virtualenv creates a copy of the python interpreter, thus passing this to RegisterExternalEvaluator should work. But I can’t try until ExternalEvaluate starts working on my system …

2

I tracked down what is happening here.

The Python expression import functionality seems to rely on a working Python evaluator (see ExternalEvaluate`ImportExport`Symbols`importPythonString).

However, it does not retrieve an evaluator from the list of registered evaluators. Instead, it tries to use various heuristics to find one on the system (see ExternalEvaluate`Private`findLangInstall).

This fails on my system, possibly because I did not install pyzmq for the system’s Python. I do not want to modify the OS’s Python because I use a private installation (Anaconda) instead. This one already has pyzmq installed, as required.

Why won’t Import[..., "PythonExpression"] use the evaluator I registered then? Why does it try to look for an evaluator on its own (and fail)? This seems like a bug.

3

For those, who like me, don’t want to modify their system python (even by installing virtualenv), here’s a hack to get things working:

First, make sure that you have successfully registered an external evaluator for Python. FindExternalEvaluators[] should return it.

Then, we trigger loading the ExternalEvaluate framework (in case it hasn’t been loaded yet):

In[1]:= ExternalEvaluate
Out[1]= ExternalEvaluate

We manually start up a session with a string return type, and assign it to appropriate the internal variable

In[2]:= ExternalEvaluate`ImportExport`Private`$ImporterPythonSession = StartExternalSession["Python" -> "String"]
Out[2]= ExternalSessionObject["9a23363e-5403-442d-8fbd-39f9aff7dc48"]

And now it works!

In[3]:= py = StartExternalSession["Python"]
Out[3]= ExternalSessionObject["5022a77c-eedb-4d39-bbb3-c8b6a644ecdf"]

In[4]:= ExternalEvaluate[py, "1+1"]
Out[4]= 2

Note: I posted this solution on the Mathematica StackExchange, where it is more likely to stay maintained than here (say, if a paclet update or 11.2.1 fixes this problem).

4

I tried to register a Python executable installed in a virtualenv with RegisterExternalEvaluator but it didn’t seem to work:

In[162]:= virtualenvPython="/Users/meng/python_virtualenvs/virtualenv-test/bin/python2.7";
RunProcess[{virtualenvPython,"--version"},All]
Out[162]= <|"ExitCode" -> 0, "StandardOutput" -> "", 
 "StandardError" -> "Python 2.7.13
  "|>
In[161]:= RegisterExternalEvaluator["Python",virtualenvPython]
During evaluation of In[161]:= RegisterExternalEvaluator::invalid: -- Message text not found -- (ExternalEvaluate`Private`reason)
Out[161]= $Failed
5

I didn’t comment on virtualenv further because when I tried it, it worked for me. It is necessary to have pyzmq installed in that virtualenv though. Did you make sure it’s installed?

6

I know this is an old post, but a lot has changed with ExternalEvaluate and its support of Python. In the current releases ExternalEvaluate will create virtual environments on the fly.

For example, this now creates a Python virtual environment that includes pandas:

session = StartExternalSession[<|
   "System" -> "Python",
   "Evaluator" -> <|"Dependencies" -> {"pandas"}|>,
   "SessionProlog" -> "import pandas as pd"|>]

And then this is some sample code that shows how you can use it. The following will return a Dataset:

ExternalEvaluate[session, "
data = {'col1': [1, 2, 3], 'col2': ['a', 'b', 'c']}
pd.DataFrame(data)"]

The attached notebook shows the code with the output included.

pandas-01.nb (79 KB)