Message Boards Message Boards

Has anyone gotten ExternalEvaluate to work with Anaconda Python on OS X?

GROUPS:

Has anyone gotten ExternalEvaluate to work with Anaconda Python on OS X? It used to work for me in the prerelease, but it's not working in the final release.

In[2]:= py = StartExternalSession["Python"]
Out[2]= ExternalSessionObject["3dac9d41-1400-48be-83ea-9b6012316ff4"]

In[3]:= ExternalEvaluate[py, "1+1"]

During evaluation of In[3]:= Import::nopythonevals: No Python external evaluator found. Use RegisterExternalEvaluator to register an external evaluator.
During evaluation of In[3]:= ExternalEvaluate::interpFail: The result 2 failed to be interpreted as a WL expression. Use "ReturnType"->"String" instead.

Out[3]= $Failed

enter image description here

Notice that Python did return the result 2. What fails is only the translation of this result to Mathematica.

I would like to know if this is broken for everyone or if my system is misconfigured.


After this problem is solved, I would also like to know if it is possible to use different virtualenvs. Can I have multiple external evaluators registered, each corresponding to a specific virtualenv? Update: It seems that virtualenv creates a copy of the python interpreter, thus passing this to RegisterExternalEvaluator should work. But I can't try until ExternalEvaluate starts working on my system ...

POSTED BY: Szabolcs Horvát
Answer
3 months ago

I tracked down what is happening here.

The Python expression import functionality seems to rely on a working Python evaluator (see ExternalEvaluate`ImportExport`Symbols`importPythonString).

However, it does not retrieve an evaluator from the list of registered evaluators. Instead, it tries to use various heuristics to find one on the system (see ExternalEvaluate`Private`findLangInstall).

This fails on my system, possibly because I did not install pyzmq for the system's Python. I do not want to modify the OS's Python because I use a private installation (Anaconda) instead. This one already has pyzmq installed, as required.

Why won't Import[..., "PythonExpression"] use the evaluator I registered then? Why does it try to look for an evaluator on its own (and fail)? This seems like a bug.

POSTED BY: Szabolcs Horvát
Answer
3 months ago

For those, who like me, don't want to modify their system python (even by installing virtualenv), here's a hack to get things working:

First, make sure that you have successfully registered an external evaluator for Python. FindExternalEvaluators[] should return it.

Then, we trigger loading the ExternalEvaluate framework (in case it hasn't been loaded yet):

In[1]:= ExternalEvaluate
Out[1]= ExternalEvaluate

We manually start up a session with a string return type, and assign it to appropriate the internal variable

In[2]:= ExternalEvaluate`ImportExport`Private`$ImporterPythonSession = StartExternalSession["Python" -> "String"]
Out[2]= ExternalSessionObject["9a23363e-5403-442d-8fbd-39f9aff7dc48"]

And now it works!

In[3]:= py = StartExternalSession["Python"]
Out[3]= ExternalSessionObject["5022a77c-eedb-4d39-bbb3-c8b6a644ecdf"]

In[4]:= ExternalEvaluate[py, "1+1"]
Out[4]= 2

Note: I posted this solution on the Mathematica StackExchange, where it is more likely to stay maintained than here (say, if a paclet update or 11.2.1 fixes this problem).

POSTED BY: Szabolcs Horvát
Answer
3 months ago

I tried to register a Python executable installed in a virtualenv with RegisterExternalEvaluator but it didn't seem to work:

In[162]:= virtualenvPython="/Users/meng/python_virtualenvs/virtualenv-test/bin/python2.7";
RunProcess[{virtualenvPython,"--version"},All]
Out[162]= <|"ExitCode" -> 0, "StandardOutput" -> "", 
 "StandardError" -> "Python 2.7.13
  "|>
In[161]:= RegisterExternalEvaluator["Python",virtualenvPython]
During evaluation of In[161]:= RegisterExternalEvaluator::invalid: -- Message text not found -- (ExternalEvaluate`Private`reason)
Out[161]= $Failed
POSTED BY: Meng Lu
Answer
3 months ago

I didn't comment on virtualenv further because when I tried it, it worked for me. It is necessary to have pyzmq installed in that virtualenv though. Did you make sure it's installed?

POSTED BY: Szabolcs Horvát
Answer
3 months ago

Group Abstract Group Abstract