Group Abstract Group Abstract

Message Boards Message Boards

Has anyone gotten ExternalEvaluate to work with Anaconda Python on OS X?

Posted 8 years ago
POSTED BY: Szabolcs Horvát
5 Replies

I know this is an old post, but a lot has changed with ExternalEvaluate and its support of Python. In the current releases ExternalEvaluate will create virtual environments on the fly.

For example, this now creates a Python virtual environment that includes pandas:

session = StartExternalSession[<|
   "System" -> "Python",
   "Evaluator" -> <|"Dependencies" -> {"pandas"}|>,
   "SessionProlog" -> "import pandas as pd"|>]

And then this is some sample code that shows how you can use it. The following will return a Dataset:

ExternalEvaluate[session, "
data = {'col1': [1, 2, 3], 'col2': ['a', 'b', 'c']}
pd.DataFrame(data)"]

The attached notebook shows the code with the output included.

Attachments:
POSTED BY: Arnoud Buzing
Posted 8 years ago

I tried to register a Python executable installed in a virtualenv with RegisterExternalEvaluator but it didn't seem to work:

In[162]:= virtualenvPython="/Users/meng/python_virtualenvs/virtualenv-test/bin/python2.7";
RunProcess[{virtualenvPython,"--version"},All]
Out[162]= <|"ExitCode" -> 0, "StandardOutput" -> "", 
 "StandardError" -> "Python 2.7.13
  "|>
In[161]:= RegisterExternalEvaluator["Python",virtualenvPython]
During evaluation of In[161]:= RegisterExternalEvaluator::invalid: -- Message text not found -- (ExternalEvaluate`Private`reason)
Out[161]= $Failed
POSTED BY: Meng Lu
POSTED BY: Szabolcs Horvát

For those, who like me, don't want to modify their system python (even by installing virtualenv), here's a hack to get things working:

First, make sure that you have successfully registered an external evaluator for Python. FindExternalEvaluators[] should return it.

Then, we trigger loading the ExternalEvaluate framework (in case it hasn't been loaded yet):

In[1]:= ExternalEvaluate
Out[1]= ExternalEvaluate

We manually start up a session with a string return type, and assign it to appropriate the internal variable

In[2]:= ExternalEvaluate`ImportExport`Private`$ImporterPythonSession = StartExternalSession["Python" -> "String"]
Out[2]= ExternalSessionObject["9a23363e-5403-442d-8fbd-39f9aff7dc48"]

And now it works!

In[3]:= py = StartExternalSession["Python"]
Out[3]= ExternalSessionObject["5022a77c-eedb-4d39-bbb3-c8b6a644ecdf"]

In[4]:= ExternalEvaluate[py, "1+1"]
Out[4]= 2

Note: I posted this solution on the Mathematica StackExchange, where it is more likely to stay maintained than here (say, if a paclet update or 11.2.1 fixes this problem).

POSTED BY: Szabolcs Horvát

I tracked down what is happening here.

The Python expression import functionality seems to rely on a working Python evaluator (see ExternalEvaluate`ImportExport`Symbols`importPythonString).

However, it does not retrieve an evaluator from the list of registered evaluators. Instead, it tries to use various heuristics to find one on the system (see ExternalEvaluate`Private`findLangInstall).

This fails on my system, possibly because I did not install pyzmq for the system's Python. I do not want to modify the OS's Python because I use a private installation (Anaconda) instead. This one already has pyzmq installed, as required.

Why won't Import[..., "PythonExpression"] use the evaluator I registered then? Why does it try to look for an evaluator on its own (and fail)? This seems like a bug.

POSTED BY: Szabolcs Horvát
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard