Message Boards Message Boards

Running Wolfram Mathematica inside Jupiter Notebook in Google Cloud + GPUs

Posted 5 years ago

I am a Data Scientist working in a startup in Brazil. Recently I was working with Social Networks in Python using networkx library to analyse posts from a given hashtag for our client. However, I was completely aware of the additional functionalities of Wolfram Mathematica. Take the following picture as an example, done with networkx, 100,000 people:

Social Network Emotions Connections

It's an interesting perspective, but adds little value to generate strategic insights for our clients, given that we need to zoom it to analyze details of this network. In my previous experience of Wolfram Mathematica, I was able to use some reasoning coming from cellular automata interactions to map, track people of interest (blue circle), and watch the evolution of mood (color), number of connections (size of circle) in a social network:

Evolution of Mood

Besides, one can use Mathematica's features of finding communities, coloring them and even highlight people of interest when passing the mouse over the social network and also speaking their names:

Communities

More than that, you can highlight communities and check the number of connections of each individual:

Communities Two

So, it's clear that Wolfram Mathematica is way ahead of networkx. However, I was dealing with a drawback: if you choose to plot more than 10,000 connections in Mathematica, that can take some time. And time is what you don't have in a startup. So, I started wondering how I could use GPUs with Wolfram Mathematica. One way it's to use webMathematica, installing Java and Apache Tomcat in a cloud instance. However, I was told one can also use Wolfram Client Python library and run Mathematica in a Python notebook.

So, I will present how I was able to run Mathematica inside a Jupyter notebook located in a Google Cloud instance with 8 V100 GPUs. My co-worker, Gustavo Gouvea, also put efforts in this solution.

First of all, go to Google Cloud Platform (GCP) Compute Engine and select Create Instance in a given region. Choose your CPUs, Memory, GPUs and regarding the boot disk, I used Debian GNU/Linux with Anaconda, PyTorch and CUDA already installed, as we also work with Deep Learning and NLP.

After that, you need to configure Jupyter in order to be able to open it in the local browser, by doing the following:

ipython
from IPython.lib import passwd
passwd()

Now that you created the password, you are going to save "sha1:49b8799c22..." Then you will edit Jupyter configuration file (use sudo or chmod -R 777 /home/anaconda3):

sudo vi ~/.jupyter/jupyter_notebook_config.py

Add (type "i") these line of code:

c=get_config()
c.NotebookApp.password =  paste your sha1 here
c.IPKernelApp.pylab = 'inline'
c.NotebookApp.ip =  "*" or 0.0.0.0
c.NotebookApp.open_browser = False

Go to GCP VPC and create a static IP for your instance, SSH into it.

So, to start Jupyter notebook you will need to run:

jupyter notebook

Jupyter

Now you will access Jupyter in the following address:

http://http://55.100.30.200:8888/

Jupyter 2

That will allow you to run Python notebooks in Jupyter. Now Mathematica. Get the mathematica .sh file:

sudo wget link_to_mathematica.sh

To install Mathematica run in another SSH session (terminal window):

sudo bash  link_to_mathematica.sh

Ok, you installed Mathematica. Now run:

math

This command will ask for your Activation key and Password. After providing them, Mathematica notebook starts in the command line

Mathematica running

Important: You must run Mathematica ("math") before you open Jupyter Notebook.

Now that you've opened Mathematica in the command line with one SSH and after that Jupyter Notebook in other SSH, access the web address, port 8888, create a new Python 3 notebook and install wolframclient library:

pip install

Now you can run your Mathematica scripts with backup of how many GPUs you need:

NVIDIA-SMI

from wolframclient.language import wl as w1
from wolframclient.language import wlexpr
from wolframclient.evaluation import WolframLanguageSession
session = WolframLanguageSession('/usr/local/Wolfram/Mathematica/12.0/Executables/MathKernel')
import numpy as np 
import imageio
import matplotlib.pyplot as plt
from PIL import Image

Output

In bird's eye:

Overview

enter image description here

The only point is that you won't have access to all Mathematica functionalities, as Manipulate, Mouse Over and the display of some images when output is truncated. My guess is that init.m config file must be adjusted to increase output size before truncation (in the command line) :

GraphPlot

POSTED BY: Rubens Zimbres
3 Replies

enter image description here -- you have earned Featured Contributor Badge enter image description here

Your exceptional post has been selected for our editorial column Staff Picks http://wolfr.am/StaffPicks and Your Profile is now distinguished by a Featured Contributor Badge and is displayed on the Featured Contributor Board. Thank you, keep it coming, and consider contributing to the The Notebook Archive!

POSTED BY: EDITORIAL BOARD
Posted 5 years ago

Thanks guys, always a pleasure to contribute to Wolfram Mathematica advancements. Now I will use Mathematica power in production, a whole new level.

All the best,

Rubens

POSTED BY: Rubens Almeida

Hi, Rubens, I found very useful your post. Thanks for sharing. I wonder if you have any idea to do the same you did here, but in Colab, the open platform of GCP. I'd appreciate any comment . Best, alberto

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract