Message Boards Message Boards

GROUPS:

Connecting to a remote kernel with a ssh key ?

Posted 2 years ago
5265 Views
|
10 Replies
|
6 Total Likes
|

According to this documentation page, it's possible to connect to a Wolfram kernel from a remote machine via SSH. I installed a kernel on a remote machine but I fail to use it as a kernel. With my first try Mathematica wasn't even asking me any password (this may not be Mathematica's fault as I can't connect from any of my machine without a SSH key anyway).

So I thought I needed a ssh key so I tried setting $SSHAuthentication to access that machine from Mathematica, and I tested it with RemoteConnect and it works surprisingly well, but when I try to actually use the kernel, Mathematica gives me the error :

SSH encountered a networking error while launching kernel 'testRemoteKernel'. Error code = 300.  

and it still doesn't even ask me a password.

So it's not like I can's use SSH (since RemoteConnect does work), I must be missing something else.

POSTED BY: Lucien Grondin
10 Replies
Anonymous User
Anonymous User
Posted 2 years ago

I believe if you google it you'll find some community chats on remote kernels, such as https://community.wolfram.com/groups/-/m/t/94917

You didn't mention if SSH is installed and configured on both machines and if you can use bash(1) to execute the remote kernel math(1) through ssh(1). If you cannot do it by hand, then Mathematica also can not do it either.

Mathematica (now) has two ways to connect to a remote kernel. The 'older' way is TCP/IP port mathlink (which is not login-based and security is "up to you and your OS"). A newly supported way is by "shell log-in access (this may merely launch math(1) with parameters on the remote machine if it is not already running, btw).

"In the Kernel Program field, give the command that would be used to run the Wolfram System through a terminal window on the remote machine". You didn't post what your setup fields were, so I think for anyone to help you, you should post more information - such as the command you used.

POSTED BY: Anonymous User

SSH is configured correctly, as a matter of fact it's the only way I can access the remote machine. I don't have that math link, though. None was created during the installation of the Mathematica kernel. As far as I know, I only have "/usr/local/bin/WolframKernel" and "/usr/bin/wolframscript". Both work and behave similarily but are different binary files (I checked with /usr/bin/diff).

'find /usr/{local,}/bin -name math' returns nothing.

Should I create a symbolic link myself, and if so, to WolframKernel or wolframscript ?

Also, the command in the 'Kernel Properties Windows' of Mathematica is (apart from username and IP) :

"`java`" -jar "`wolframssh`" username@ip-address "/usr/local/bin/WolframKernel" -wstp -LinkMode Connect -LinkProtocol TCPIP -LinkName "`linkname`"

So it seems to use what you call the old method (LinkProtocol TCPIP? ). How do I change it to use the shell log-in ?

POSTED BY: Lucien Grondin

Hello. As several people have already reported (see for example this search results on Mathematica Stackexchange), connecting to a remote kernel is a rather obscure thing. It seems that nowhere in the docs is written clearly how the front-end (FE) actually connects to a non-local kernel. For example, if you inspect a working remote connection you will find that the ports the FE is listening on are random; and if they are not really random, they aren't fixed (see here).

That being said, if the remote kernel is running on a local machine, then I'm able to connect from a FE running Wolfram Desktop 12.1.1 with the following steps. Note that my FE is on macOS (10.15) and the remote machine runs GNU/Linux (Arch Linux, but that shouldn't matter at all).

  1. Make sure you can connect via ssh with a private/public key pair. You should be able to run a simple ssh -i /path/to/your/private_key user@host wolfram and play around with a bare kernel.
  2. Go to "Evaluation > Kernel Configuration Options", select "Advanced Options", and set
    • Arguments to MLOpen: -LinkMode Listen -LinkProtocol TCPIP -LinkOptions MLDontInteract
    • Launch command: $(which ssh) -i /path/to/your/private_key user@host "wolfram" -wstp -LinkMode Connect -LinkProtocol TCPIP -LinkName "`linkname`" -LinkHost `ipaddress`
  3. You should now be able to open a new notebook, select your newly created remote kernel (whichever name you gave it), and evaluate some commands. For example, here's a screenshot of a few successful command I ran on FE connected to a remote machine: enter image description here

A completely different story is launching a remote kernel on a non-local machine (e.g., an Amazon EC2 instance). That is, to my understanding, not straightforward and perhaps not even possible. But I hope someone more expert will contradict me and find a working solution.

I hope this helps.

POSTED BY: Edoardo Baldi

A completely different story is launching a remote kernel on a non-local machine (e.g., an Amazon EC2 instance).

Well, that's more or less what I was trying to do, to be frank. Not Amazon but Google compute. I don't quite understand what's so difficult about it, though. As long as I can connect through ssh it should work fine.

On a side note though : what's this "wolfram" program you're running ? I've installed the wolfram engine for developers and it gave me two executables : wolframscript and WolframEngine. That issue and the "math" program mentioned above are quite confusing.

By the way I've learned today that Windows 10 has a built-in ssh client, now. So I can put the key in %HOME%/.ssh and connect with just ssh username@host. It works fine so I assumed I could replace "java" -jar "wolframssh" with just ssh but that didn't work : either it seems to hang with no error or Mathematica just crashes, it's weird.

POSTED BY: Lucien Grondin

I just realized that the “Launch command” line has been truncated and one cannot scroll. Sorry. This is the full line:

$(which ssh) -i /path/to/ssh-key user@host "wolfram" -wstp -LinkMode Connect -LinkProtocol TCPIP -LinkName "`linkname`" -LinkHost `ipaddress`

what's this "wolfram" program you're running ? I've installed the wolfram engine for developers and it gave me two executables : wolframscript and WolframEngine

You should have also WolframKernel (at least) and the other executables, including math and wolfram. They should be found in the root installation directory, e.g., /usr/local/Wolfram/WolframEngine/12.1/Executables. In fact, wolfram is just a symbolic link placed in /usr/local/bin pointing to that directory. To my understanding, both wolfram and math should start the Wolfram Kernel with the right options, including the MLOpen arguments.

Well, that's more or less what I was trying to do, to be frank. Not Amazon but Google compute. I don't quite understand what's so difficult about it, though. As long as I can connect through ssh it should work fine.

As I said, it's not completely clear to me but one hint comes from which remote command is executed when trying to connect to a non-local (AWS, GCP) machine. It's something like

/path/to/WolframKernel -wstp -LinkMode Connect -LinkProtocol TCPIP -LinkName <random-port-1>@1.2.3.4,<random-port-2>@1.2.3.4

Besides the already mentioned port randomization, there's an evident problem with the IP address the kernel is trying to connect to: that's usually your local IP address and it's never going to be reachable from outside. One may suggest that a couple of port forwarding rules should work but there are at least two issues:

  1. You may not have the possibility to add/modify port forwarding rules. It depends on which network you are connected (e.g., your workplace LAN)
  2. It's not clear at all how one can provide the correct IP address. One could, in principle, set up a “manual” link (see here for example) or even try to set up a VPN server. As far as I know, none of these solutions always work, and they strongly depend on the OS you are using (both front-end and remote) and, of course, Wolfram Mathematica/Desktop version.
POSTED BY: Edoardo Baldi

Besides the already mentioned port randomization, there's an evident problem with the IP address the kernel is trying to connect to: that's usually your local IP address and it's never going to be reachable from outside.

Yes that seems to be an issue indeed. I've tried following instructions on a support page about manually creating a remote Wolfram Language kernel connection, and after evaluating $Version I got a link name with clearly a local address (192.168.x.x).

POSTED BY: Lucien Grondin

I had almost given up, but reading this conversation I got the idea to install and configure a local VPN server and connect the remote machine (an AWS instance in my case) to the VPN. And... it works, because front-end and remote kernel are now part of the same network. I tested also a Manipulate and a trivial NetTrain that are known to require the "dynamic features" between the kernel and the front-end.

The setup is not so complicated: the most laborious part (let's say) is to configure the VPN server and prepare the client profile for the remote machine. For the moment, I also have to start the VPN client on the AWS machine manually, but I'm sure it can be done automatically with a systemd service; I just have to look up how.

In my case I could do it directly on my router (an ASUS with the Merlin Wrt firmware) that can run an OpenVPN server. The kernel configuration is identical, except for the different remote IP address which is now part of the local subnet created by VPN.

POSTED BY: Edoardo Baldi

Congrats. Please consider giving us details on how you did it.

Following your previous posts, I too tried to set up a vpn, using openvpn and instructions found on the debian wiki, but it didn't work on the client side (I'm using Windows Subsystem for Linux there so I suspect that made things less compatible).

I'm supposed to have a vpn server and client on my Internet Provider's router, but it's through a GUI and I don't see any part where I could set up private IPs, so it's kind of confusing.

There is also an integrated VPN client on Windows 10 and I somehow managed to connect to my IP's router server, but I have no idea if Mathematica would use it instead of the normal internet address.

me trying to do network administration

Here are the machines I have access to :

  1. my ISP's router, which offers a VPN client and server, with admnistration through an obscure GUI, though.
  2. my PC running Mathematica 12.1.1, under Windows 10, with WSL running Debian.
  3. remote virtual machine from google compute, running Debian. I installed Mathemica's free engine for developers there and it works fine from the terminal.

I have set ssh to connect from 2 to 3 easily, with a ssl key that I duplicated in order to connect both from cmd.exe and from WSL. Works totally fine.

Any help appreciated.

I guess I should just educate myself on VPNs, though.

POSTED BY: Lucien Grondin

I admit that setting up the local VPN server was easy only thanks to the fully customizable firmware that operates on my router (an Asus RT-AX88U, if you want to know). I can outline the procedure I followed, but I cannot write down a step-by-step solution since it won't be applicable to all the cases.

I can give you a link of a really detailed procedure that includes installing a custom firmware if your router supports it. Briefly, it boils down to three steps:

  1. Setting up the VPN server: installing the OpenVPN packages and creating the certificates and keys for both the server and the client(s)
  2. Putting together an OpenVPN configuration profile for your client. Once your server is configured, it provides you with a generic profile to which you have to add the certificate and key that you created in step 1. You will also have to create at least a pair of user credentials (username/password) for your client: this can be done once again during the server configuration.
  3. Installing the VPN client on your remote machine and connecting to the server with a simple openvpn --config your_profile.ovpn.

Be careful that, if you are connected via SSH to your remote machine, it's very likely that the connection drops because of the way the VPN server works. In brief, this happens because when connecting to the VPN server, you are suddenly changing the default gateway: the traffic that came in from your local machine through SSH is now going back "from a different route", and that breaks the SSH tunnel. Here is explained better, but the quick solution is to add an additional route to your remote machine before starting up the OpenVPN client with

sudo ip route add -host <public IP of VPN server> gw <default gateway of remote>

Another issue related to this is that if your ISP assigns you a dynamic IP (which is very likely: either you pay for a static IP or you are on some kind of Virtual LAN which can happen if you are connected to an optical fiber network). If that's your case, don't despair: the guide of the first link explains how to deal with it with Dynamic DNS. I know that it seems very convoluted, and the truth is that it's indeed a complicated setup and a lot of networking concepts are overlapping. Yet, the guide above is very thorough in explaining all the steps. That guide also assumes that you are doing all your configuration on Windows, so you should be more comfortable and maybe you can avoid using the WSL.

Once all is set on the VPN side, then your remote machine will have a new IP address in the subnet created by the VPN server. That's the address you should use in the "Kernel configuration" profile in Wolfram Mathematica/Desktop. On my machine, the configuration is the same I wrote here a couple of posts above.

As I said, I have to find out the proper way to start the VPN client automatically. That means that there should be a way to store the credentials securely, but at the moment I don't know how. Also bear in mind that I have just run a few simple tests and I want to work with it more extensively to be able to report any issue or other problems that may arise.

If you have questions, I will try to answer at best of my knowledge.

POSTED BY: Edoardo Baldi

I didn't have to install that particular firmware as apparently my router offers a similar service out-of-the-box. I could extract the opvn configuration file and indeed to my amazement running the client on the remote machine was as simple as running : openvpn --config myconfig.opvn

Now if I understand correctly I need to connect my Windows machine, the one Mathematica is installed on. I've tried with the native VPN client but without success.

I suppose I could install openvpn on my windows machine but I find it odd that the native client on Windows 10 can't do the job.

PS. According to this stack exchange page, openvpn is not supported by the native VPN client on Windows 10, so it seems that I do need to install the openvpn client. I'll do it tomorrow, it's getting late here.

PS#2. I couldn't sleep so I actually did it and wow, it actually worked ! Very cool. I Few things :

  • I had to use the bridged openvpn and not the routed one, for some reason I've read the latter doesn't work on Windows 10.
  • In each config file it's necessary to add a line like ifconfig 192.168.X.X 255.255.255.0 to setup a virtual address for each client.
  • I replaced the java wolframssh client call by the native Windows 10 ssh client so that I can use the ssl key more easily.
  • after execution on the remote kernel Mathematica sill shows me a (Running...) message as if the computation was sill on, yet I got my output, so I don't know what's up with that.

Thanks a lot for your help, man. I've learned something today.

POSTED BY: Lucien Grondin
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract