In practice, you only need to log in to the head node via SSH. Having no direct SSH access to the compute nodes from outside the cluster is a good thing!

My understanding is that once a job is running on the cluster, that job's resources (the compute nodes provided to the job) can freely communicate with one another. Considering that MPI parallelization works on the cluster, I would be surprised that you could not also use my described method to have Mathematica run on the SLURM cluster.

You first need to truly test your SSH issues before going any further. I suggest starting an interactive session so you have access to the command line, but request more than one compute node. In the following $> represents the terminal input. Echo the cluster manager commands e.g.

$> !scontrol show hostname $SLURM_JOB_NODELIST
$> echo $SLURM_TASKS_PER_NODE

Then try a simple SSH command from your main node to one of the other compute nodes you've requested. Something like

$> ssh remote-compute-node-name pwd

If password-less SSH is available then this will display the remote compute node's working directory (via the 'pwd' command). If the nodes use a cloned file system then the working directory should be your home directory on the cluster. Resolve any issues before moving on.

People who are looking at this might find this independent implementation, made for SGE and a specific HPC cluster, useful: https://bitbucket.org/szhorvat/crc/src

An issue specific to this system was that ssh would not work, and rsh (a specific version of rsh!) had to be used.

I can't comment on how the Mathematica book (do you mean Elementary Introduction to the Wolfram Language?) presents how to use the command line, but yes, this method has been around since Mathematica's inception.

I have seen the method I presented above carried out on a SLURM managed cluster, and I used a Torque/PBS managed cluster in my previous employment. From a user's point-of-view I am familiar with how managed clusters work. However, I have not used all cluster managers, thus why I requested community support. My goal was to have instructions in how to use Mathematica with the most common cluster managers, documented in one place.

As a follow up, one common-use scenario is to

1. log in to the cluster's head node with X-windows forwarding enabled
2. submit an "interactive session" to the cluster
3. wait for the resources to be provided and control of the terminal returned to the client
4. launch a Mathematica front end

Keep in mind, though, that all commands must be sent back-and-forth from the remote cluster to your screen. This can have a pretty annoying delay. The benefit is that most clusters have nodes with more CPU cores than your desktop, and each CPU core may be more powerful than those in your desktop.

The downside (besides the lag) is that if you've requested multiple nodes, then you don't know which node is acting as the head node. And if you use LaunchKernels[], then only the local kernels (on the head node) will launch. Because interactive jobs are intended as debugging sessions, one approach is to just exclusively request a single node.

Alternatively, you can query the environment variables of the job session and launch remote kernels like I showed earlier.