Hi，

I was granted SSH access privilege from the cluster manager and successfully ran parallel computing through SLURM. The attached file is copied from my test note then.

Note: By the time I was trying to solve this issue, Mathematica working with SLURM follows this approach "Slurm creates a resource allocation for the job and then mpirun launches tasks using some mechanism other than Slurm, such as SSH or RSH", and alternative approaches that do not require SSH access was not supported.

I think supporting alternative approaches working w/ SLURM w/o requiring SSH access would be a very nice feature to be added to Mathematica's capability, just like being already available in MATLAB.

Hopefully, this could be helpful to you.

Attachments:

mathematica_slur...docx

Dear Kevin,

Thanks for your reply.

Yes, there are 40 cores on atulya049 as well.

Ok, I will do one heavy calculation and get back to you.

With thanks, Sachin.

PS- In your First post, You suggested to remove the Wolfram controlling kernel from the list of available resources. So in my program, Out of requested cores (80), 79 are going into the calculation.

Could you tell us why do we need to remove the controlling Wolfram kernel ?

Kindly Help:

I access mathematica (through MobaTerm), which is installed in my user area on Torque managed HPC, to connect multiple nodes assigned to my Job submitted in Queue.

PROBLEM

Following is the program to connect 2 nodes, and each node consists of 40 cores.

Everything looks fine, expect the time taken by ParallelTable at the end of program, which is much higher than time taken by single node (without launching remote kernels)

Note that allotted two nodes atulya095 and atulya049 appear 40 times each in the output.

PROGRAM

nodes=ReadList[Environment["PBS_NODEFILE"], "String"];
Print["alloted node are  ", nodes];    (*get association of resources, name of local host and remove local host from available resources*)
hosts = Counts[nodes];                                             
local = First[StringSplit[Environment["HOSTNAME"],"."]];
Print["local node is ", local];
hosts[local]--;
Needs["SubKernels`RemoteKernels`"];               
Map[If[hosts[#] > 0, LaunchKernels[RemoteMachine[#, "ssh -x -f -l `3` `1` wolfram  -wstp -linkmode Connect `4` -linkname '`2`' -subkernel -noinit", hosts[#]]]]&,    Keys[hosts]];                
Print["kernel count is  ",$KernelCount];           
Print[" machine name is  ", ParallelEvaluate[$MachineName]];
Print[" kernel id is  ",ParallelEvaluate[$KernelID]];
Print["processor count is  ",$ProcessorCount];
Print[AbsoluteTiming[ParallelTable[Exp[Sin[x]]^Sin[x],{x,0.1,200000,0.0001}];][[1]]];   
CloseKernels[];
Quit[];

OUTPUT

alloted node are  {atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049}
local node is atulya095
kernel count is  79
machine name is  {atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya095, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049, atulya049}
kernel id is  {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79}
processor count is  40
189.309181

I will be thankful to your help here.

Your shared link is broken, Could you share it again. It might be helpful to me.

In practice, you only need to log in to the head node via SSH. Having no direct SSH access to the compute nodes from outside the cluster is a good thing!

My understanding is that once a job is running on the cluster, that job's resources (the compute nodes provided to the job) can freely communicate with one another. Considering that MPI parallelization works on the cluster, I would be surprised that you could not also use my described method to have Mathematica run on the SLURM cluster.

You first need to truly test your SSH issues before going any further. I suggest starting an interactive session so you have access to the command line, but request more than one compute node. In the following $> represents the terminal input. Echo the cluster manager commands e.g.

$> !scontrol show hostname $SLURM_JOB_NODELIST
$> echo $SLURM_TASKS_PER_NODE

Then try a simple SSH command from your main node to one of the other compute nodes you've requested. Something like

$> ssh remote-compute-node-name pwd

If password-less SSH is available then this will display the remote compute node's working directory (via the 'pwd' command). If the nodes use a cloned file system then the working directory should be your home directory on the cluster. Resolve any issues before moving on.

- Congratulations! This post is now Staff Pick! Thank you for your wonderful contributions. Please, keep them coming!