Group Abstract Group Abstract

Message Boards Message Boards

Setting Up Lightweight Grid On Multiple Clients Programmatically

GROUPS:
This is essentially a cross-post of this Mma.StackExchange post.

I have a number of computers which all might need to use the same (client) configuration for the Wolfram Lightweight Grid at some point.

I know that there are ways to programmatically set certain front-end options with commands akin to
SetOptions[$FrontEnd, RenderingOptions -> {"HardwareAntialiasingQuality" -> 1}]
My question is: are there similar commands which, when I evaluate them on a given computer will set preferences in the Local->Lightweight Grid preference pane? That is, I would love to have a notebook that I could distribute to my colleagues that contained code like
SetOptions[Something, LookForKernelsAt->IPAddress(es)OfTheComputer(s)RunningLightweightGrid]
that would configure their computers correctly and through Mathematica restarts (ie. so that they only have to evaluate the code one time).

Is this possible? Do such commands exist? I have been trying to understand what is provided in the LightweightGridClient and Parallel`Developer contexts but nothing seems to fit the bill.

Alternatively, are there commands that would correctly and permanently setup access to additional cores via SSH, as in Parallel->Remote Kernels ?


**Refinement of question below**

Thank you both for your responses, I think they have helped me learn enough to almost solve, but then need to refine my question.


Suppose I go to the Mma preferences and click "Reset to Defaults" in the Parallel tab.

Then, when I evaluate
$ConfiguredKernels
I get, for example
{<<2 local kernels>>}
and if I LaunchKernels[] two kernels launch, as expected, since that is the default on my machine alone.

If I CloseKernels[] I can then execute
Needs["LightweightGridClient`"]
AppendTo[$ConfiguredKernels,
   LightweightGrid[{
    "Agent" -> "http://THE.IP.ADD.RESS:3737/WolframLightweightGrid/Manager",
    "KernelCount" -> SomeNumber,
    "LocalLinkMode" -> "Connect",
    "Service" -> "",
    "Timeout" -> 30}] ]
If I go to the Parallel preference pane, it shows "Total number of configured kernels: 2+SomeNumber", which is promising. Something curious is that if I go to the Lightweight Grid sub-pane, it doesn't show the machine I just added to $ConfiguredKernels at all (even though it does show that 2+SomeNumber are set up). It simply shows what it shows after clicking Reset to Defaults: "Ready to connect to computers running the Lightweight grid Manager". I think this might just be that the interface is not refreshing.

If I run LaunchKernels[] now, 2+SomeNumber kernels launch, as expected. If I close and re-open the preferences and go to the Lightweight Grid subpane, now it shows a computer at THE.IP.ADD.RESS as expected, with the "Enable" box checked, but unexpectedly the Kernels box is 0, not SomeNumber as expected.

Finally, if I CloseKernels[], $ConfiguredKernels evaluates to {<<2 local kernels>>} again, instead of a list of the two local kernels + the lightweight grid kernels. If I return to the Lightweight Grid preference pane, all knowledge of kernels at THE.IP.ADD.RESS is gone.

If I go through the same rigamaroll but now manually increase the kernel count next to THE.IP.ADD.RESS to any number by clicking the + sign in the preference pane, I can CloseKernels[], Quit[], quit Mma, etc., and knowledge of that lightweight grid survives.

My questions are:
(1) Am I having a typical experience? If it matters, my $Version is "9.0 for Mac OS X x86 (64-bit) (November 20, 2012)" (9.0.0.0, I think).
(2) What can I do to get the Lightweight Grid stuff to stick around through CloseKernels[]s, Quit[]s, and full quits of Mma that isn't clicking around with my mouse in the preference pane?
POSTED BY: Evan Berkowitz
Answer
1 year ago
Regarding LightweightGrid, yes, there is a way.  However, it is not with SetOptions[].

The LightweightGrid section of the Preferences dialog is ultimately making calls to functions in the 
LightweightGridClient package:
Needs["LightweightGridClient`"]

Loading the package will give you access to functions such as RemoteServicesAgents[] and 
RemoteKernelServices[].  Notice that the outputs from these functions directly match information 
displayed in the LightweightGrid section of the Preferences.

While not very prominant, there is a ">>" link in the LightweightGrid section of the Preferences 
dialog which takes you to the documentation that decribes how to launch such kernels manually
(that is, by calling functions instead of using the dialog).

Paste the following into the Documentation Center search field to go directly to the guide pages:
  LightweightGridClient/guide/LightweightGridClient

If you are not able to find what you need in the documentation pages, please comment in this 
thread and we can make another attempt to find what you need.
POSTED BY: Kurt Gimbel
Answer
1 year ago
Regarding the Remote Kernels alternative, once you have a working setup in the Preferences dialog, you should 
have everything that you need in order to call functions in a similar way to what I described for LightweightGrid.

That is also covered on the same documentations page:
  ParallelTools/tutorial/ConnectionMethods

in the section that contains:
Needs["SubKernels`RemoteKernels`"]

I hope you find a solution using LightweightGrid, though.  It is meant to be much easier to setup than any similar
solution that uses RemoteKernels.
POSTED BY: Kurt Gimbel
Answer
1 year ago
Typically, you need to set up proper host names in the etc/hosts file on the machines of your collegues. If they have the following setup in the hosts file: (ip's are just examples)
192.168.2.10 gridMathematicaMachine1
192.168.2.11 gridMathematicaMachine2
Then they should be able to launch kernels with
Needs["LightweightGridClient`"]
LaunchKernels[{
LightweightGrid["Agent"->"gridMathematicaMachine1","KernelCount"->4],
LightweightGrid["Agent"->"gridMathematicaMachine2","KernelCount"->4]
}]
This command will launch 4 kernels on each machine. Just FYI,  Two gyms rules should be noticed:
  1. "Re-rack after your excercise": Always include CloseKernels[] function in the notebook that you are send out and let your buddies to evalute this line
  2. "Check before kick":  You may include the url like
http://gridMathematicaMachine1:3737/
in your notebook so that your colleage can check if kernels are already running. If they see the kernels are running, they should not launch again but wait for the previous job to quit. 
POSTED BY: Shenghui Yang
Answer
1 year ago
Thank you both for your responses, I think they have helped me learn enough to almost solve, but then need to refine my question.

Suppose I go to the Mma preferences and click "Reset to Defaults" in the Parallel tab.

Then, when I evaluate
$ConfiguredKernels
I get, for example
{<<2 local kernels>>}
and if I LaunchKernels[] two kernels launch, as expected, since that is the default on my machine alone.

If I CloseKernels[] I can then execute
 Needs["LightweightGridClient`"]
 AppendTo[$ConfiguredKernels,
  LightweightGrid[{"Agent" ->
     "http://THE.IP.ADD.RESS:3737/WolframLightweightGrid/Manager",
    "KernelCount" -> SomeNumber,
    "LocalLinkMode" -> "Connect",
    "Service" -> "",
    "Timeout" -> 30}]
  ]

If I go to the Parallel preference pane, it shows "Total number of configured kernels: 2+SomeNumber", which is promising.  Something curious is that if I go to the Lightweight Grid sub-pane, it doesn't show the machine I just added to $ConfiguredKernels at all (even though it does show that 2+SomeNumber are set up).  It simply shows what it shows after clicking Reset to Defaults: "Ready to connect to computers running the Lightweight grid Manager".  I think this might just be that the interface is not refreshing.

If I run LaunchKernels[] now, 2+SomeNumber kernels launch, as expected.  If I close and re-open the preferences and go to the Lightweight Grid subpane, now it shows a computer at THE.IP.ADD.RESS as expected, with the "Enable" box checked, but unexpectedly the Kernels box is 0, not SomeNumber as expected.

Finally, if I CloseKernels[], $ConfiguredKernels evaluates to {<<2 local kernels>>} again, instead of a list of the two local kernels + the lightweight grid kernels.  If I return to the Lightweight Grid preference pane, all knowledge of kernels at THE.IP.ADD.RESS is gone.

If I go through the same rigamaroll but now manually increase the kernel count next to THE.IP.ADD.RESS to any number by clicking the + sign in the preference pane, I can CloseKernels[], Quit[], quit Mma, etc., and knowledge of that lightweight grid survives.

My questions are:
(1) Am I having a typical experience?  If it matters, my $Version is "9.0 for Mac OS X x86 (64-bit) (November 20, 2012)" (9.0.0.0, I think).
(2) What can I do to get the Lightweight Grid stuff to stick around through CloseKernels[]s, Quit[]s, and full quits of Mma that isn't clicking around with my mouse in the preference pane?
POSTED BY: Evan Berkowitz
Answer
1 year ago
After some research, I still don't think I have the answers that you are looking for here,  but I will say what I have figured out after talking with someone who knows more about this area than I do. The preference settings end up being saved in the file Preferences.m in the directory:  
$UserBaseDirectory/ApplicationData/Parallel/Preferences
I have not successfully modified the Preferences.m file, however the commands that do modify it seem to be the ones listed by:
Names["Parallel`Preferences`*"]
This is as deep as I am planning on going.  From here I can only wish you luck.
POSTED BY: Kurt Gimbel
Answer
1 year ago