https://community.wolfram.com RSS Feed for Wolfram Community showing ideas tagged with gridMathematica sorted by most likes. https://community.wolfram.com/groups/-/m/t/984003 I'm going to show you how you can use existing functionality to run a Mathematica script across a managed HPC cluster. Before I start, I must be upfront with you: though the individual commands are documented, this method, as a whole, is not. Thus, support for this procedure is outside the scope of Wolfram Technical Support. However, I'm hoping that once the ground-work has been laid, that Wolfram Community members can work together to fill in the missing details. My assumptions: 1. Mathematica is installed and properly licensed on the managed cluster 2. once your job has been given resources, that you can freely SSH between them (1) This is up to your local cluster's System Admin to figure out by talking with their organization and a Wolfram Sales Representative, and possibly Wolfram Technical Support ([support.wolfram.com][1]). (2) Again, this is up to your local SysAdmin to ensure. It's also known as a public/private key pair between nodes. In the following, I'm assuming the cluster uses Torque ([Torque SysAdmin Guide][2]), but in principle other managers can be used. A generic Mathematica script job submission may look like the following: #PBS -N Job_name #PBS -l walltime=10:30 #PBS -l nodes=4:ppn=6 #PBS -m be math -script hpc.wl In this example, - the job is called "Job_name" - the job will finish in 10 and a half minutes - it is requesting 4 nodes with 6 processors-per-node, for a total of 24 resources (CPU cores) - an email will be sent to the account associated with the username when the job (b)egins and when it (e)nds If you are not familiar with job submissions to a managed HPC cluster, then I suggest you read any guides provided by your organization The Wolfram Language script "hpc.wl" does the rest of the work. It generically follows this order: 1. gather the environment variables associated with the list of provided resources 2. launch remote subkernels for each CPU core 3. do the parallel computations 4. close the subkernels 5. end the job (*get association of resources, name of local host, and remove local host from available resources*) hosts = Counts[ReadList[Environment["PBS_NODEFILE"], "String"]]; local = First[StringSplit[Environment["HOSTNAME"],"."]]; hosts[local]--; (*launch subkernels and connect them to the controlling Wolfram Kernel*) Needs["SubKernels`RemoteKernels`"]; Map[If[hosts[#] > 0, LaunchKernels[RemoteMachine[#, hosts[#]]]]&, Keys[hosts]]; (* ===== regular Wolfram Language code goes here ===== *) Print[ {$MachineName, $KernelID} ] (* ===== end of Wolfram Language program ===== *) CloseKernels[]; Quit On Torque there is the environment variable "PBS\_NODEFILE" ([Torque environment variables][3]) that lists the different nodes that are provided to the job. It is my understanding that the name is repeated for each CPU core. That's why a simple Count of the node list tells us everything. The other piece of information, which is probably not necessary, is "HOSTNAME". This is where the Wolfram controlling kernel is running. In the above, we remove it from the list of available resources, but I don't believe this is necessary. According to the documentation ([3]), this may be known as "PBS\_O\_HOSTNAME". The Mathematica script should not need to change save for the code between the commented lines. I'm also assuming that `$RemoteCommand` (provided by Subkernels\`RemoteKernels\`) is the same on each node. This is usually the case as most clusters use a cloned file system. SLURM should be very similar except that the environment variables will be different. It is my understanding that headNode = Environment["SLURMD_NODENAME"]; nodes = ReadList["!scontrol show hostname $SLURM_NODELIST",String]; provides the headnode and list of resources. I encourage discussion. [1]: http://support.wolfram.com [2]: http://docs.adaptivecomputing.com/torque/4-0-2/help.htm [3]: http://docs.adaptivecomputing.com/torque/4-0-2/help.htm#topics/2-jobs/exportedBatchEnvVar.htm Kevin Daily 2016-12-19T21:29:53Z https://community.wolfram.com/groups/-/m/t/613588 Since there are many folks on this forum that are new to using Mathematica/WolframLanguage I'd suggest that each order a copy of Stephen Wolfram's soon-to-be-released (December 7 2015) "An Elementary Introduction to the Wolfram Language": [http://www.amazon.com/Elementary-Introduction-Wolfram-Language/dp/1944183000/][1] though I haven't read it yet, there is a clear need for a tutorial that helps beginning users "start right" and avoid the many issues that arise by simply diving in without a helping hand to get the basics in place. Also another excellent book (but which assumes a certain amount of exposure to scientific, engineering and mathematical subjects but no knowledge of *Mathematica* and very little if any of programming in general) is Paul Wellin's "Programming with Mathematica®: An Introduction", which I have read. [http://www.amazon.com/Programming-Mathematica-Introduction-Paul-Wellin/dp/1107009464/][2] So, beginners, get a couple of good introductions to Mathematica and dive in to learning the language! [1]: http://www.amazon.com/Elementary-Introduction-Wolfram-Language/dp/1944183000/ [2]: http://www.amazon.com/Programming-Mathematica-Introduction-Paul-Wellin/dp/1107009464/ David Reiss 2015-11-14T17:09:12Z https://community.wolfram.com/groups/-/m/t/511430 Announcing [Tunnel][1]. Tunnel is a set of scripts that simplify launching remote Mathematica kernels, where all established MathLink connections are automatically tunneled through a secure shell connection. Tunnel is an improved version of the [Remote Kernel Strategies][2] solution originally presented at the Mathematica User Conference 2008. The new version contains compatibility fixes for newer versions of Mathematica and also supports remote compute kernels. [1]: https://github.com/sakra/Tunnel [2]: http://library.wolfram.com/infocenter/Conferences/7250/ Sascha Kratky 2015-06-09T18:13:37Z https://community.wolfram.com/groups/-/m/t/165103 This post demonstrates some basic techniques to help the gridMathematica users to use their own clusters over different networks. A typical example is a mixture of home LAN and a VPN to your remote location.  [b]1. Installation[/b] Detailed information about Installation is covered [url=http://reference.wolfram.com/mathematica/LightweightGridManager/tutorial/Overview.html]here[/url]. Just make sure that you install the [i]gridMathematica[/i] with a user name of low privilage so you won't be able to modify important locations like /usr/bin or windows/system32 [b]2. Test [/b] Once services are started on each node, you can [url=http://reference.wolfram.com/mathematica/LightweightGridManager/tutorial/WebInterface.html#262527504]check the[i] localhost:3737[/i] of each slave machine[/url]. You should have admin password to activate the licenses. gridMathematica server won't be working properly until license is activated.   [b]3. Work on the distributed system[/b] To have a full control of the net work in terms of the Mathematica process, I recommand the build-in package and utility function.  In this case, the master machine is my macbook pro. There is no problem as I work with two Windows nodes. Every piece of data is transferred via web packet so it is OS independent .  [center][img=width: 385px; height: 69px;]/c/portal/getImageAttachment?filename=master.png&userId=23928[/img][/center]  I load the "LightweightGridClient" package, which comes with Mathematica by default. [mcode]Needs["LightweightGridClient`"][/mcode] The package is fairly rich. You can find a bundle of useful utitlity functions here.  [mcode]Names["LightweightGridClient`*"] // TableForm[/mcode][center][img=width: 266px; height: 259px;]/c/portal/getImageAttachment?filename=utilities.png&userId=23928[/img][/center]If your DNS is correctly setup over the network, you should be able to launch the subkernels directly with the host name of the nodes. If you have some issue, in general, you can edit the hosts file on your master machine to include the ip address and the host name of the same machine. Search online to find the location of this file based on the OS.   It is always a good idea to check the connectivity in case of launching kernels with chunk of error messages. I use the following functions to check the configuration of the distributed system. "shenghui-pc" is a machine in my home network while "shenghuiywin" is the machine in the VPN domain. To make sure the two machines are recognizable by the master node, I added the full domain name for the machine in VPN to indicate that it is a machine in a network different from that for "shenghui-pc".  [mcode]agent1 = RemoteServicesAgentInformation["shenghui-pc"] agent2 = RemoteServicesAgentInformation["shenghuiywin.my.vpn.domain"][/mcode] Simple as the result, the two output are objects containing useful set-up information for each grid server, among which the most important are ContactURL and DefaultKernelCommand. The first indicates the location of the machine online and the latter tells me where the executable on the device.   [img=width: 800px; height: 549px;]/c/portal/getImageAttachment?filename=config.png&userId=23928[/img] You can launch kernel via the following command. The name of the options are straightforward, which are defined with [url=http://reference.wolfram.com/mathematica/LightweightGridClient/tutorial/Introduction.html]LightweightGrid[/url] function [img=width: 686px; height: 271px;]/c/portal/getImageAttachment?filename=config2.png&userId=23928[/img] If you want to also launch the kernels on the master machine, you can simply put a number in the list.  [mcode]kernels = LaunchKernels[{ 4, LightweightGrid[{"Agent" -> "shenghui-pc", "KernelCount" -> 4}], LightweightGrid[{"Agent" -> "shenghuiywin.my.vpn.domain", "KernelCount" -> 4}] }];[/mcode] here 4 simply tells the master kernel to launch 4 subkernels on the local machine.  [b]4. Use MathLink to assign parallel jobs[/b] This is a technique that is useful if you have computational heavy task for subkernels and you still want to work in the same session. Otherwise you may have to wait until a result is returned from a function like ParallelEvaluate. Pick up a subkernel from the list of link objects [mcode]link2Subkeners=Cases[ Links[], LinkObject[x_,_,_]/;StringMatchQ[x,RegularExpression["\\d{4,5}"]~~__](*pick subkernels' link only*) ]//TableForm [/mcode] Do manual [url=http://reference.wolfram.com/mathematica/ref/LinkWrite.html?q=LinkWrite&lang=en]LinkWrite[/url] and [url=http://reference.wolfram.com/mathematica/ref/LinkRead.html?q=LinkRead&lang=en]LinkRead[/url] on LAN:  [mcode]link = LinkObject["53802@192.168.2.2,53803@192.168.2.2", 211, 7]; LinkWrite[link, Unevaluated[Pause[5]; FactorInteger[2^32 - 3]]][/mcode] Comparing to ParallelEvaluate, you can work within the same notebook during the evaluation. You can pick up the result at any time after, in my case ,5 second computation time by  [mcode]LinkRead[link][/mcode] You may verify my observation by yourself : (cell brackets will be highlighted for about 5 seconds)  [mcode]ParallelEvaluate[Pause[5]; FactorInteger[2^32 - 3], 1][/mcode] MathLink is very flexible and you can also work on functions that require FrontEnd:  [mcode]LinkWriteHeld[link, Hold[Image[ {{0, 63, 127, 191, 255}, {0, 50, 100, 150, 200}}, "Byte"]]][/mcode][img=width: 298px; height: 111px;]/c/portal/getImageAttachment?filename=10213res.png&userId=23928[/img] Eventually, do not forget to close kernels at the end of the evaluation:  [mcode]CloseKernels[] // TableForm [/mcode] Note: This type of network is excellent for parallelization with big chunk of computation. The limit is latency and bandwith (LAN is much faster than VPN, while the latter determines the overall time). Therefore, it is not recommended to use ParallelSubmit which cause speed down due to many submission over the network if tasks are rather small. Shenghui Yang 2013-12-04T03:34:51Z https://community.wolfram.com/groups/-/m/t/616290 Recently [Macports][1] is gaining more attention from the open source community on the Mac platform. It has a build for OpenCV which is every intuitive to install in terminal. Here is the package info of this OpenCV port: $ port info opencv opencv @3.0.0_1 (graphics, science) Variants: contrib, dc1394, debug, eigen, java, opencl, openni, python27, python34, qt4, qt5, tbb, universal, vtk Description: OpenCV is a library that is mainly aimed at real time computer vision. Some example areas would be Human-Computer Interaction (HCI), Object Identification, Segmentation and Recognition, Face Recognition, Gesture Recognition, Motion Tracking, Ego Motion, Motion Understanding, Structure From Motion (SFM), and Mobile Robotics. Homepage: http://opencv.org Extract Dependencies: unzip Build Dependencies: cmake, pkgconfig Library Dependencies: zlib, bzip2, libpng, jpeg, jasper, tiff, ilmbase, openexr, ffmpeg Platforms: darwin License: BSD Maintainers: stromnov@macports.org, openmaintainer@macports.org Usually default install location is `/opt/local/`. We use the `image_external` example from [this link][2] and the original source code is from /Applications/Mathematica.app/Contents/SystemFiles/Links/LibraryLink/LibraryResources/Source/image_external.c assuming you have the default mathematica installation on the Mac machine. First we load the `LibraryLink` and `CCompilerDriver` Needs["LibraryLink`"] Needs["CCompilerDriver`"] I put a modified version of the source code in the same directory as the working notebook (`image_extern2.c` contains some type casting of `cvCreateImage -> ImageData` which returns a `char *`) Load the path to the source file: In[3]:= sourcePath=NotebookDirectory[] Out[3]= /Users/shenghuiyang/Desktop/opencv_Mathematica/ To build this file, we have 2 options. First, we can use the `CreateLibrary` function: lib=CreateLibrary[{FileNameJoin[{sourcePath,"image_extern2.c"}]},"image_external", "IncludeDirectories" -> { "/opt/local/include", "/opt/local/include/opencv", "/opt/local/include/opencv2/imgproc", "/opt/local/include/libraw" }, "LibraryDirectories" -> {"/opt/local/lib"}, "Libraries" -> { "opencv_core", "opencv_highgui","opencv_imgproc", "raw"}, "Debug"->False ]; The key to use this function with Macports's OpenCV (and `libra`) is to set the proper directories. Use the following command in the terminal to find them pkg-config --cflags --libs opencv pkg-config --cflags --libs libraw One thing to note is that the actual file of these libraries are `libopencv_core` and so on. You should omit the `lib` before the file name on OS X. Load the dynamic library with `LibraryFunctionLoad` function opencvDilate = LibraryFunctionLoad[lib, "opencv_dilate", {{LibraryDataType[Image], "Constant"}, Integer}, {LibraryDataType[Image]}] The second method is to use a `makefile` to build the dynamic library first and use the `LibaryFunctionLoad` to call the name of the library directly. CC = clang TargetLocation = /Users/shenghuiyang/Library/Mathematica/SystemFiles/LibraryResources/MacOSX-x86-64 $(shell rm $(TargetLocation)/image_external.dylib) SOURCE=image_extern2.c OBJECTS =image_external.dylib CFLAGS =... IFLAGS = ... FRAMEWORKFLAGS =... LIBDIR = -L/opt/local/lib LIBFLAGS = -lopencv_core -lopencv_highgui -lopencv_imgproc LIBFLAGS += -lraw -lstdc++ -lstdc++ Target = image_external.dylib $(TargetLocation)/image_external.dylib:$(SOURCE) $(CC) -dynamiclib -o $(TargetLocation)/image_external.dylib $(CFALGS) $(FRAMEWORKFLAG) $(IFLAGS) $(SOURCE) $(LIBDIR) $(LIBFLAGS) 2>&1 How do I know this file works? Well I called `CreateLibrary` with this option in `ShellCommandFunction `: lib=CreateLibrary[{FileNameJoin[{sourcePath,"image_extern2.c"}]},"image_external", "IncludeDirectories" -> {...}, "ShellCommandFunction"->Print ]; /usr/bin/clang -dynamiclib -o "/Users/shenghuiyang/Library/Mathematica/SystemFiles/LibraryResources/MacOSX-x86-64/Working-shenghuiy-37929-2049441792-2/image_external.dylib" -m64 -fPIC -O2 -mmacosx-version-min=10.6 -framework Foundation -I"/Applications/Mathematica.app/Contents/SystemFiles/IncludeFiles/C" -I"/Applications/Mathematica.app/Contents/SystemFiles/Links/MathLink/DeveloperKit/MacOSX-x86-64/CompilerAdditions" -I"/opt/local/include" -I"/opt/local/include/opencv" -I"/opt/local/include/opencv2/imgproc" -I"/opt/local/include/libraw" "/Users/shenghuiyang/Desktop/opencv_Mathematica/image_extern2.c" -F"/Applications/Mathematica.app/Contents/SystemFiles/Links/MathLink/DeveloperKit/MacOSX-x86-64/CompilerAdditions" -L"/opt/local/lib" -framework "mathlink" -l"opencv_core" -l"opencv_highgui" -l"opencv_imgproc" -l"raw" -lstdc++ 2>&1 You can download the `makefile` in the attachment. After this makefile is executed, the `.dylib` is transferred to a member of `$LibraryPath` automatically. Then we can load this library with opencvDilate=LibraryFunctionLoad["image_external","opencv_dilate",{{LibraryDataType[Image],"Constant"},Integer},{LibraryDataType[Image]}] Finally you can use the opencvDilate function in Mathematica directly as the following screenshot. In a very similar way we can load the `readRawImage` to load the `.arw` file into Mathematica (about 60MB in size) ![enter image description here][3] [1]: https://www.macports.org [2]: https://reference.wolfram.com/language/LibraryLink/tutorial/ImageProcessing.html [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2015-11-18at1.11.32AM.png&userId=23928 Shenghui Yang 2015-11-18T07:16:01Z