https://community.wolfram.com RSS Feed for Wolfram Community showing any discussions tagged with gridMathematica with no replies sorted by active. https://community.wolfram.com/groups/-/m/t/2903058 Mathematica 13.2 is still not processor group aware, i.e. able to utilize multiple NUMA nodes on a single machine. More and more of today's CPU's are exceeding 32 physical cores (64 logical), not to mention dual CPU workstations that are quite common in high performance settings. Windows does not allow more than 64 logical processors in a single processor group, which means Mathematica cannot access the remaining processing power -- also apparent when you check $ProcessorCount. The current workaround is to disable hyperthreading/SMT which only raises the ceiling to 64 physical processors, but it is less than ideal and poor solution for dual CPU systems. Are there plans or methods to make Mathematica processor group aware? Not sure if there's any real technical difficulty in allowing different subkernels to be launched on different processor groups. I know that 3rd party tools such as Process Lasso can work but it would be better for Mathematica to have this ability natively. Sheldon Wu 2023-04-21T06:40:22Z https://community.wolfram.com/groups/-/m/t/930286 Hi everone, I am using Mathematica 10.4 in GUI mode on stampede supercomputer. I am using a single node on this machine and each node has two 8 core chips. Whenever I try to parallelize my code on 16 cores(using only one node) by launching kernels, it does not allow me and continue to compute the solution on a single core. The $ProcessorCount command showed me that there is only one processor. If on each node there are two 8 core chips that it should show me 8 or 16. I cannot figure out a way to connect all cores on one node. If anyone knows about it, please let me know asap. Thanks ashu Ashu Sharma 2016-09-26T20:55:54Z https://community.wolfram.com/groups/-/m/t/616290 Recently [Macports][1] is gaining more attention from the open source community on the Mac platform. It has a build for OpenCV which is every intuitive to install in terminal. Here is the package info of this OpenCV port: $ port info opencv opencv @3.0.0_1 (graphics, science) Variants: contrib, dc1394, debug, eigen, java, opencl, openni, python27, python34, qt4, qt5, tbb, universal, vtk Description: OpenCV is a library that is mainly aimed at real time computer vision. Some example areas would be Human-Computer Interaction (HCI), Object Identification, Segmentation and Recognition, Face Recognition, Gesture Recognition, Motion Tracking, Ego Motion, Motion Understanding, Structure From Motion (SFM), and Mobile Robotics. Homepage: http://opencv.org Extract Dependencies: unzip Build Dependencies: cmake, pkgconfig Library Dependencies: zlib, bzip2, libpng, jpeg, jasper, tiff, ilmbase, openexr, ffmpeg Platforms: darwin License: BSD Maintainers: stromnov@macports.org, openmaintainer@macports.org Usually default install location is `/opt/local/`. We use the `image_external` example from [this link][2] and the original source code is from /Applications/Mathematica.app/Contents/SystemFiles/Links/LibraryLink/LibraryResources/Source/image_external.c assuming you have the default mathematica installation on the Mac machine. First we load the `LibraryLink` and `CCompilerDriver` Needs["LibraryLink`"] Needs["CCompilerDriver`"] I put a modified version of the source code in the same directory as the working notebook (`image_extern2.c` contains some type casting of `cvCreateImage -> ImageData` which returns a `char *`) Load the path to the source file: In[3]:= sourcePath=NotebookDirectory[] Out[3]= /Users/shenghuiyang/Desktop/opencv_Mathematica/ To build this file, we have 2 options. First, we can use the `CreateLibrary` function: lib=CreateLibrary[{FileNameJoin[{sourcePath,"image_extern2.c"}]},"image_external", "IncludeDirectories" -> { "/opt/local/include", "/opt/local/include/opencv", "/opt/local/include/opencv2/imgproc", "/opt/local/include/libraw" }, "LibraryDirectories" -> {"/opt/local/lib"}, "Libraries" -> { "opencv_core", "opencv_highgui","opencv_imgproc", "raw"}, "Debug"->False ]; The key to use this function with Macports's OpenCV (and `libra`) is to set the proper directories. Use the following command in the terminal to find them pkg-config --cflags --libs opencv pkg-config --cflags --libs libraw One thing to note is that the actual file of these libraries are `libopencv_core` and so on. You should omit the `lib` before the file name on OS X. Load the dynamic library with `LibraryFunctionLoad` function opencvDilate = LibraryFunctionLoad[lib, "opencv_dilate", {{LibraryDataType[Image], "Constant"}, Integer}, {LibraryDataType[Image]}] The second method is to use a `makefile` to build the dynamic library first and use the `LibaryFunctionLoad` to call the name of the library directly. CC = clang TargetLocation = /Users/shenghuiyang/Library/Mathematica/SystemFiles/LibraryResources/MacOSX-x86-64 $(shell rm $(TargetLocation)/image_external.dylib) SOURCE=image_extern2.c OBJECTS =image_external.dylib CFLAGS =... IFLAGS = ... FRAMEWORKFLAGS =... LIBDIR = -L/opt/local/lib LIBFLAGS = -lopencv_core -lopencv_highgui -lopencv_imgproc LIBFLAGS += -lraw -lstdc++ -lstdc++ Target = image_external.dylib $(TargetLocation)/image_external.dylib:$(SOURCE) $(CC) -dynamiclib -o $(TargetLocation)/image_external.dylib $(CFALGS) $(FRAMEWORKFLAG) $(IFLAGS) $(SOURCE) $(LIBDIR) $(LIBFLAGS) 2>&1 How do I know this file works? Well I called `CreateLibrary` with this option in `ShellCommandFunction `: lib=CreateLibrary[{FileNameJoin[{sourcePath,"image_extern2.c"}]},"image_external", "IncludeDirectories" -> {...}, "ShellCommandFunction"->Print ]; /usr/bin/clang -dynamiclib -o "/Users/shenghuiyang/Library/Mathematica/SystemFiles/LibraryResources/MacOSX-x86-64/Working-shenghuiy-37929-2049441792-2/image_external.dylib" -m64 -fPIC -O2 -mmacosx-version-min=10.6 -framework Foundation -I"/Applications/Mathematica.app/Contents/SystemFiles/IncludeFiles/C" -I"/Applications/Mathematica.app/Contents/SystemFiles/Links/MathLink/DeveloperKit/MacOSX-x86-64/CompilerAdditions" -I"/opt/local/include" -I"/opt/local/include/opencv" -I"/opt/local/include/opencv2/imgproc" -I"/opt/local/include/libraw" "/Users/shenghuiyang/Desktop/opencv_Mathematica/image_extern2.c" -F"/Applications/Mathematica.app/Contents/SystemFiles/Links/MathLink/DeveloperKit/MacOSX-x86-64/CompilerAdditions" -L"/opt/local/lib" -framework "mathlink" -l"opencv_core" -l"opencv_highgui" -l"opencv_imgproc" -l"raw" -lstdc++ 2>&1 You can download the `makefile` in the attachment. After this makefile is executed, the `.dylib` is transferred to a member of `$LibraryPath` automatically. Then we can load this library with opencvDilate=LibraryFunctionLoad["image_external","opencv_dilate",{{LibraryDataType[Image],"Constant"},Integer},{LibraryDataType[Image]}] Finally you can use the opencvDilate function in Mathematica directly as the following screenshot. In a very similar way we can load the `readRawImage` to load the `.arw` file into Mathematica (about 60MB in size) ![enter image description here][3] [1]: https://www.macports.org [2]: https://reference.wolfram.com/language/LibraryLink/tutorial/ImageProcessing.html [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2015-11-18at1.11.32AM.png&userId=23928 Shenghui Yang 2015-11-18T07:16:01Z https://community.wolfram.com/groups/-/m/t/587691 Hello, I cannot find any reference Ligthweight Grid Manager can be installed on Mathematica 10. In Wolfram Site, there is just installation guide for version 9 not version 10. gridMathematica is just for expanding the computing kernel, but it is hard to setup RemoteKernel manually. So I think Ligthweight Grid Manager is easy-to-use. Please let me know if Lightweight Grid Manager can be installed on Version 10. Thanks, Jiwon jiwon hwang 2015-10-22T02:33:04Z https://community.wolfram.com/groups/-/m/t/515112 I am having issues trying to create a batch file to run math.exe . I am running this as a windows batch file In the batch file I run math -initfile test.m, I only get math kernel command prompt, ready to accept inputs, with no reaction to my .m file, like this In[1]: how do I get it to run the commands? I followed this post and still no joy: http://mathematica.stackexchange.com/questions/47977/running-a-notebook-with-a-batch-file David Sheetz 2015-06-17T18:08:08Z https://community.wolfram.com/groups/-/m/t/372280 ... Martin Kralovec 2014-10-18T19:02:26Z https://community.wolfram.com/groups/-/m/t/165103 This post demonstrates some basic techniques to help the gridMathematica users to use their own clusters over different networks. A typical example is a mixture of home LAN and a VPN to your remote location.  [b]1. Installation[/b] Detailed information about Installation is covered [url=http://reference.wolfram.com/mathematica/LightweightGridManager/tutorial/Overview.html]here[/url]. Just make sure that you install the [i]gridMathematica[/i] with a user name of low privilage so you won't be able to modify important locations like /usr/bin or windows/system32 [b]2. Test [/b] Once services are started on each node, you can [url=http://reference.wolfram.com/mathematica/LightweightGridManager/tutorial/WebInterface.html#262527504]check the[i] localhost:3737[/i] of each slave machine[/url]. You should have admin password to activate the licenses. gridMathematica server won't be working properly until license is activated.   [b]3. Work on the distributed system[/b] To have a full control of the net work in terms of the Mathematica process, I recommand the build-in package and utility function.  In this case, the master machine is my macbook pro. There is no problem as I work with two Windows nodes. Every piece of data is transferred via web packet so it is OS independent .  [center][img=width: 385px; height: 69px;]/c/portal/getImageAttachment?filename=master.png&userId=23928[/img][/center]  I load the "LightweightGridClient" package, which comes with Mathematica by default. [mcode]Needs["LightweightGridClient`"][/mcode] The package is fairly rich. You can find a bundle of useful utitlity functions here.  [mcode]Names["LightweightGridClient`*"] // TableForm[/mcode][center][img=width: 266px; height: 259px;]/c/portal/getImageAttachment?filename=utilities.png&userId=23928[/img][/center]If your DNS is correctly setup over the network, you should be able to launch the subkernels directly with the host name of the nodes. If you have some issue, in general, you can edit the hosts file on your master machine to include the ip address and the host name of the same machine. Search online to find the location of this file based on the OS.   It is always a good idea to check the connectivity in case of launching kernels with chunk of error messages. I use the following functions to check the configuration of the distributed system. "shenghui-pc" is a machine in my home network while "shenghuiywin" is the machine in the VPN domain. To make sure the two machines are recognizable by the master node, I added the full domain name for the machine in VPN to indicate that it is a machine in a network different from that for "shenghui-pc".  [mcode]agent1 = RemoteServicesAgentInformation["shenghui-pc"] agent2 = RemoteServicesAgentInformation["shenghuiywin.my.vpn.domain"][/mcode] Simple as the result, the two output are objects containing useful set-up information for each grid server, among which the most important are ContactURL and DefaultKernelCommand. The first indicates the location of the machine online and the latter tells me where the executable on the device.   [img=width: 800px; height: 549px;]/c/portal/getImageAttachment?filename=config.png&userId=23928[/img] You can launch kernel via the following command. The name of the options are straightforward, which are defined with [url=http://reference.wolfram.com/mathematica/LightweightGridClient/tutorial/Introduction.html]LightweightGrid[/url] function [img=width: 686px; height: 271px;]/c/portal/getImageAttachment?filename=config2.png&userId=23928[/img] If you want to also launch the kernels on the master machine, you can simply put a number in the list.  [mcode]kernels = LaunchKernels[{ 4, LightweightGrid[{"Agent" -> "shenghui-pc", "KernelCount" -> 4}], LightweightGrid[{"Agent" -> "shenghuiywin.my.vpn.domain", "KernelCount" -> 4}] }];[/mcode] here 4 simply tells the master kernel to launch 4 subkernels on the local machine.  [b]4. Use MathLink to assign parallel jobs[/b] This is a technique that is useful if you have computational heavy task for subkernels and you still want to work in the same session. Otherwise you may have to wait until a result is returned from a function like ParallelEvaluate. Pick up a subkernel from the list of link objects [mcode]link2Subkeners=Cases[ Links[], LinkObject[x_,_,_]/;StringMatchQ[x,RegularExpression["\\d{4,5}"]~~__](*pick subkernels' link only*) ]//TableForm [/mcode] Do manual [url=http://reference.wolfram.com/mathematica/ref/LinkWrite.html?q=LinkWrite&lang=en]LinkWrite[/url] and [url=http://reference.wolfram.com/mathematica/ref/LinkRead.html?q=LinkRead&lang=en]LinkRead[/url] on LAN:  [mcode]link = LinkObject["53802@192.168.2.2,53803@192.168.2.2", 211, 7]; LinkWrite[link, Unevaluated[Pause[5]; FactorInteger[2^32 - 3]]][/mcode] Comparing to ParallelEvaluate, you can work within the same notebook during the evaluation. You can pick up the result at any time after, in my case ,5 second computation time by  [mcode]LinkRead[link][/mcode] You may verify my observation by yourself : (cell brackets will be highlighted for about 5 seconds)  [mcode]ParallelEvaluate[Pause[5]; FactorInteger[2^32 - 3], 1][/mcode] MathLink is very flexible and you can also work on functions that require FrontEnd:  [mcode]LinkWriteHeld[link, Hold[Image[ {{0, 63, 127, 191, 255}, {0, 50, 100, 150, 200}}, "Byte"]]][/mcode][img=width: 298px; height: 111px;]/c/portal/getImageAttachment?filename=10213res.png&userId=23928[/img] Eventually, do not forget to close kernels at the end of the evaluation:  [mcode]CloseKernels[] // TableForm [/mcode] Note: This type of network is excellent for parallelization with big chunk of computation. The limit is latency and bandwith (LAN is much faster than VPN, while the latter determines the overall time). Therefore, it is not recommended to use ParallelSubmit which cause speed down due to many submission over the network if tasks are rather small. Shenghui Yang 2013-12-04T03:34:51Z