https://community.wolfram.com RSS Feed for Wolfram Community showing any discussions tagged with High-Performance Computing sorted by most likes. https://community.wolfram.com/groups/-/m/t/1085633 The neural network and machine learning framework has become one of the key features of the latest releases of the Wolfram Language. Training neural networks can be very time consuming on a standard CPU. Luckily the Wolfram Language offers an incredible easy way to use a GPU to train networks - and do lots of other cool stuff. The problem with this was/is that most current Macs do not have an NVIDIA graphics card, which is necessary to access this framework within the Wolfram Language. Therefore, Wolfram Inc. had decided to drop support for GPUs on Macs. There is however a way to use GPUs on Macs. For example you can use an [external GPU like the one offered by Bizon][1]. ![enter image description here][2] Apart from the BizonBox there a couple of cables and a power supply. You can buy/configure different versions of the BizonBox: there is a range of different graphics cards available and you can buy a the BizonBox 2s which basically connects via Thunderbolt and the BizonBox 3 which connects to USB-C. Luckily, Wolfram have decided to reintroduce support for GPUs in Mathematica 11.1.1 - see [the discussion here][3]. I have a variety of these BizonBoxes (both 2s and 3) and a range of Macs. I thought it would be a good idea to post a how-to. The essence of what I will be describing in this post should work for most Macs. I ran Sierra on all of them. Here is the recipe to get the thing to work: Installation of the BizonBox, the required drivers, and compilers ----------------------------------------------------------------- 0. I will assume that you have Sierra installed and that Xcode is running. One of the really important steps if you want to use compilers is to ***downgrade*** the command line tools to version 7.3 You will have to log into your Apple Developer account and download the Command Line Tools version 7.3. Install the tools and run the terminal command (not in Mathematica!): sudo xcode-select --switch /Library/Developer/CommandLineTools 1. Reboot your Mac into safe mode, i.e. hold CMD+R while rebooting. 2. Open a terminal (under item Utilities at the top of the screen). 3. Enter csrutil disable 4. Shut the computer down. 5. Connect your BizonBox to the mains and to either the thunderbolt or USB-C port of your Mac. 6. Restart your Mac. 7. Click on the Apple symbol in the top left. Then "About this Mac" and "System Report". In the Thunderbolt section you should see something like this: ![enter image description here][4] 8. In the documentation of the BizonBox you will find a link to a program called bizonboxmac.zip. Download that file and unzip it. 9. Open the folder and click on "bizonbox.prefPane" to install. (If prompted to, do update!) 10. You should see this window: ![enter image description here][6] 11. Click on Activate. Type in password if required to do so. It should give something like this: ![enter image description here][7] Then restart. 12. Install the CUDA Toolkit: [https://developer.nvidia.com/cuda-downloads][8]. You'll have to click through some questions for the download. ![enter image description here][9] what you download should be something like cuda_8.0.61_mac.dmg and it should be more or less 1.44 GB worth. 13. Install the toolkit with all its elements. ![enter image description here][10] 14. Restart your computer. First tests ----------- Now you should be good to go. Open Mathematica 11.1.1. Execute Needs["CUDALink`"] Needs["CCompilerDriver`"] CUDAResourcesInstall[] Then try: CUDAResourcesInformation[] which should look somewhat like this: ![enter image description here][11] Then you should check SystemInformation[] Head to Links and then CUDA.This should look similar to this: ![enter image description here][12] So far so good. Next is the really crucial thing: CUDAQ[] should give TRUE. If that's what you see you are good to go. Be more daring and try CUDAImageConvolve[ExampleData[{"TestImage","Lena"}], N[BoxMatrix[1]/9]] // AbsoluteTiming ![enter image description here][13] You might notice that the non-GPU version of this command runs faster: ImageConvolve[ExampleData[{"TestImage","Lena"}], N[BoxMatrix[1]/9]] // AbsoluteTiming runs in something like 0.0824 seconds, but that's ok. Benchmarking (training neural networks) --------------------------------------- Let's do some Benchmarking. Download some example data: obj = ResourceObject["CIFAR-10"]; trainingData = ResourceData[obj, "TrainingData"]; RandomSample[trainingData, 5] You can check whether it worked: RandomSample[trainingData, 5] should give something like this: ![enter image description here][14] These are the classes of the 50000 images: classes = Union@Values[trainingData] ![enter image description here][15] Let's build a network module = NetChain[{ConvolutionLayer[100, {3, 3}], BatchNormalizationLayer[], ElementwiseLayer[Ramp], PoolingLayer[{3, 3}, "PaddingSize" -> 1]}] net = NetChain[{module, module, module, module, FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]}, "Input" -> NetEncoder[{"Image", {32, 32}}], "Output" -> NetDecoder[{"Class", classes}]] When you train the network: {time, trained} = AbsoluteTiming@NetTrain[net, trainingData, Automatic, "TargetDevice" -> "GPU"]; you should see something like this: ![enter image description here][16] So the thing started 45 secs ago and it supposed to finish in 2m54s. In fact, it finished after 3m30s. If we run the same on the CPU we get: ![enter image description here][17] The estimate kept changing a bit, but it settled down at about 18h20m.That is slower by a factor of about 315, which is quite substantial. Use of compiler --------------- Up to now we have not needed the actual compiler. Let's try this, too. Let's grow a Mandelbulb: width = 4*640; height = 4*480; iconfig = {width, height, 1, 0, 1, 6}; config = {0.001, 0.0, 0.0, 0.0, 8.0, 15.0, 10.0, 5.0}; camera = {{2.0, 2.0, 2.0}, {0.0, 0.0, 0.0}}; AppendTo[camera, Normalize[camera[[2]] - camera[[1]]]]; AppendTo[camera, 0.75*Normalize[Cross[camera[[3]], {0.0, 1.0, 0.0}]]]; AppendTo[camera, 0.75*Normalize[Cross[camera[[4]], camera[[3]]]]]; config = Join[{config, Flatten[camera]}]; pixelsMem = CUDAMemoryAllocate["Float", {height, width, 3}] srcf = FileNameJoin[{$CUDALinkPath, "SupportFiles", "mandelbulb.cu"}] Now this should work: mandelbulb = CUDAFunctionLoad[File[srcf], "MandelbulbGPU", {{"Float", _, "Output"}, {"Float", _, "Input"}, {"Integer32", _, "Input"}, "Integer32", "Float", "Float"}, {16}, "UnmangleCode" -> False, "CompileOptions" -> "--Wno-deprecated-gpu-targets ", "ShellOutputFunction" -> Print] Under certain circumstances you might want to specify the location of the compiler like so: mandelbulb = CUDAFunctionLoad[File[srcf], "MandelbulbGPU", {{"Float", _, "Output"}, {"Float", _, "Input"}, {"Integer32", _, "Input"}, "Integer32", "Float", "Float"}, {16}, "UnmangleCode" -> False, "CompileOptions" -> "--Wno-deprecated-gpu-targets ", "ShellOutputFunction" -> Print, "CompilerInstallation" -> "/Developer/NVIDIA/CUDA-8.0/bin/"] This should give: ![enter image description here][18] Now mandelbulb[pixelsMem, Flatten[config], iconfig, 0, 0.0, 0.0, {width*height*3}]; pixels = CUDAMemoryGet[pixelsMem]; Image[pixels] gives ![enter image description here][19] So it appears that all is working fine. Problems -------- I did come up with some problems though. There is quite a number of CUDA functions: Names["CUDALink`*"] ![enter image description here][20] Many work just fine. res = RandomReal[1, 5000]; ListLinePlot[res] ![enter image description here][21] ListLinePlot[First@CUDAImageConvolve[{res}, {GaussianMatrix[{{10}, 10}]}]] ![enter image description here][22] The thing is that some don't and I am not sure why (I have a hypothesis though). Here are some functions that do **not** appear to work: CUDAColorNegate CUDAClamp CUDAFold CUDAVolumetricRender CUDAFluidDynamics and some more. I would be very grateful if someone could check these on OSX (and perhaps Windows?). I am not sure if the this is due to some particularity of my systems or something that could be flagged up to Wolfram Inc for checking. When I wanted to try that systematically I wanted to use the function WolframLanguageData to look for the first example in the documentation of the CUDA functions, but it appears that no CUDA function is in the WolframLanguageData. I think tit would be great to have them there, too, and am not sure why they wouldn't be there. In spite of these problems I hope that this post will help some Mac users to get CUDA going. It is a great framework and simple to use in the Wolfram Language. With the BizonBox and Mathematica 11.1.1 Mac users are no longer excluded from accessing this feature. Cheers, Marco PS: Note, that there is anecdotal evidence that one can even use the BizonBox under Windows running in a virtual box under OSX. I don't have Windows, but I'd like to hear if anyone get this running. [1]: https://bizon-tech.com [2]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.09.10.png&userId=48754 [3]: http://community.wolfram.com/groups/-/m/t/902394 [4]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot1.png&userId=48754 [5]: http://bizon-tech.com/bizonboxmac.zip [6]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2.png&userId=48754 [7]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot3.png&userId=48754 [8]: https://developer.nvidia.com/cuda-downloads [9]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot6.png&userId=48754 [10]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot7.png&userId=48754 [11]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.38.22.png&userId=48754 [12]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-06at18.46.46.png&userId=48754 [13]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.49.15.png&userId=48754 [14]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.52.43.png&userId=48754 [15]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at22.53.30.png&userId=48754 [16]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at20.37.20.png&userId=48754 [17]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at20.39.02.png&userId=48754 [18]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at23.04.38.png&userId=48754 [19]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at21.50.42.png&userId=48754 [20]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at23.10.05.png&userId=48754 [21]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at23.15.59.png&userId=48754 [22]: http://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2017-05-07at23.16.36.png&userId=48754 Marco Thiel 2017-05-07T22:21:42Z https://community.wolfram.com/groups/-/m/t/2542490 ![enter image description here][1] The excellent [community contribution][2] by [@Greg Hurst][at0] and a Wikipedia [animation][3], Inspired me to look further into this "rolling shutter effect on rotating objects". When we capture a video of a rotating disk with a rolling shutter, we have two independent movements: the colored disk rotating at *rps revolutions per second* and the shutter line sweeping one frame at *fps frames per second*. The ratio rps/fps is the driver of the rolling shutter effect (or the disk rps alone if we normalize the shutter fps to 1 frame per second). In order to best demonstrate this effect, ratios of rps/fps are taken to be in the range 1.5-2.5 This is a colored disk of m pixels wide, rotated over an angle theta. colors = {RGBColor[1, 0, 1], RGBColor[0.988, 0.73, 0.0195], RGBColor[ 0.266, 0.516, 0.9576], RGBColor[0.207, 0.652, 0.324], RGBColor[ 0, 0, 1], RGBColor[1, 0, 0]}; colorDisk[theta_, m_, cols_] := ImageResize[ Image[Graphics[ MapThread[{#3, Disk[{0, 0}, 1, {#1, #2} + theta]} &, {Pi Range[0, 5, 1]/3, Pi Range[1, 6]/3, colors}]]], m] A video of the rotating disk consists of a series of frames. Each frame is captured during one passage of the shutter line. The function *angularPosition* links the angular progress of the disk to the frame number (frm) and the row number at the position of the shutter line: angularPosition[frm_, row_, rps_, m_] := -2 Pi (-1 + (-1 + frm) m + row) rps/m Th function *diskFrameImage* computes the result of the shutter line swiping a colored disk (of size m, rotating at rps revolutions per second) at frame number frm and up to row number toRow: diskFrameImage[frm_, toRow_, rps_, m_, cols_] := ImageAssemble[ Transpose@{ParallelTable[ ImageTake[ colorDisk[angularPosition[frm, r, rps, m], m, colors], {r}], {r, toRow}]}] This is the first frame of a video of a disk rotating at a speed ratio rps/fps of 1 : With[{m = 200, frm = 1, rps = 1}, diskFrameImage[frm, m, rps, m, colors]] ![enter image description here][4] This shows the influence of the disk rps/fps ratio on the appearance of the first frame captured: With[{m = 200, frm = 1}, Grid[{{"rps=0.512", "rps=1.512", "rps=2.512"}, diskFrameImage[frm, m, #, m, colors] & /@ {.512, 1.512, 2.512}}]] ![enter image description here][5] Below is a GIF showing the capture of the first frame of a disk rotating at rps/fps 1. The code is used to generate all of the following GIFs. With[{m = 200, frm = 1, rps = 1.}, Animate[ Grid[{{ ImageCompose[ colorDisk[angularPosition[frm, row, rps, m], m, colors], Graphics[Line[{{-m, 0}, {m, 0}}]], Scaled[{.5, (m - row)/m}]], ImageCompose[diskFrameImage[frm, row, rps, m, colors], Graphics[Line[{{-m, 0}, {m, 0}}]], Scaled[{.5, .01}]]}}, Alignment -> Top], {row, 1, m}]] ![enter image description here][6] Below are 4 examples showing the capture of the first frame of a video of a rotating disk with a roller shutter at 1fs. The disk rotates at 0.5 rps (top left), 1.0 rps (top right), 1.5 rps (bottom left) and 2.0 rps (bottom right). As the disk rotates faster relative to the shutter, the captured image becomes more complex. ![enter image description here][7] ![enter image description here][8] The subsequent frames are captured the same way. Below are the first 5 frames of a video of a disk rotating at 1.5123 rps: ![enter image description here][9] There ought to be a lot more images that result from the transformation of a rotation into a capture with a rolling shutter. I hope this contribution can inspire more community members. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=combirotodisk15and2.gif&userId=20103 [2]: https://community.wolfram.com/groups/-/m/t/2489445 [3]: https://upload.wikimedia.org/wikipedia/commons/1/15/Rolling_shutter_effect_animation.gif [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=8345locusfulldiskrps1.png&userId=68637 [5]: https://community.wolfram.com//c/portal/getImageAttachment?filename=9528colordiskrpscompare.png&userId=68637 [6]: https://community.wolfram.com//c/portal/getImageAttachment?filename=newrotodiskfrm1rps05small.gif&userId=68637 [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=combirotodisk05and10.gif&userId=68637 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=combirotodisk15and2.gif&userId=68637 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=allframesvideofinal.gif&userId=68637 [at0]: https://community.wolfram.com/web/ghurst Erik Mahieu 2022-06-02T12:22:57Z https://community.wolfram.com/groups/-/m/t/3067969 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=_2_ezgif.com-optimizecopy.gif&userId=11733 [2]: https://www.wolframcloud.com/obj/34c7205a-40cd-4587-b6a1-f1fadc1ed8ce Vitaliy Kaurov 2023-11-21T01:14:10Z https://community.wolfram.com/groups/-/m/t/2763509 *Crossposted: https://mathematica.stackexchange.com/q/278250/13 * ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=RealRational_560.jpg&userId=11733 [2]: https://www.wolframcloud.com/obj/4dd2dfe1-b258-4a55-a8c4-39f6b4f0cfe9 Vitaliy Kaurov 2023-01-06T18:30:28Z https://community.wolfram.com/groups/-/m/t/3022104 An axes aligned bounding box (AABB) tree, a type of [bounding volume hierarchy](https://en.wikipedia.org/wiki/Bounding_volume_hierarchy), is a spatial data structure designed to efficiently find overlapping bounding boxes. It is often used to find broad-level collisions in applications such as collision detection and ray tracing. ## Question I have an implementation in WL that uses some [`Compile`](https://reference.wolfram.com/language/ref/Compile) and [`DataStructure`](https://reference.wolfram.com/language/ref/DataStructure), but is mostly top-level WL code. I'm looking to increase performance, while staying in the WL ecosystem. Are there any tips to speed up my code? In particular any tips to port this over to compiled code using [`FunctionCompile`](https://reference.wolfram.com/language/ref/FunctionCompile.html) & friends? Side note, would be nice if this was a builtin `DataStructure` :) I've posted the code below. Feel free to ask any questions about how it works, etc. ## Examples ### Basic example Bounding boxes are of the form $\{\{x_{min}, x_{max}\}, \{y_{min}, y_{max}\}, \{z_{min}, z_{max}\}\}$. Let's create the AABB tree over 3 bounding boxes and make 1 query bounding box: bboxes = { {{0, 1}, {0, 1}, {0, 1}}, {{0.5, 1.5}, {0.25, 1.5}, {0.3, 0.7}}, {{0.7, 0.8}, {0.8, 2}, {0.2, 0.8}} }; testbbox = {{0.6, 0.9}, {1.1, 1.4}, {0.6, 0.9}}; Create the data structure: aabbtree = AABBTree[bboxes]; Find the indices of the bounding boxes that overlap `testbbox`: overlaps = OverlappingBBoxes[aabbtree, testbbox] (* {2, 3} *) Visualize (query box in blue, overlapping in red, disjoint in green): Graphics3D[{ {FaceForm[Green], Cuboid @@@ Transpose[Delete[bboxes, Partition[overlaps, 1]], {1, 3, 2}]}, {FaceForm[Red], Cuboid @@@ Transpose[bboxes[[overlaps]], {1, 3, 2}]}, {Blue, Cuboid @@ Transpose[testbbox]} }] <img src="https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2023-09-26at10.48.42AM.png&userId=46025" width="360"> ### 4000 bounding boxes, 1 query randomBBox[n_] := With[{llcorners = RandomReal[{-10, 10}, {n, 3}]}, Transpose[{llcorners, llcorners + RandomReal[{1, 1}, {n, 3}]}, {3, 1, 2}] ] SeedRandom[1]; bboxes = randomBBox[4000]; testbbox = randomBBox[1][[1]]; aabbtree = AABBTree[bboxes]; // AbsoluteTiming (* {0.018875, Null} *) (overlaps = OverlappingBBoxes[aabbtree, testbbox]) // AbsoluteTiming (* {0.000692, {1817, 3935, 878, 87}} *) Graphics3D[Cuboid @@@ Transpose[bboxes, {1, 3, 2}]] <img src="https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2023-09-26at10.52.43AM.png&userId=46025" width="360"> Graphics3D[{ {FaceForm[Green], Cuboid @@@ Transpose[Delete[bboxes, Partition[overlaps, 1]], {1, 3, 2}]}, {FaceForm[Red], Cuboid @@@ Transpose[bboxes[[overlaps]], {1, 3, 2}]}, {Blue, Cuboid @@ Transpose[testbbox]}}, PlotRange -> testbbox, PlotRangePadding -> Scaled[0.25] ] <img src="https://community.wolfram.com//c/portal/getImageAttachment?filename=Screenshot2023-09-26at10.52.58AM.png&userId=46025" width="360"> ### 100000 bounding boxes, 100000 queries SeedRandom[1]; bboxes = randomBBox[100000]; testbboxes = randomBBox[100000]; aabbtree = AABBTree[bboxes]; // AbsoluteTiming (* {0.235465, Null} *) OverlappingBBoxes[aabbtree, testbboxes[[1]]]; // AbsoluteTiming (* {0.001988, Null} *) I suspect this can be sped up substantially: Scan[OverlappingBBoxes[aabbtree, #]&, testbboxes]; // AbsoluteTiming (* {44.8445, Null} *) ## Code AABBTree[bboxes_] := Block[{$RecursionLimit = ∞}, iAABBTree[1, MapIndexed[Join, Flatten /@ bboxes][[Ordering[bboxes[[All, 1, 1]]]]]] ] iAABBTree[d_, bboxestagged_] := Block[{len, n, x, Sleft, Scent, Sright, node}, len = Length[bboxestagged]; If[len <= 16, Return[centerNode3D[d, Null, bboxestagged]]]; n = Ordering[bboxestagged[[All, 2d-1]], {Quotient[len+1, 2]}][[1]]; x = Mean[bboxestagged[[n, 2d-1 ;; 2d]]]; {Sleft, Scent, Sright} = partitionIntervals3D[d, bboxestagged, x]; node = centerNode3D[d, x, Scent]; If[Length[Sleft] > 0, node["SetLeft", iAABBTree[Mod[d+1, 3, 1], Sleft]];]; If[Length[Sright] > 0, node["SetRight", iAABBTree[Mod[d+1, 3, 1], Sright]];]; node ] centerNode3D[d_, x_, {}] := CreateDataStructure["BinaryTree", {x, {}, {}, {}, {}, {}, {}, {}, -$MaxMachineNumber, $MaxMachineNumber, d}] (* {center ordinate, xmins, ymins, zmins, xmaxs, ymaxs, zmaxs, inds, dmin, dmax, d} *) (* xmins are the xmin values of all bboxes in this node, ymins, etc follow suit *) (* inds are the respective indices of the bboxes in this node *) (* dmin & dmax are the bounds of all bounding boxes in dimension d (the splitting dimension) *) centerNode3D[d_, x_, intstagged_] := CreateDataStructure["BinaryTree", { x, #1, #2, #3, #4, #5, #6, Developer`ToPackedArray[#7, Integer], Switch[d, 1, #1[[1]], 2, Min[#3], 3, Min[#5]], Max[Switch[d, 1, #2, 2, #4, 3, #6]], d }& @@ Transpose[intstagged] ] partitionIntervals3D[d_, intstagged_, x_] := Block[{data, spec}, data = Partition[partitionIntervals3DC[d, intstagged, x], 7]; spec = Round[data[[-1]]]; {data[[1 ;; spec[[1]]]], data[[spec[[1]]+1 ;; spec[[2]]]], data[[spec[[2]]+1 ;; -2]]} ] partitionIntervals3DC = Compile[{{d, _Integer}, {intstagged, _Real, 2}, {x, _Real}}, Block[{mnp, mxp, cright = 0., ccenter = 0., bagright = Internal`Bag[Most[{0.}]], bagcenter = Internal`Bag[Most[{0.}]], bagleft = Internal`Bag[Most[{0.}]]}, mnp = 2d-1; mxp = mnp+1; Do[ If[Compile`GetElement[i, mxp] < x, Internal`StuffBag[bagright, i, 1]; cright++, If[Compile`GetElement[i, mnp] > x, Internal`StuffBag[bagleft, i, 1], Internal`StuffBag[bagcenter, i, 1]; ccenter++ ] ], {i, intstagged} ]; Join[Internal`BagPart[bagright, All], Internal`BagPart[bagcenter, All], Internal`BagPart[bagleft, All], {cright, cright + ccenter, 0., 0., 0., 0., 0.}] ], CompilationTarget -> "C", RuntimeOptions -> "Speed" ]; OverlappingBBoxes[tree_?DataStructureQ, bbox:{{x1_, x2_}, {y1_, y2_}, {z1_, z2_}}] := Block[{bag, stack, node, a, b, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11}, bag = Internal`Bag[]; stack = CreateDataStructure["Stack"]; stack["Push", tree]; While[!stack["EmptyQ"], node = stack["Pop"]; {d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11} = node["Data"]; {a, b} = bbox[[d11]]; If[Length[d2] > 0 && (d9 <= a <= d10 || d9 <= b <= d10 || a <= d9 <= d10 <= b), Internal`StuffBag[bag, Pick[d8, bboxOverlaps[d2, d3, d4, d5, d6, d7, x1, x2, y1, y2, z1, z2], 1], 1]; ]; If[d1 =!= Null, If[a < d1 && !node["LeftNullQ"], stack["Push", node["Left"]]]; If[d1 < b && !node["RightNullQ"], stack["Push", node["Right"]]]; ]; ]; Internal`BagPart[bag, All] ] bboxOverlaps = Compile[{{epx1, _Real}, {epx2, _Real}, {epy1, _Real}, {epy2, _Real}, {epz1, _Real}, {epz2, _Real}, {x1, _Real}, {x2, _Real}, {y1, _Real}, {y2, _Real}, {z1, _Real}, {z2, _Real}}, Boole @ And[ epx1 <= x1 <= epx2 || epx1 <= x2 <= epx2 || (x1 <= epx1 <= epx2 <= x2), epy1 <= y1 <= epy2 || epy1 <= y2 <= epy2 || (y1 <= epy1 <= epy2 <= y2), epz1 <= z1 <= epz2 || epz1 <= z2 <= epz2 || (z1 <= epz1 <= epz2 <= z2) ], CompilationTarget -> "C", Parallelization -> True, RuntimeOptions -> "Speed", RuntimeAttributes -> {Listable} ]; Greg Hurst 2023-09-26T15:12:15Z https://community.wolfram.com/groups/-/m/t/96823 Recently [b][url=http://community.wolfram.com/groups/-/m/t/91868]Christopher coded a neat wavelet-based method for peak detection[/url][/b]. Peak detection is often needed in various scientific fields: all sorts of spectral analysis, time series and general feature recognition in data come to mind right away.  There are many methods for peak detection. Besides his wavelet-based code Christopher also mentions a built-in MaxDetect function rooted in image processing. [b][url=http://reference.wolfram.com/mathematica/ref/MaxDetect.html]MaxDetect[/url][/b], though, being a rather elaborate tool for multi-dimensional data (2D, 3D images) and with a specific image-processing-minded parameter tuning, was not meant to target time series and other 1D data. This got me thinking. [b]Can we come up with a minimal compile-able peak detection code that would be accurate, robust and fast in most situations for 1D data?[/b] I am not an expert in the subject, but intuitively peak detection consists of two stages.  [list] [*][b]Finding all maxima[/b]. This can be done via Differences with selection of neighbor difference pairs that change from positive to negative. Such pair would indicate local maximum. [*][b]Filtering out peaks[/b]. Selecting those maxima that are “high” splashes of amplitude with respect to its “immediate neighborhood”. Quoted words are relative and depend on particular data. This is why they are set as tuning parameters in the algorithm. [/list]To illustrate 2nd point lets take a look at the [b][url=http://www.wolframalpha.com/input/?i=Albert+Einstein+Wikipedia+page+hits+history]Albert Einstein Wikipedia page hits history[/url][/b] below. Obviously a large peak in the past can be lower than current average in case if there is strong trend in the data. This is why we need “windowing” when looking for peaks, - to compare a peak to its immediate neighborhood.  [url=http://www.wolframalpha.com/input/?i=Albert+Einstein+Wikipedia+page+hits+history][img=width: 600px; height: 469px;]/c/portal/getImageAttachment?filename=ScreenShot2013-08-14at3.13.07PM.png&userId=11733[/img][/url] Without further ado here is a function written specifically in terms of functions that can be compiled. For example I do not use MovingAverage, but do trick with Partition instead.[mcode]PeakDetect = Compile[{{data, _Real, 1}, {width, _Integer}, {cut, _Real}}, (Table[0, {width}]~Join~ Map[UnitStep[# - cut] &, data[[1 + width ;; -1 - width]] - Map[Mean, Partition[data, 1 + 2 width, 1]]]~Join~ Table[0, {width}]) ({0}~Join~ Map[Piecewise[{{1, Sign[#] == {1, -1}}, {0, Sign[#] != {1, -1}}}] &, Partition[Differences[data], 2, 1]]~Join~{0}), CompilationTarget -> "C"];[/mcode]The legend for the function arguments is the following:[list] [*]data – 1D numerical list of data [*]width – half-width of the moving average window not including central point [*]cut – threshold at which to cut off the peak in natural units of data amplitudes  [/list]Now let’s see some usage cases. Let’s import the same Albert Einstein data as a proof of concept. [mcode]raw = WolframAlpha[ "albert einstein", {{"PopularityPod:WikipediaStatsData", 1}, "TimeSeriesData"}]; data = raw[[All, 2]][[All, 1]];[/mcode]We use total window width of 5 points here and cut off peak at a standard deviation of the whole data. The peak labeled “May 2008” is nicely picked up even though it is comparable then current average. This peak is most probably due to publication on May 13, 2008 of [b][url=http://www.amazon.com/Einstein-Life-Universe-Walter-Isaacson/dp/0743264746]one of the most famous books about Einstein[/url][/b] marked as New York Times bestseller that also got award Quill Award. Of course you can play with controls to pick or drop peaks. On the top plot one sees data, moving average, and bands formed by moving average displaced up and down by fraction of standard deviation. Any maximum above the top band becomes a peak. [url=http://www.amazon.com/Einstein-Life-Universe-Walter-Isaacson/dp/0743264746][img=width: 800px; height: 382px;]/c/portal/getImageAttachment?filename=9672ScreenShot2013-08-14at3.47.18PM.png&userId=11733[/img][/url] The code for the app is at the very end. Lets try a different data set – recent sun spot activity. [mcode]raw = WolframAlpha["sun spot", {{"SunspotsPartialTimeSeries:SpaceWeatherData", 1}, "TimeSeriesData"}]; data = raw[[All, 2]];[/mcode]We right away found on May 2013 mark - a [b][url=http://en.wikipedia.org/wiki/Solar_cycle_24#May_2013]most powerful recent event described in Wikipedia page here[/url][/b]. Please let me know if you have suggestions how to speed this up or improve it generally. I would be very curious to know your opinion and critique.  [color=#ff0000][i]The .GIF below is large - wait till it is loaded.[/i][/color] [img=width: 800px; height: 380px;]/c/portal/getImageAttachment?filename=sunspot.gif&userId=11733[/img] The following reference could be useful:[list] [*][b][url=http://www.tcs-trddc.com/trddc_website/pdf/SRL/Palshikar_SAPDTS_2009.pdf]Simple Algorithms for Peak Detection in Time-Series[/url][/b] [*][b][url=http://www.mdpi.com/1999-4893/5/4/588]An Efficient Algorithm for Automatic Peak Detection in Noisy Periodic and Quasi-Periodic Signals[/url][/b] [/list]The code for the interactive app:[mcode]Manipulate[ tt = {#, Rotate[DateString[#, {"MonthNameShort", " ", "Year"}], Pi/2]} & /@ Pick[raw, PeakDetect[data, wid, thr StandardDeviation[data]], 1][[ All, 1]]; Column[{ ListLinePlot[{data, ArrayPad[MovingAverage[data, 1 + 2 wid], wid, "Fixed"], ArrayPad[MovingAverage[data, 1 + 2 wid], wid, "Fixed"] + thr StandardDeviation[data], ArrayPad[MovingAverage[data, 1 + 2 wid], wid, "Fixed"] - thr StandardDeviation[data]}, AspectRatio -> 1/6, ImageSize -> 800, Filling -> {2 -> {1}, 3 -> {4}}, FrameTicks -> {None, Automatic}, FillingStyle -> {Directive[Red, Opacity[.7]], Directive[Blue, Opacity[.7]], Directive[Gray, Opacity[.1]]}, PlotStyle -> Opacity[.7], PlotRange -> All, Frame -> True, GridLines -> Automatic, PlotRangePadding -> 0], Show[ DateListPlot[raw, Joined -> True, AspectRatio -> 1/6, ImageSize -> 800, Filling -> Bottom, Ticks -> {tt, Automatic}, Frame -> False, Mesh -> All, PlotRange -> All], DateListPlot[ If[# == {}, raw[[1 ;; 2]], #, #] &[ Pick[raw, PeakDetect[data, wid, thr StandardDeviation[data]], 1]], AspectRatio -> 1/6, ImageSize -> 800, PlotStyle -> Directive[Red, PointSize[.007]], PlotRange -> All] , PlotRangePadding -> {0, Automatic}] }], Row[{ Control[{{thr, 1, "threshold"}, 0, 2, Appearance -> "Labeled"}], Spacer[100], Control[{{wid, 3, "hal-width"}, 1, 10, 1, Setter}] }] ][/mcode] [b]============== UPDATE =================[/b] Thank you all very much for contributing. I collected everyone's efforts and Danny's two functions in a single completely compile-able expression which seems to give shortest time; - but just vaguely faster then Danny's ingenious maneuver. I very much liked format suggested by Christopher, the one that Michael also kept in his packages. But I wanted to make some benchmarking and thus followed the format returned by the function [b][url=http://reference.wolfram.com/mathematica/ref/MaxDetect.html]MaxDetect[/url][/b] - simply for the sake of speed comparison. This format is just a binary list of the length of original data with 1s in positions of found peaks. Here is the function:[mcode]PeakDetect = Compile[{{data, _Real, 1}, {width, _Integer}, {cut, _Real}}, (Table[0, {width}]~Join~ UnitStep[ Take[data, {1 + width, -1 - width}] - (Module[{tot = Total[#1[[1 ;; #2 - 1]]], last = 0.}, Table[tot = tot + #1[[j + #2]] - last; last = #1[[j + 1]]; tot, {j, 0, Length[#1] - #2}]]/#2) &[data, 1 + 2 width] - cut] ~Join~Table[0, {width}]) ({0}~Join~ Table[If[Sign[{data[[ii + 1]] - data[[ii]], data[[ii + 2]] - data[[ii + 1]]}] == {1, -1}, 1, 0], {ii, 1, Length[data] - 2}]~Join~{0}), CompilationTarget -> "C"]; dat = RandomReal[1, 10^7]; pks = MaxDetect[dat]; // AbsoluteTiming Total[pks] (* ======== output ======== {62.807928, Null} 3333361 ======== output ======== *) pks = PeakDetect[dat, 1, 0]; // AbsoluteTiming Total[pks] (* ======== output ======== {1.560074, Null} 3333360 ======== output ======== *)[/mcode]And here are the speed benchmarks on 10 million data points which shows 40 times speed-up: Vitaliy Kaurov 2013-08-14T21:21:54Z https://community.wolfram.com/groups/-/m/t/3024465 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Leadimage2.png&userId=20103 [2]: https://www.wolframcloud.com/obj/1bbdeed3-67fe-4d71-b844-4827250a00a6 Jari Kirma 2023-09-30T08:43:43Z https://community.wolfram.com/groups/-/m/t/2434669 > *GitHub repository:* https://github.com/ben-izd/cDateFunctions Recently, I was working on an [infographic](https://mathematica.meta.stackexchange.com/a/2608/77079) which heavily involves working with dates and times. After using `DateObject` and some date-related functions, it surprises me how slow they are (in the context of tens of thousands of samples). So I decided to rewrite the main one (`DateDifference`) in Wolfram language. But after seeing the performance boost gained by rewriting, I decided to write some other functions too. After a couple of months, I end up rewriting 19 functions in wolfram language and 9 in kernel-level (with Rust), in this post you'll see the result and interesting points I discovered in this journey. Before we start, remember these notes: - Most of the implementations were done with minimum options, many of them do not support options that their built-in ones support (like `TimeZone`, `TimeSystem`, ...) and functions in the first section written in pure Wolfram language without using `Compile` or related functions - Default calendar is `"Gregorian"`, which doesn't have year `0` (is also true for `"ArithmeticPersian"` calendar) unless it specified - In the 'Possible Issues/Bugs + Suggestions' section order is not important, some of them relate to the design decisions that developers decided (it would be awesome to shed some light on them) - All the implementations have the same name with 'c' added to start (`DayName` is `cDayName`, which has nothing to do with `C` language) - Some built-in functions were not `Listable`, which is understandable because of date formats like `DateList`, but some of my implementations are `Listable` - All the functions were tested with a wide range of random dates and numbers - Except some cases which will be discussed. The result of the functions with their built-in ones are equal. If you find any case with a wrong result, please comment it - Platform is Mathematica 13.0.0 for Microsoft Windows 10 20H2 (64-bit) on AMD Ryzen 1700 with 16 GB RAM with time-zone offset: +3.5 - You can access the code in [GitHub](https://github.com/ben-izd/cDateFunctions) which also include the `LibraryLink` section (`DLL` files and their source code) # Performance Comparison Here are the result of comparing c* functions with their built-in ones in terms of timing (`RepeatedTiming` were used). Tested on different input formats and the number shown here is the floored average of the result. ## LeapYearQ  ## DayName  ## BusinessDayQ  ## DayMatchQ  ## DayRound  ## DateBounds  ## DateOverlapQ  ## DateWithinQ  ## DayCount  ## CurrentDate  ## NextDate  ## PreviousDate  ## DateDifference  ## DayPlus  ## DatePlus   ## DayRangee  ## DateRange  # Possible Issues/Bugs + Suggestions During testing and developing the code, I notice some strange/unexpected cases + some ideas which I'll discuss for each function. ## LeapYearQ ### 1. `"ArithmeticPersian"` uses `"Gregorian"` formula With `CalendarType -> "ArithmeticPersian"` input will be converted to `"Gregorian"` calendar and the result will be calculated like `CalendarType -> "Gregorian"` (note that they have different formulas): ``` LeapYearQ[{1403, 1, 1}, CalendarType -> "ArithmeticPersian"] (* Out: True *) LeapYearQ[{1403, 12, 1}, CalendarType -> "ArithmeticPersian"] (* Out: False *) ``` The reason for above results is that date `{1403, 1, 1}` in `"ArithmeticPersian"` is `{2024, 3, 20}` in `"Gregorian"` which is a leap year in `"Gregorian"` but `{1403, 12, 1}` fall into `{2025, 2, 19}` which is not a leap year, according to `"Gregorian"` calendar. More comprehensive test: ``` And @@ ( LeapYearQ[{#}, CalendarType -> "ArithmeticPersian"] === LeapYearQ[ CalendarConvert[ DateObject[{#}, CalendarType -> "ArithmeticPersian"], "Gregorian"], CalendarType -> "Gregorian"] & /@ Range[1, 5000]) (* Out: True *) ``` Note that in `"ArithmeticPersian"` leap year gap (which has different methods to calculate, Mathematica uses 2820 period) is 4 and sometimes 5 years which is different than `"Gregorian"`. ## BusinessDayQ ### 1. Handling `TimeObject` without raising any message (related to AbsoluteTime-1 problem) When using `TimeObject` in `BusinessDayQ`, it will be converted to a date and return a result! instead of raising an error: ``` DayName[] (* Out: Thursday *) BusinessDayQ[TimeObject[]] (* Out: True *) ``` ## DayMatchQ ### 1. Missing/displace Veterans Day Veterans Day which starts in 1938, on YYYY-11-11 except 1971-1978, which was on Oct 4th Monday [\[Source\]](https://www.timeanddate.com/holidays/us/veterans-day). Considering observed holidays (holiday fall on Saturday/Sunday the day before/after will be observed day), a holiday could be either the specified day or a day before or after. ``` Select[Range[1938, 2050], Not@Or[DayMatchQ[{#, 11, 10}, "Holiday"], DayMatchQ[{#, 11, 11}, "Holiday"], DayMatchQ[{#, 11, 12}, "Holiday"]] &] ``` Result: ``` {1939, 1940, 1944, 1950, 1961, 1967, 1972, 1978, 1989, 1995, 2000, 2006, 2017, 2023, 2028, 2034, 2045} ``` which if you apply `Differences`, we'll get a pattern: ``` {1, 4, 6, 11, 6, 5, 6, 11, 6, 5, 6, 11, 6, 5, 6, 11} ``` Some years are missing the holiday and in the range of 1971-1978 it should not be on YYYY-11-11, but for some years it is. Correct result (use `cDayMatchQ` instead of `DayMatchQ`): ``` {1971, 1972, 1973, 1974, 1975, 1976, 1977} ``` Also, it misses some dates in the future [\[Source\]](https://www.timeanddate.com/calendar/?year=2045&country=1): ``` DayMatchQ[{2045, 11, 10}, "Holiday"] (* Out: False *) (* 1 day before/after also is not a holiday*) DayMatchQ[{2045, 11, 9}, "Holiday"] (* Out: False *) DayMatchQ[{2045, 11, 11}, "Holiday"] (* Out: False *) ``` ### 2. Handling `TimeObject` (related to AbsoluteTime-1 problem) When using `TimeObject` in `DayMatchQ`, it will convert it to a date and return the result ! instead of raising an error: ``` DayName[] (* Out: Thursday *) DayMatchQ[TimeObject[], Thursday] (* Out: True*) ``` ### 3. Documentation typo Based on [`DayMatchQ`](http://reference.wolfram.com/language/ref/DayMatchQ.html) documentation, default value for `DayMatchQ` is `All` while giving error to a single argument: ``` DayMatchQ[{2021, 11, 12}] ```  ## DateWithinQ ### 1. Does not support `DateList` format for input: The documentation notes that input should be `DateObject` of any calendar, it's a good feature to support different calendars but what about `DateList` format? What's the reason behind it? Is it much different from `DateDifference` which supports this format? ``` DateWithinQ[{2021}, {2021, 1}] ``` Also, it should be noted that a *Q function in this example returned an `Unevaluated` expression instead of a Boolean, is this a normal behavior? ### 2. Returning `Unevaluated` expression without raising any message: ``` DateWithinQ[DateObject@{2020, 1, 1, 1, 1, 1}, DateObject@{2020, 1, 1}] ```  ## DayCount ### 1. Round results If it's a day and more than half, it adds one day, otherwise, it doesn't (opposite of what documentation in `Properties & Relations` says, equal to the length of `DayRange` with some options). ``` DayCount[{2020, 1, 1}, {2020, 1, 1, 11}, All] (* Out: 0 *) DayCount[{2020, 1, 1}, {2020, 1, 1, 13}, All] (* Out: 1 *) ``` `cDayCount` returns 0 for both of the above cases. ### 2. Wrong result in year 1 ``` DayCount[{1, 12, 23}, {2, 3, 25}, "BeginningOfMonth"] (* Out: 13 *) ``` ## NextDate ### 1. Inconsistent behavior When using a granularity with an input that doesn't have enough precision, on most of the types, the upper-bound will be used as a start date to find the next occurrences but not for weekdays (not exactly the upper-bound is used). ``` NextDate[{2020, 1}, "Day"] (* Out: DateObject[{2020,2,1}, "Day", "Gregorian", 3.5`] *) NextDate[{2020, 1}, Sunday] (* Out: DateObject[{2020,1,5}, "Day", "Gregorian", 3.5`] *) ``` Also, note that calendar type and time-zone offset was added to the output (input argument does not have those). ### 2. Handling `TimeObject` `NextDate` support `TimeObject`. It can even give us the next `Day` of a `TimeObject`: ``` First@TimeObject[] (* Out: {15, 38, 31.} *) First@NextDate[TimeObject[], "Hour"] (* Out: {2021, 11, 14, 16} *) First@NextDate[TimeObject[], "Day"] (* Out: {2021, 11, 15} *) ``` `cNextDate` result: ``` {16} ``` ### 3. Wrong result in year 1 or -1 ``` DateList@NextDate[{1, 12, 1}, "BeginningOfMonth"] (* Out: {1, 2, 1, 0, 0, 0.} *) DateList@NextDate[{-1, 12, 1, 0, 0, 0}, "Quarter"] (* Out: {-1, 1, 1, 0, 0, 0.} *) ``` Correct result (`cNextDate`): ``` {2, 1, 1, 0, 0, 0.} {1, 1, 1, 0, 0, 0.} ``` ### 4. Inconsistent keyword in documentation [`NextDate`](http://reference.wolfram.com/language/ref/NextDate.html) documentation uses `"MonthFirstDay"` and `"MonthLastDay"` instead of `"BeginningOfMonth"` and `"EndOfMonth"`. ## DateDifference ### 1. Type "quarter"/"Year" includes year 0 As noted in the beginning, the `"Gregorian"` calendar in Mathematica doesn't have year 0 but in these cases it does! ``` DateDifference[{-1, 9, 1}, {1, 9, 1}, "Quarter"] (* Out: Quantity[9.96739, "QuarterYears"] *) ``` Correct Result (using `cDateDifference`): ``` Quantity[4., "QuarterYears"] ``` How many quarters do you see in this picture?  Another example: ``` DateDifference[{-1, 10, 1, 0, 0, 0}, {1, 1, 1, 0, 0, 0}, "Year"] (* Out: Quantity[1.74795, "Years"] *) ``` ### 2. Slightly different result ``` DateDifference[{2020, 3, 5, 15}, {2020, 7, 15, 5}, "Week"] DateDifference[AbsoluteTime@{2020, 3, 5, 15}, AbsoluteTime@{2020, 7, 15, 5}, "Week"] ``` Result (see the last digits): ``` Quantity[18.797619047619047, "Weeks"] Quantity[18.797619047619044, "Weeks"] ``` `cDateDifference` returns the first result. The real decimal can be calculated manually, which is `67/84` or: ``` 0.7976190476190476 ``` ### 3. Raise error for multiple units including `Decade` /`Century` / `Millennium` ``` DateDifference[{2020, 1, 5}, {2120, 1, 4}, "Decade"] DateDifference[{2020, 1, 5}, {2120, 1, 4}, {"Decade", "Year"}] cDateDifference[{2020, 1, 5}, {2120, 1, 4}, {"Decade", "Year"}] ```  ## DayPlus ### 1. Does not support `Week` It supports `"BeginningOfMonth"` and `"EndOfMonth"` but not `"Week"`: ``` DayPlus[{2020, 1, 1}, 1, "Week"] ```  Possible with `cDayPlus`: ``` {2020, 1, 8} ``` ## DatePlus ### 1. Different results with Quantity and List ``` DatePlus[{2020, 1, 1}, {1, "Quarter"}] (* Out: {2020, 4, 1} *) DatePlus[{2020, 1, 1}, Quantity[1, "QuarterYears"]] (* Out: {2020, 4, 1, 12} *) ``` ### 2. Odd Behavior with year 1 ``` DatePlus[{1, 8, 15}, {3, "EndOfMonth"}] (* Out: {1, 3, 31} *) ``` Correct result (with `cDatePlus`): ``` {1, 10, 31} ``` Another example: ``` DatePlus[{1, 10, 5}, {2, "BeginningOfMonth"}] (* Out: {1, 3, 1} *) ``` Correct result: ``` {1, 12, 1} ``` ### 3. Strange result with an empty list When we use an empty list (`{}`) as offset, it uses the first argument as day offset from now: ``` First@DatePlus[0, {}] (* Out: {2021, 11, 13, 15, 7, 0.} *) First@DatePlus[1, {}] (* Out: {2021, 11, 14, 15, 7, 0.} *) DatePlus[0, {1, "Day"}] (* Out: 86400 *) ``` ## DayRange ### 1. Odd behavior with year 1 or -1 It starts from `1-1-31`, just to avoid clutter, you'll see the length: ``` Length@DayRange[{1, 7, 9}, {2, 2, 15}, "EndOfMonth"] (* Out: 13 *) Length@DayRange[{-1, 7, 9}, {1, 2, 15}, "EndOfMonth"] (* Out: 12 *) ``` Proper result (with `cDayRange`): ``` 7 7 ``` ## DateRange ### 1. The backward calculation is done using the forward formula When the end date is larger than the start date, we add the granularity (should be positive) to the start until we reach the end, but if the start is bigger than the end and granularity is negative, we should subtract it from the start until we reach the end. It's true for all the types except `"BeginningOfMonth"` and `"EndOfMonth"`. ``` DateRange[{2019, 12, 5}, {2019, 7, 1}, {-3, "BeginningOfMonth"}] (* Out: {{2019, 10, 1, 0, 0, 0.}, {2019, 7, 1, 0, 0, 0.}} *) ``` Which should be (using `cDateRange`): ``` {{2019, 12, 1, 0, 0, 0.}, {2019, 9, 1, 0, 0, 0.}} ``` Pay attention to hour element: ``` (* Normal *) DateRange[{2019, 12, 5, 12}, {2019, 7, 1}, {-3, "Month"}] (* Out: {{2019, 12, 5, 12, 0, 0.}, {2019, 9, 5, 12, 0, 0.}} *) (* Odd *) DateRange[{2019, 12, 5, 12}, {2019, 7, 1}, {-3, "EndOfMonth"}] (* Out: {{2019, 10, 31, 0, 0, 0.}, {2019, 7, 31, 0, 0, 0.}} *) (* Odd *) DateRange[{2019, 12, 5, 12}, {2019, 7, 1}, {-3, "BeginningOfMonth"}] (* Out: {{2019, 10, 1, 0, 0, 0.}, {2019, 7, 1, 0, 0, 0.}} *) ``` In other terms, the result of the examples below are equal except in different ordering: ``` DateRange[{2019, 12, 5}, {2019, 7, 1}, {-3, "BeginningOfMonth"}] (* Out: {{2019, 10, 1, 0, 0, 0.}, {2019, 7, 1, 0, 0, 0.}} *) DateRange[{2019, 7, 1}, {2019, 12, 5}, {3, "BeginningOfMonth"}] (* Out: {{2019, 7, 1, 0, 0, 0.}, {2019, 10, 1, 0, 0, 0.}} *) ``` ## AbsoluteTime ### 1. Handling `TimeObject` `AbsoluteTime` convert `TimeObject` by adding `Today` value, which I think could be counterintuitive when somebody wants to compare `AbsoluteTime` of two `TimeObject` which was applied in different days, should use `Mod[,86400]` first. ``` DateList@AbsoluteTime@TimeObject[] (* Out: {2021, 11, 13, 15, 42, 56.} *) ``` Why not return this result ( `cAbsoluteTime`): ``` {1, 1, 1, 15, 42, 56.} ``` ## UnixTime ### 1. Truncating sub-second precision: Why `UnixTime` does not have floating numbers to handle more accurate dates? ``` DecimalForm[UnixTime[{2021, 11, 12, 12, 30, 30.35}], 30] (* Out: 1636700430 *) ``` What it could be: `1636707630.35`. ## DateObject ### 1. Raise overlapping message for non-overlap inputs For years, before `812` and after `2989` equality of two consecutive days raise `Message` while `DateOverlapsQ` returning `False`: ``` DateObject[{-1, 1, 1}] == DateObject[{-1, 1, 2}] DateOverlapsQ[DateObject[{-1, 1, 1}], DateObject[{-1, 1, 2}]] (* Out: False *) DateOverlapsQ[DateObject[{-1, 1, 2}], DateObject[{-1, 1, 1}]] (* Out: False *) ```  Far future example: ``` DateObject[{2988, 10, 12}] == DateObject[{2988, 10, 13}] (* No Message, Out: False*) (* But *) DateObject[{2989, 10, 12}] == DateObject[{2989, 10, 13}] ```  ### 2. Suggestion - Round 'Second' in `Second` granularity Is this considered `Second` granularity or `Sub-Second`/`Instant` granularity ? ``` DateObject[{2020, 1, 1, 30, 45, 12.25}, "Second"] (* Out: DateObject[List[2020,1,2,6,45,12.25`],"Second","Gregorian",3.5`] *) ``` ### 3. Incorrect result with AbsoluteTime values with custom granularity Consider date `{2020, 12, 10}` which is `Thursday` and every week starts on Monday (default option): ``` DateList@DateObject[3816547200, "Week"] (* Out: {2020, 12, 10, 0, 0, 0.} *) DateList@DateObject[DateList@3816547200, "Week"] (* Out: {2020, 12, 7, 0, 0, 0.} *) DateList@DateObject[FromAbsoluteTime@3816547200, "Week"] (* Out: {2020, 12, 7, 0, 0, 0.} *) ``` This also applies to `WeekBeginningSunday`, `Century`, `CenturyBeginning01`, `Millennium`, `MillenniumBeginning01`. ## TimeObject ### 1. Suggestion - Embed TimeZone offset to TimeObject like DateObject Since `DateObject` uses `$TimeZone` and can be manipulated in a session while `TimeObject` uses system clock, why not use the time-zone offset from the system ? (My system time-zone offset is +3.5) ``` First@TimeObject[] (* Out: {14, 4, 3.} *) First@TimeZoneConvert[TimeObject[], "GMT"] (* Out: {14, 4, 3.} *) First@TimeZoneConvert[TimeObject[TimeZone -> 3.5], "GMT"] (* Out: {10, 34, 3.} *) (* Real answer *) TimeZoneConvert[DateObject[], "GMT"][[1, -3 ;;]] (* Out: {10, 34, 3.} *) ``` ## CalendarConvert ### 1. Negative/Zero in the month! For far future years, conversion to `"AstronomicalPersian"` will give negative/zero months ``` First@CalendarConvert[DateObject@{9999997, 9, 24}, "AstronomicalPersian"] (* Out: {9999386, -15, 5} *) ``` Another example (wrong output probably starts from this year): ``` First@CalendarConvert[DateObject@{82380, 7, 4}, "AstronomicalPersian"] (* Out: {81758, 12, 29} *) First@CalendarConvert[DateObject@{82380, 7, 5}, "AstronomicalPersian"] (* Out: {81759, 0, 26} *) ``` If we don't apply `First` to the `CalendarConvert` you'll see:  ### 2. Two consecutive leap years in the Persian calendar Persian Calendar has a little complex leap year system, that's why it has an `"ArithmeticPersian"` and `"AstronomicalPersian"` option in calendar conversion. `"AstronomicalPersian"` type is the more accurate system. I don't think two consecutive leap years is correct. Also, I should mention, leap month in the `Persian` calendar is the 12th month which in normal years has 29 days, and leap years have 30 days. ``` First@CalendarConvert[DateObject@{622, 3, 21}, "AstronomicalPersian"] First@CalendarConvert[DateObject@{621, 3, 21}, "AstronomicalPersian"] ``` If we calculate with `cCalendarConvert`: ``` {-1, 12, 30, 0, 0, 0.} {-1, 1, 1, 0, 0, 0.} ``` ## TimeZoneConvert ### 1. Wrong result For version 13 live stream, I want to schedule to watch it live, but I live in a different time-zone, so I notice this strange case: First to show the real offset from my place to `New_York`: ``` TimeZoneOffset["America/New_York", "Asia/Tehran", DateObject[{2021, 12, 13, 15, 30}, TimeZone -> "America/New_York"]] (* Out: -8.5 *) ``` Now, with `8.5` hour offset, convert the date to the Persian calendar, then convert time zones: ``` First@TimeZoneConvert[ CalendarConvert[ DateObject[{2021, 12, 13, 15, 30}, TimeZone -> "America/New_York"], "AstronomicalPersian"], "Asia/Tehran"] (* Out: {1400, 9, 22, 23, 51} *) ``` We get `8h 21m` offset! Further investigation, shows that the offset used in the calculation may differ from the real one: ``` First@TimeZoneConvert[ DateObject[{1400, 9, 22, 3, 31, 15}, CalendarType -> "AstronomicalPersian"], "Europe/London"] (* Out: {1400, 9, 22, 0, 0, 0.} *) ``` If we change the destination to `GMT` the correct result shows up: ``` First@TimeZoneConvert[ DateObject[{1400, 9, 22, 3, 30, 0}, CalendarType -> "AstronomicalPersian"], "GMT"] (* Out: {1400, 9, 22, 0, 0, 0.} *) ``` # Bonus If you have read this far, congratulations. These functions were absent in wolfram but are useful to have, so here you are: ## cHolidayName For getting the name of a holiday: ``` cHolidayName[{2021, 11, 25}] (* Out: "Thanksgiving Day" *) ``` Observed Holidays, the original day is not considered a holiday: ``` cHolidayName[{2021, 7, 5}] (* Out: "Independence Day' day off" *) cHolidayName[{2021, 7, 4}] (* Out: Missing["Holiday"] *) ``` Also if you're more interested in holidays, there are two related internal functions for getting next/previous holiday: ``` DateList@cDateFunctions`Private`cNextHoliday[{2021, 11, 12}] (* Out: {2021, 11, 25, 0, 0, 0.} *) DateList@cDateFunctions`Private`cPreviousHoliday[{2021, 11, 12}] (* Out: {2021, 11, 11, 0, 0, 0.} *) ``` Also for getting the previous holiday, there is a lower limit: ``` cDateFunctions`Private`cPreviousHoliday[{1500, 11, 11}] (* Message: "No official holiday exists before 1863-11-26." *) ``` ## cCalendarView For getting a month/multiple month/custom range calendar views with `Grid` which supports highlighting and is only for `"Gregorian"` calendar (See Advance section for other calendars): ### View a month at a glance: ``` cCalendarView[{2021, 11}] ```  ``` cCalendarView[{2021, 11}, {2021, 11, 12} -> Red] ```  ``` cCalendarView[{2021, 11}, 12 ;; 15] ```  ``` cCalendarView[{2021, 11}, {6, 7} ;; ;; 7] ```  ``` cCalendarView[{2021, 11}, "Holiday"] ```  Use `cCalendarMultipleView` to view multiple months in a row. Could be positive or negative: ``` (* show 2021-11 and 1 month after that with highlighting *) cCalendarMultipleView[{2021, 11}, 1, 6 ;; ;; 6] ```  Use `cCalendarRangeView` to view a range: ``` cCalendarRangeView[{2021, 11}, {2021,12}, {{{2021, 11, 14}, {2021, 11, 15}, {2021, 11, 16}} -> Gray, {2021, 11, 13} -> Green}] ```  Note that `cCalendarMultipleView` and `cCalendarRangeView` automatically highlight the first day of each month. ## Suggestion - N-th Weekday How should we program to express 4th Thursday (for Thanksgiving ) / last Monday (for Memorial Day) ? Here is my proposal inspired by the `C++ Chrono` library, implemented via `UpValues` (minimum viable product): ``` (Unprotect[#]; UpValues[#] = {}; # /: {year_Integer, month_Integer, #[n_Integer?Positive]} := DatePlus[{year, month, 0}, {n, #}]; # /: {year_Integer, month_Integer, #[n_Integer?Negative]} := DatePlus[{year, month + 1, 1}, {n, #}]; # /: {year_Integer, month_Integer, #[First]} := DatePlus[{year, month, 0}, {1, #}]; # /: {year_Integer, month_Integer, #[Last]} := DatePlus[{year, month + 1, 1}, {-1, #}]; Protected[#];) & /@ {Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday}; ``` Now, we can use it anywhere (should be fixed): ``` {2021, 11, Thursday[4]} (* Out: {2021, 11, 25} *) {2021, 5, Monday[Last]} (* Out: {2021, 5, 31} *) AbsoluteTime[{2021, 5, Monday[Last]}] (* Out: 3831408000 *) (* Second to last friday *) {2021, 12, Friday[-2]} (* Out: {2021, 12, 24} *) ``` Limiting UpValues to date functions only and supporting month names, would greatly improve functionality and readability. # For Advanced Users This journey of rewriting date functions in wolfram as joyful and teachable as it was, was ending and after looking at my code, I notice in many places, `DateList` and `AbsoluteTime` were used many times, I thought, if I could make them faster, the whole package could benefit this speed up. In many parts, I reached my goal but I didn't test the performance gained from switching. Since these functions are Kernel-Level functions (probably written in `C` language), Wolfram Language is not the right tool to compete, so I choose Rust. You could read about its benefits and guarantees over `C` on other sites. Here you'll see the comparison. `C` language is still used as the interface for Rust functions. (If you have a better solution, I'm more than happy to hear about Rust integration.) This journey just like the previous one doesn't end with `DateList` and `AbsoluteTime`, but it also came with writing 3 calendars (`"Greogiran"`, `"ArithmeticPersian"` with 5 method, `AstronomicalPersian`, `"Islamic"` which is a civil arithmetic calendar, with 4 method). For additional options they support, visit the [Github Page](https://github.com/ben-izd/cDateFunctions). ## Installation For testing this section, you need to follow the [Github](https://github.com/ben-izd/cDateFunctions) instructions. ## `AbsoluteTime` / `FromAbsoluteTime` `cAbsoluteTime` is on average **0.9 times slower** than `AbsoluteTime` over 1,000,000 random `DateList`. `cAbsoluteTime` is on average the **2 times faster** as `AbsoluteTime` over 100,000 random `DateObject`. If we call `AbsoluteTime` to get the current value without any argument, `cAbsoluteTime` is on average **42 times faster** than `AbsoluteTime` while having **6 times lower** memory usage. `cFromAbsoluteTime` on the other hand is **1.5 times faster** than `FromAbsoluteTime` over 100,000 random real, which could be faster considering creating a `DateObject` overhead (see Comparing Internals section). Also unlike Mathematica, `cAbsoluteTime` does support other calendars: ``` cDateList@cAbsoluteTime[{1400, 7, 20}, CalendarType -> "ArithmeticPersian"] (* Out: {2021, 10, 12, 0, 0, 0} *) cDateList@cAbsoluteTime[{1443, 3, 5}, CalendarType -> "Islamic"] (* Out: {2021, 10, 12, 0, 0, 0} *) ``` ## `DateList` `cDateList` is on average **~1.7 times faster** than `DateList` over 1,000,000 random numbers (`Real`/`Integer`). `cDateList` is on average **~1.9 times faster** than `DateList` over 1,000,000 random un-normal `DateList`. ## `CalendarConvert` All the numbers come from testing 10,000 random samples. In `"Gregorian"` to `"ArithmeticPersian"`\ `cCalendarConvert` is on average **16 times faster** than `CalendarConvert`. In `"Gregorian"` to `"AstronomicalPersian"`\ `cCalendarConvert` is on average **140 times faster** than `CalendarConvert`. In `"Gregorian"` to `"Islamic"`\ `cCalendarConvert` is on average **9 times faster** than `CalendarConvert`. In `"Islamic"` to `"ArithmeticPersian"`\ `cCalendarConvert` is on average **19 times faster** than `CalendarConvert`. `"AstronomicalPersian"` uses an algorithm introduced by Edward M. Reingold, Nachum Dershowitz in their book, "Calendrical Calculations: The Ultimate Edition". It was implemented to perform the test but because of its license, it's not included in the source code until I get approval from the author (the source is commented in the interface code). Since both `"Islamic"` and `"ArithmeticPersian"` have methods to use, converting one to another while using `Method` option, input `Method` will be used for the output format and input will be converted using the default method, if you want more control, first convert it to `"Gregorian"`, then convert it to your specified calendar. ## `LeapYearQ` Because of different methods of calculations, this `LeapYearQ` uses kernel-level functions (instead of pure wolfram language shown earlier, which will be overridden) to handle 3 calendars. Tests were done on over 100,000 random samples. `"Gregorian"`\ `cLeapYearQ` is on average **8 times faster** than `LeapYearQ`. `"ArithmeticPersian"`\ `cLeapYearQ` is on average **9 times faster** than `LeapYearQ` (as discussed earlier, `LeapYearQ` gives the wrong result for this type of calendar). `"AstronomicalPersian"`\ `cLeapYearQ` is on average **102 times faster** than `LeapYearQ`. `"Islamic"`\ `cLeapYearQ` is on average **6 times faster** than `LeapYearQ`. And remember in `cLeapYearQ` with `"ArithmeticPersian"` and `"Islamic"`, you can choose different methods of leap-year calculation, visit [Github](https://github.com/ben-izd/cDateFunctions) for more information. ## `JulianDate` / `FromJulianDate` `cJulianDate` is on average **3 times faster** than `JulianDate` over 10,000 random `DateList`. `cFromJulianDate` is on average **1.8 times faster** than `FromJulianDate` over 100,000 real, actually, it's faster than stated, creating a `DateObject` takes most of the time. ## `UnixTime` / `FromUnixTime` `cUnixTime` is on average **7 times faster** than `UnixTime` over 100,000 random `DateList` samples. `cFromUnixTime` is on average **2 times faster** than `FromUnixTime` but it shares the same problem as `cFromJulianDate`. ## `cMonthView` / `cPartialView` With access to these calendars and different methods of calculation, it was the best opportunity to create a function that prints month/year view. See 2021 calendar at a glance:  or other calendars (use `CalendarType`)  Unlike earlier `cMonthView` it does not support highlighting yet. For printing a range/month continuously use `cPartialView` which accepts a start and end date. Wolfram technology conference at a glance (with manually highlighting): ``` MapAt[Highlighted, cPartialView[{2021, 11, 7}, {2021, 11, 20}], {{1, 1, 1, 6 ;; 7}, {1, 1, 2, 1 ;; 2}}] ```  # Limitations Since all these functions are written in low-level and should support `C` types, not arbitrary precision that Wolfram-Language has (although Wolfram has its own limitation, like rounding error). ## `cAbsoluteTime`: `cAbsoluteTime` has a limit, going further will not raise any error but cause overflow (wrap around). ``` (* max date - second is not important *) cAbsoluteTime[{292277026526, 12, 5, 15, 30}] (* Out: 9223372036854720000 *) (* going further lead to overflow *) cAbsoluteTime[{292277026526, 12, 5, 15, 31}] (* Out: -9223372036854775756 *) (* min date*) cAbsoluteTime[{-292277022728, 1, 27, 0, 0, 0}] (* Out: -9223372036854720000 *) (* going further lead to overflow *) cAbsoluteTime[{-292277022728, 1, 26, 0, 0, 0}] (* Out: 9223372036854745216 *) ``` ## `cDateList`: Unlike `cAbsoluteTime` which crossing the limit will not raise any error, here Mathematica will raise an error because it can not convert numbers greater than `2^63-1` to machine size 64-bit integers. But because of some internal design decisions, the safe range is shifted towards the negative side. So there is also a range before `2^63-1`, which doesn't raise any error but leads to overflow. (We're discussing around year `292,277,024,627`, which if you're interested, I'll add more details.) This pattern also applies to other kernel-level functions. # Comparing `Internal`s In the built-in `Internal` context, there are three functions that do the `DateList` functionalities separately: - `DateListToDateList` - `DateListToSeconds` - `SecondsToDateList` Like Mathematica, c\* functions also have kernel-level functions to call but there is a caveat. Since passing arrays to/from `C` language (interface), should be a single type, I couldn't find an easy way to handle `DateList` mix-type arrays beside `WSTP` (second could be Real, `LibraryLink` developers, if you're reading, I'm waiting for integration of `TypeProduct` to use structs), so these kernel functions only support maximum 5 elements as input/output (`{Y,M,D,H,M}`, second will be calculated in the interface, not in the kernel part). Comparing Mathematica `Internal` with `cDateFunctions` > `LibraryLink` over random 1,000,000 samples without considering second (`Internal` support second but not `cDatefunctions Kernel`): `DateListToSecond` (`AbsoluteTime`): `LibraryLink` is on average **0.7 times slower** than `Internal`. `SecondToDateList` (`DateList`): `LibraryLink` is on average **0.8 times faster** than `Internal`. `DateListToDateList`: `LibraryLink` is on average **1.12 times faster** than `Internal`. Calling raw `LibraryLink` outperforms the internals but input/output doesn't include second, the result shown above came from calling the simplest interface. (For those curious people, calling raw interface is on average **8, 7, 9 times faster** respectively). As discussed, `cDateFunctions`\``LibraryLink` is fast but the reason that their interface is not as fast as their kernel, is mainly because of the pattern matching (to support `CalendarType` and `Method`) and second calculation. # Final note For those who are interested in calendars and their implementations, I recommend "Calendrical Calculation" by Edward M. Reingold, Nachum Dershowitz which I think the Wolfram/Microsoft team has seen this book because of some similarities I noticed in their sources. Both book's authors and developer teams did a great job which is appreciated (Microsoft: .Net). The book also comes with a source file in `Lisp` language. The implementation of `cCalendarConvert` `"AstronomicalPersian"` is ported from this book. Because of my limited time, I did cover 4 calendars and introduced different methods of calculation to cover different needs and you should know both `"Persian"` and `"Islamic"` in some countries use observation not algorithms to determine leap year/month length. The Rust code has a framework within, to support other calendars with similar characteristics as Gregorian and ... , with providing months length and two functions for checking and counting leap years and some little details, you'll have `AbsoluteTime` and `DateList` engine + `Month/YearView` (currently for 12 months calendars) for your calendar for free. `"ArithmeticPersian"` and `"Islamic"` also use frameworks, so for any future changes in their leap year calculations that follow the old structure, you can add them as a new method under 20 lines of code. Improving is a never-ending process. For me, I reached beyond the main goal I set for myself, of course, there were a lot of moments which I was so close to abandoning the project, especially when I was figuring out how `LibraryLink` works and build the project without compiling errors but looking back, it's worth it and I feel more powerful than before. I hope you learn something and if you had any questions/comments/suggestions, feel free to add them. Benjamin Izadpanah 2022-01-01T06:44:51Z https://community.wolfram.com/groups/-/m/t/1057588 My project, Parallel Mathematica Environment on the RaspberryPi using OOP, is a sample application of **Object Oriented Programming for the Mathematica** cluster computing, implemented with a Mac and three RaspberryPi Zero connected with a USB hub and three USB cables. Basic idea is to deploy a constructed instance image to calculating servers (RaspberryPi) and send messages to the instance. [OOP on the Mathematica is already developed and shown][1] in this community, and further detail is shown on [slidesshare][2] titled of "OOP for Mathematica." ![enter image description here][3] ---------- Preparing for RaspberryPi Zero is as follows using SSH connection from a Mac, - naming each Zero as raspberypi,raspberrypi1,raspberrypi2,... - set the server program "init" to each RaspberryPi, init is, $ cat init While[True, Run[“nc -l 8000>input”]; temp=ReleaseHold[<<input]; temp >>output; Run[“nc your-mac-hostname.local 8002<output”] ] where, socket numbers must be identical. - Run Mathematica manually, and wait the booting Mathematica up. $ wolfram <init& Checking each RaspberryPi is useful as, $ nc -l 8002 >output|nc raspberrypi.local 8000 <<EOF > 10! > EOF $ cat output 3628800 ---------- Cluster controller program on a Mac is, - set directory SetDirectory[NotebookDirectory[]]; - setup socket communication process com1="nc -l 8002 >output1 |nc raspberrypi.local 8000 <input1"; com2="nc -l 9002 >output2 |nc raspberrypi1.local 9000 <input2"; com3="nc -l 9502 >output3 |nc raspberrypi2.local 9500 <input3"; - set object property obj={ <|"name"->node1,"comm"->com1,"in"->"input1","out"->"output1","p"->{2000,3500}|>, <|"name"->node2,"comm"->com2,"in"->"input2","out"->"output2","p"->{3501,4000}|>, <|"name"->node3,"comm"->com3,"in"->"input3","out"->"output3","p"->{4000,4500}|>}; - define calculation server class, where is a sample Mersenne prime number calculation new[nam_]:=Module[{ps,pe}, mersenneQ[n_]:=PrimeQ[2^n-1]; setv[nam[{s_,e_}]]^:={ps,pe}={s,e}; calc[nam]^:=Select[Range[ps,pe],mersenneQ] ]; - construct instances Map[new[#name]&,obj]; - deploy instances to calculation servers Map[Save[#in,#name]&,obj]; Map[Run[#comm]&,obj]; - send message to each instance Map[Put[Hold@setv[#name[#p]],#in]&,obj]; Map[Run[#comm]&,obj]; - start calculation Map[Put[Hold@calc[#name],#in]&,obj]; proc=Map[StartProcess[{$SystemShell,"-c",#comm}]&,obj] - wait for the process termination (mannualy in this sample code) Map[ProcessStatus[#]&,proc] {Finished,Finished,Finished} - gather the results Map[FilePrint[#out]&,obj]; {2203, 2281, 3217} {} {4253, 4423} [1]: http://community.wolfram.com/groups/-/m/t/897081?p_p_auth=o5qxZhNR [2]: https://www.slideshare.net/kobayashikorio/oop-for-mathematica [3]: http://community.wolfram.com//c/portal/getImageAttachment?filename=2017-04-10.jpg&userId=897049 Hirokazu Kobayashi 2017-04-10T01:15:22Z https://community.wolfram.com/groups/-/m/t/2516662 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/afc17dcc-20ac-4990-a933-02fed606f0af Anton Antonov 2022-04-22T17:14:10Z https://community.wolfram.com/groups/-/m/t/3082695 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Heroimage.gif&userId=20103 [2]: https://www.wolframcloud.com/obj/3e0cfefc-7168-4b87-87b7-554df0aa62c5 Kotaro Okazaki 2023-12-13T13:45:49Z https://community.wolfram.com/groups/-/m/t/3725603 [![Filling Mars with water: Martian dichotomy and oceans versus continents at Earth's 71% water surface][1]][2] &[Wolfram Notebook][3] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=sdsfswae.jpg&userId=11733 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=sdsfswae.jpg&userId=11733 [3]: https://www.wolframcloud.com/obj/9c752467-44a5-4184-a280-22a6dbf960a8 Vitaliy Kaurov 2026-06-02T04:19:42Z https://community.wolfram.com/groups/-/m/t/2961701 [![enter image description here][1]][2] &[Wolfram Notebook][3] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=43t45h545qerfgvfdsfh.jpg&userId=11733 [2]: https://community.wolfram.com//c/portal/getImageAttachment?filename=43t45h545qerfgvfdsfh.jpg&userId=11733 [3]: https://www.wolframcloud.com/obj/ec0d857e-09b2-4862-87b8-0d0c23869f69 Vitaliy Kaurov 2023-07-13T17:19:37Z https://community.wolfram.com/groups/-/m/t/2416125 &[Wolfram Notebook][1] [1]: https://www.wolframcloud.com/obj/a5d46446-4ecd-4c49-9f10-57561e3961a5 Mohammad Bahrami 2021-11-30T17:22:04Z https://community.wolfram.com/groups/-/m/t/2429964 World's computation is gradually switching to GPUs and other SIMD devices, and if Mathematica doesn't adapt, it'll end up cornered as a tool for toy computations. Are there plans to make it easy to use GPUs in Mathematica? You can see the reason for this in the graph below ![enter image description here][1] 2006 witnessed the death of Dennard scaling and since them, much of new performance growth went into parallel computation rather than sequential. You can now buy a gaming card RTX 3090 with 36 teraflops of computation, compared to [something like](https://bioinfo.lifl.fr/yass/iedera_dominance/boinc_cpu_list_additional.html) 200 gigaflops for CPU, and the gap is growing. These SIMD flops have gotten so cheap, that companies like Google are giving them out for free ([https://colab.research.google.com/](colab) runs things for free on a low-end GPU) As a concrete example is this [issue](https://mathematica.stackexchange.com/questions/261012/optimizing-sequential-computation). I couldn't find an easy way to move this computation to GPU so had to reimplement it in Python. The changes you need to switch from CPU to GPU are [minimal](https://colab.research.google.com/drive/1rX78CmRkZJFAvzzyY8nS__H--1GvDky2#scrollTo=javoz9t7FAMu) in modern numpy-like frameworks (ie, Jax, CuPY, PyTorch): add ".cuda()" in front of some tensors and their computations are automatically on GPU. Mathematica could use similar approach to gradually enter the realm of user-friendly GPU computation -- first add an operation to move an array to GPU and back, and then implement GPU version of operation for the most common operations (matrix multiply), then gradually expand to more operations requested by users. [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=ScreenShot2021-12-21at4.12.13PM.png&userId=1046145 Yaroslav Bulatov 2021-12-22T00:28:20Z https://community.wolfram.com/groups/-/m/t/3062083 ![Conformal transformation of hexagonal grid][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=plane36r.gif&userId=610054 [2]: https://www.wolframcloud.com/obj/1a895dac-598e-4187-baa0-ea87e5c163e2 Clayton Shonkwiler 2023-11-08T18:00:47Z https://community.wolfram.com/groups/-/m/t/2932299 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=Glam-ifying3DGraphicsSimpleVideoAnimationsandTechniques.gif&userId=20103 [2]: https://www.wolframcloud.com/obj/cce345b0-10b4-49ec-b233-219984d53fce Anika Patel 2023-06-07T18:27:09Z https://community.wolfram.com/groups/-/m/t/2510803 *MODERATOR NOTE: a submission to computations art contest, see more:* https://wolfr.am/CompArt-22 ---------- ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=WolfFractal.jpg&userId=20103 [2]: https://www.wolframcloud.com/obj/c862eb6d-895d-489a-a4bb-2b3362f3b033 Sander Huisman 2022-04-14T21:59:45Z https://community.wolfram.com/groups/-/m/t/2472460 *MODERATOR NOTE: a submission to computations art contest, see more:* https://wolfr.am/CompArt-22 ![enter image description here][1] &[Wolfram Notebook][2] [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=test2_15_2_22.gif&userId=20103 [2]: https://www.wolframcloud.com/obj/a17f36f5-1294-41f4-87c2-6f5cb188d12f Ben Bartlett 2022-02-15T17:50:34Z https://community.wolfram.com/groups/-/m/t/1927307 ![demo][1] ##Abelian Sandpile Model## We call this Mathematica function to retrieve some background information from Wiki page: `WikipediaData["Abelian sandpile model"]` > The Abelian sandpile model, also known as the Bak\[Dash]Tang\[Dash]Wiesenfeld model, was the first discovered example of a dynamical system displaying self-organized criticality. It was introduced by Per Bak, Chao Tang and Kurt Wiesenfeld in a 1987 paper.The model is a cellular automaton. Also [in NKS][2] there is discussion about self-organized criticality and some simple demo codes in Mathematica. You may also find [this Numberhile video][3] on this topic interesting, typically it talks about where "Abelian" comes from based on group theory point of view. [![note][4]](https://www.wolframscience.com/nks/notes-7-9--self-organized-criticality/) I found [a GitHub repo][5] with a nice implementation of the model in C++ and it works quite well for large number ( ~ 1 million initial sand particles at origin). I think this can be a good example to show how to use `LTemplate` package to call this C++ package from Mathematica if you do not want to start from scratch using the basic LibraryLink C API's. ##Implementation## Follow [this link][6] to install `LTemplate` package and refer to the document page coming with this package to find the basic example and some type conversion information (well, not like Mathematica you need to specify the type by yourself). In particular we need `mma::IntMatrixRef` as return type and to transfer a 2D array back to Mathematica. Meanwhile, we also need `ra = mma::makeMatrix<mint>` (`ra` stands for raw array) to store the grid information during computation. This `ra` also replaces the double pointer in the original code to handle the array for plot later. Here is the code #include <LTemplate.h> #include <numeric> #include <iomanip> class sandpile { public: const int dx[4] = { 1,0,-1,0 }; const int dy[4] = { 0,1,0,-1 }; const int N_size = 781; const int N_size2 = 390; struct L_coord { unsigned int x; unsigned int y; }; mma::IntMatrixRef CreateSandpile(int n ) { auto ra = mma::makeMatrix<mint>(N_size,N_size); std::fill(ra.begin(),ra.end(),0); bool** V_sites = new bool*[N_size]; bool** Move_ = new bool*[N_size]; unsigned int** Odometer = new unsigned int*[N_size]; for (int k = 0; k < N_size; k++) { V_sites[k] = new bool[N_size]; Odometer[k] = new unsigned int[N_size]; Move_[k] = new bool[N_size]; for (int i = 0; i<N_size; i++) { Move_[k][i] = false; V_sites[k][i] = false; } } L_coord* walking_ = new L_coord[N_size*N_size]; V_sites[N_size2][N_size2] = true; Move_[N_size2][N_size2] = true; ra(N_size2,N_size2) = n; walking_[0].x = N_size2; walking_[0].y = N_size2; unsigned int x = 0, y = 0; int top_ = 0; unsigned int lx = 0, ly = 0; unsigned int d = 0; unsigned int max_of_top = 0; unsigned long N_of_Moves = 0; while (top_ >= 0) { N_of_Moves += 1; if (max_of_top < top_) { max_of_top = top_; } x = walking_[top_].x; y = walking_[top_].y; top_--; Move_[x][y] = false; d = (ra(x,y) >> 2); ra(x,y) = ra(x,y) - (d << 2); for (int k = 0; k < 4; k++) { lx = x + dx[k]; ly = y + dy[k]; V_sites[lx][ly] = true; ra(lx,ly) += d; if (Move_[lx][ly] == false && ra(lx,ly) >= 4) { walking_[++top_].x = lx; walking_[top_].y = ly; Move_[lx][ly] = true; } } } for (int k = 0; k<N_size; k++) { delete[] V_sites[k]; delete[] Odometer[k]; delete[] Move_[k]; } delete[] V_sites; delete[] Odometer; delete[] Move_; delete[] walking_; return ra; } }; I keep most part intact, only made several essential modification according to the document of LTemplate: - Include `LTemplate` head file - Create a class called `sandpile` which wraps around the `CreateSandpile` function to be imported - Return `mma::IntMatrixRef` object instead of `Clock` object used to print computation time cost ##Compile and Load## In a Mathematica notebook, I run the following lines. First I need to load the package << LTemplate` SetDirectory[NotebookDirectory[]] (*put sandpile.h in the same directory as the working nb*) we need to tell Mathematica which class to load and to pick the right member function at the same time template =LClass["sandpile",{ LFun["CreateSandpile", {Integer}, {Integer, 2}] }]; If our compiler is set up correctly, with verbose log, we should compile the class to dll just like in VS developer command or xcode or some clang/gcc environment on Linux. CompileTemplate[template, "ShellCommandFunction" -> Print, "ShellOutputFunction"->Print ] ![output][7] The last line is the dll in the `$UserBaseDirectory`. Then we can load the dll with LoadTemplate[template] Import the class sp = Make["sandpile"] and call the member function: sp@"CreateSandpile"[100] I omitted one ArrayTrim function in C++ code to simplify the process. We actually need it to trim margins with all zeros in a row or column. I choose to implement it quickly in Mathematica with much shorter lines than the c++ version, even though we might transferred many null points back to kernel wasted. Nevertheless, cleanRow[arr_] := DeleteCases[arr, _?(Total@# == 0 &)] For small set, the With[{arr = cleanRow@Transpose@cleanRow@(sp@"CreateSandpile"[5000])}, ArrayPlot[arr, ColorRules -> {0 -> Black, 1 -> Magenta, 2 -> Red, 3 -> Blue}]] ![demo2][8] Growing complexity Style[Grid[Partition[Map[With[{arr=cleanRow@Transpose@cleanRow@(sp@"CreateSandpile"[#])}, ArrayPlot[arr,ColorRules->{0->Black,1->Magenta,2-> Red,3-> Blue}]]&, {10,20,50,80,100,200,500,800,1000,5000,10000,50000,100000,200000,500000}],5]], ImageSizeMultipliers->0.4] ![gc][9] I tested the code on 10 million initial points and no problem at all to display it nicely with `ArrayPlot`. The animation at the beginning is only shows a quarter of the whole graph due to symmetry. The color scheme is the same as the [original post][10]. - 1 Million particles at center at t-zero With[{arr = cleanRow@Transpose@cleanRow@(sp2@"CreateSandpile"[1000000])}, ArrayPlot[arr,ColorRules -> {0 -> Black, 1 -> Magenta, 2 -> Red, 3 -> Blue}]] ![demo3][11] - 10 Million particles at center at t-zero (only show quarter) and the result is attached in csv format. ImageCrop@ Colorize[sp1@"CreateSandpile"[10000000], ColorRules -> {0 -> White, 1 -> Green, 2 -> Purple, 3 -> RGBColor["#D4AF37"]}] ![demo4][12] ##Summary## - Use `LibraryLink` based package `LTemplate` to call a C++ function with one integer input and 2D integer output. - Demonstrate how to pass 2D array in C++ back to Mathematica - Use `ArrayPlot` to visualize tensor computed in C++ environment - Safely wrap `Manipulate` or `ListAnimate` around external library function - Mathematica implementation can also be found [here][13] - [Analytic research][14] on this topic's complexity [1]: https://community.wolfram.com//c/portal/getImageAttachment?filename=demo.GIF&userId=23928 [2]: https://www.wolframscience.com/nks/notes-7-9--self-organized-criticality/ [3]: https://www.youtube.com/watch?v=1MtEUErz7Gg [4]: https://community.wolfram.com//c/portal/getImageAttachment?filename=7183note.jpg&userId=23928 [5]: https://github.com/hayk314/Sandpiles/blob/master/C++/AbelSand/AbelSand/Abel.cpp [6]: https://github.com/szhorvat/LTemplate [7]: https://community.wolfram.com//c/portal/getImageAttachment?filename=2285output.jpg&userId=23928 [8]: https://community.wolfram.com//c/portal/getImageAttachment?filename=5719demo2.jpg&userId=23928 [9]: https://community.wolfram.com//c/portal/getImageAttachment?filename=3273demo4.jpg&userId=23928 [10]: https://github.com/hayk314/Sandpiles/blob/master/C%2B%2B/AbelSand/Debug/Abel1000000.png [11]: https://community.wolfram.com//c/portal/getImageAttachment?filename=3492demo3.jpg&userId=23928 [12]: https://community.wolfram.com//c/portal/getImageAttachment?filename=nb.png&userId=23928 [13]: https://www.complex-systems.com/abstracts/v17_i01_a02/ [14]: https://www.complex-systems.com/abstracts/v17_i01_a02/ Shenghui Yang 2020-04-07T04:51:16Z