Group Abstract Group Abstract

Message Boards Message Boards

0
|
2.8K Views
|
2 Replies
|
2 Total Likes
View groups...
Share
Share this post:
GROUPS:

Using DynamicArray in Parallel Kernels

Posted 1 year ago

I'm working on an application that which iterates through 6 billion combinations, appending selected items for further use. It is a natural for data parallism. The new DynamicArray datastructure is also a good match since the number of aggregations is quite long and ill-suited for AppendTo.

The DynamicArray structure cannot (currently) be used as Shared Variable so I developed a work-around. I started by adapting the shared variable example in the Help page for ParallelDo to an example using kernel data parallelism:

SetSharedVariable[ppr];
kernels = 4;
ppr = Table[{}, kernels];
LaunchKernels[kernels];
ParallelDo[
  Do[
   If[PrimeQ[2^i - 1], AppendTo[ppr[[k]], i]], {i, ((k - 1)*1000) + 1,
     k*1000}
   ],
  {k, 1, kernels}
  ];
CloseKernels[];
UnsetShared[ppr];
pr = Flatten[ppr];
Print["pr = ", pr];

pr = {2,3,5,7,13,17,19,31,61,89,107,127,521,607,1279,2203,2281,3217}

Within it, I created DynamicArray structures for aggregation, and then stored their Normal contents to the kernel array element:

SetSharedVariable[ppr];
kernels = 4;
ppr = Table[{}, kernels];
LaunchKernels[kernels];
ParallelDo[
  tmp = CreateDataStructure["DynamicArray"];
  Do[
   If[PrimeQ[2^i - 1], tmp["Append", i]], {i, ((k - 1)*1000) + 1, 
    k*1000}
   ];
  ppr[[k]] = Normal[tmp],
  {k, 1, kernels}
  ];
CloseKernels[];
UnsetShared[ppr];
pr = Flatten[ppr];
Print["pr = ", pr];

pr = {2,3,5,7,13,17,19,31,61,89,107,127,521,607,1279,2203,2281,3217}

Thanks to the Wolfram team for adding this family of data structures to Mathematica.

POSTED BY: Richard Frost
2 Replies

It’s a good structure to have. It was implicitly there already, in Sow/Reap. This new structure makes it more obvious and is better suited to a procedural style of coding. Much easier than trying to remember the layout of Reap results.

POSTED BY: Daniel Lichtblau

I've noticed that my Intel Comet Lake shared memory multiprocessor architecture is very "lazy" about garbage collection (not just with Mathematica), procrastinating until the situation is dire. This can happen too late with parallel kernels, resulting in application crash. My worst experience in large memory applications is with functional programming. I suspect this is due to the many implicit structures that can be created within a single expression. I found that converting these to procedural statements with explicitly named structures improved the situation. When I've finished with a structure I assign an empty list to it. I tried Clear but it doesn't have the same effect.

POSTED BY: Richard Frost
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard