Message Boards Message Boards

0
|
2923 Views
|
13 Replies
|
0 Total Likes
View groups...
Share
Share this post:

How do I parallelize my code to improve its speed?

Posted 1 year ago

Hi,

I'm not sure if this is the correct group for this question, if not please suggest a better group.

I'm running Mathematica 13.0.1.0 with up to 8 kernels and would like to parallelize

If[MemberQ[symitems, list[[#]]], {}, list[[#]]] & /@ Range[2, len]

to get it to run faster but so far every attempt has resulted in significantly longer computation times. "len" is the length of "list." Both "symitems" and "list" are lists of lists of lists where "symitems" are all possible symmetry mappings (which range from S(2)xS(2)xS(2)xS(1) to S(5)xS(5)xS(5)xS(4)) of "list[[1]]." I've never understood when or how to use any of Mathematica's "Share" functions, which might be the source of my problems. When "symitems" and "list" are short, the serial code is acceptably fast, however as they get larger the serial code become extremely slow (there is nothing I can do to speed up "MemberQ"). I've tried:

ParallelMap[If[MemberQ[symitems, list[[#]]], {}, list[[#]]] &, Range[2, len]]
ParallelTable[If[MemberQ[symitems, list[[j]]], {}, list[[j]]], {j,2, len}]
ParallelTable[If[MemberQ[symitems, list[[j]]], {}, list[[j]]], {j,2, len},Method->"CoarsestGrained"]

What else can I try?

Thanks,
Troy

POSTED BY: Troy Wahl
13 Replies
Posted 1 year ago

Can you provide a concrete method for generating list and symitems, or at least some more specifics about their dimensions. I can't test without some data.

POSTED BY: Eric Rimbey
Posted 1 year ago

Did you try Intersection (but that will lose duplicates--not sure if that's an issue).

Actually, since you generate {} on True in your If, maybe you want to use Complement somehow.

Did you try sorting first and then using an algorithm that short-circuits? Or leveraging any other "structure" in the data?

POSTED BY: Eric Rimbey
Posted 1 year ago

Hi Eric,

The attached file should be enough to allow you to test things out. This example is relatively fast, but still slow enough to get a feel for what the problem is. The goal is to end up with 1 member of each subgroup without a priori knowledge of how many subgroups there are (which rules out the use of Intersection and Complement - the order of items in list needs to be preserved, which neither of those functions do).

Attachments:
POSTED BY: Troy Wahl
Posted 1 year ago

It's taken me awhile to get traction on what your code is trying to do. I haven't made any progress on performance, but I'd like to verify that I've got the semantics right so far.

If you add this definition:

PermRules[list_] := Map[Thread[Rule[list, #]] &, Permutations@list, {-2}]

and then change PowSym to this:

PowSym[vars_] := Flatten[Tuples[PermRules /@ vars], {{1}, {2, 3}}]

do things still work as expected?

POSTED BY: Eric Rimbey
Posted 1 year ago

Hi Eric,

Definitely not

PermRules[list_] := Map[Thread[Rule[list, #]] &, Permutations@list, {-2}]

unless I'm misunderstanding (why just the second to last position, AKA {-2}?). I'm a day into running my code, so I can't test this right now (my list is 5118 rows long, with each symitems containing 11520 rows).

Best thing to do to see what is happening is to add

Print["vars: ", MatrixForm@vars]

after the local variables have been defined,

Print["symelement: ", MatrixForm@symelement]

after "symelement=...;" , and "Print["Result: ",MatrixForm@Result]" between "Remove[...];" and "Result" in PermRules[vars_].

I might be able to do Result=Flatten[Tuples[symelement, 4],1]. In many situations Tuples has caused me lots of problems when the output is large, so I tend to avoid using it.

Thanks for looking into this.
Troy

POSTED BY: Troy Wahl
Posted 1 year ago

Definitely not PermRules[list_]...

That was a helper method I added, you don't need to test it, except indirectly by testing PowSym. What I'm asking is whether my updated version of PowSym produces equivalent results as your original. Does your PowSymmetryCheck1 function, which depends on PowSym, still work if you make the changes I suggested?

Whether Tuples causes problems or not is irrelevant. I'm not proposing Tuples as a performance improvement yet (although I'm almost certain that it would actually be more performant than your Table approach). I'm just verifying that I've correctly reverse-engineered your code.

POSTED BY: Eric Rimbey
Posted 1 year ago

Hi Eric,

I don't know. As I said I'm not at a point where I can test it and won't be able to for a few days at the rate things are running. If you do

vars=Variables@list; 
lvars = {PolyCoefList3 \[Intersection] vars, 
   PolyCoefList4 \[Intersection] vars, 
   PolyPowerList3 \[Intersection] vars, 
   PolyPowerList4 \[Intersection] vars};
symgroup = PowSym[lvars];

with your version of PowSym and get exactly the same result as using my version of PowSym, then PowSymmetryCheck1 will work correctly.

In any event, PowSym works correctly and is acceptably fast as written. In fact, the code is acceptably fast everywhere except for the two spots with

If[MemberQ[symitems, list[[#]]], {}, list[[#]]] & /@ Range[2, len]

and

If[MemberQ[symitems, temp1[[#]]], {}, temp1[[#]]] & /@ 
 Range[i + 1, len]

which I'd like to replace with some sort of parallelized function.

POSTED BY: Troy Wahl
Posted 1 year ago

Okay, well then I'll need some of the "short" examples you mentioned. The notebook you provided takes too long to execute for any reasonable testing.

POSTED BY: Eric Rimbey
Posted 1 year ago

That is the short example.

POSTED BY: Troy Wahl
Posted 1 year ago

Just so I'm clear...

You won't test my refactoring of part of your system with your data to make sure I'm heading in the right direction...

You won't provide a useable set of test data for me to work with...

You won't provide a semantic description of what you're trying to achieve so that I could come up with my own test data...

You expect me to consider one and exactly one category of fix--parallelization...

You expect me to focus on one, and exactly one line of code that is surrounded by a dense mess of inscrutable code...

I don't think I'll be able to help you.

POSTED BY: Eric Rimbey
Posted 1 year ago

Eric, I appreciate your time trying to answer my question, but

..."symitems" are all possible symmetry mappings (which range from S(2)xS(2)xS(2)xS(1) to S(5)xS(5)xS(5)xS(4)) of "list[[1]]."

is the description of the problem, which was included in my original post. I also said I only need one member of each subgroup. If you do not know what that means, you could have asked for clarification. Elsewhere I create raw data, "list," that can contain multiple members of subgroups when I only need one from each subgroup. S(n) is the n-permutation group, in this case the permutation group mapping a set of labels within itself. The fact that I have the product of 4 symmetry groups means that I have 4 sets of labels. As for what "list" represents, that is immaterial to the issue I wish help with.

I did not say that I would not test your ideas, just that I was not at a point where I could do it now. When I'm at a breaking point, I will try the Tuples approach since built-in code is always faster than interpreted code.

I asked a specific question regarding parallelization in a line of code because that line of code is the slowest part of the process and, to me, there is no obvious reason why parallelization is resulting in a considerably slower process than the serial process. That means I've already gone through the code, identified the bottleneck, and considered alternatives (besides re-ordering lists, both intersection and complement also change their length which is a serious problem when one is running a loop based on how long a list is).

I've tried to indulge your process for approaching problems while biting my tongue for your arrogance of second guessing me. This is not the first time I've had someone in this community insist on solving every other problem but what I asked for, and it is very annoying.

POSTED BY: Troy Wahl
Posted 1 year ago

I have verified that the suggested Tuples approach does not produce the desired result. To produce the same result as my table approach (which is all ORDERED combinations of the parts) one would have to do something along the lines of

Union@Sort@#&/@Flatten[Tuples[list,4],3]

which is much less efficient than my table.

POSTED BY: Troy Wahl
Posted 1 year ago

I solved the problem, though it took a complex solution. In the end I got a method that was 2 at least twice as fast as the original serial version and >10x the parallel one.

POSTED BY: Troy Wahl
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract