Message Boards Message Boards

0
|
4472 Views
|
5 Replies
|
0 Total Likes
View groups...
Share
Share this post:

Preventing Parallel Mathematica from using up 100% computer memory

Posted 2 years ago

Hello,

Is there a way to cap the amount of memory parallel Mathematica uses per Kernel? I am running a computation using ParallelDo where if I activate even 70% of the total Kernels in my computer, the code uses up 100% of my computers memory. Using MemoryConstrained and ClearSystemCache has not helped. I have my calculation wrapped up as in

MemoryConstrained[ParallelDo[ Working code],Byte specification]

It did not work. The ClearSystemCache is withing the ParalleldDo code. I am using Mathematica 12.1. Any suggestions? Thanks.

POSTED BY: Mag Phys
5 Replies

Sure you can reach out to me at https://jon.mcloone.info/contact.html I won't be able to spend any real time on it, but if I see any obvious things to do I will let you know.

POSTED BY: Jon McLoone
Posted 2 years ago

Thanks. Would it be possible to share the code privately to discuss? I cannot make the code public. Please let me know if this is a possibility.

POSTED BY: Mag Phys

Yes, memory constrained can only abort the calculation.

Without seeing the code its hard to give advice, but in the end you need to make sure 1) make sure you are not accumulating data unecessarily ($HistoryLength=0 only helps in the control notebook. But making sure that you clear variables when finished with them, write data to disk if you need it later rather than accumulate in memory etc), 2) Make sure single calculations are not too big (you can use MemoryAvailable[] to plan the size of a calculation, although that is global so each kernel can only acquire a share of it).

POSTED BY: Jon McLoone
Posted 2 years ago

Jon, Thanks for the suggestion. But, Mathematica quits the calculation if this memory constrained is reached. If I track my CPU Memory usage (Windows 10), it has a see-saw profile before finally quitting on me. How do I sustain the calculation? Can I place $HistoryLength somewhere? I would appreciate any suggestions. Thanks.

POSTED BY: Mag Phys

Try ParallelDo[ MemoryConstrained[Working code,Byte specification]] Putting MemoryContrained around the outside is only constraining the control kernel's memory use. Since the work is all being done on the compute kernels, I expect that is where you want to apply the limits.

POSTED BY: Jon McLoone
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract