Is there a way to cap the amount of memory parallel Mathematica uses per Kernel? I am running a computation using ParallelDo where if I activate even 70% of the total Kernels in my computer, the code uses up 100% of my computers memory. Using MemoryConstrained and ClearSystemCache has not helped. I have my calculation wrapped up as in
MemoryConstrained[ParallelDo[ Working code],Byte specification]
It did not work. The ClearSystemCache is withing the ParalleldDo code. I am using Mathematica 12.1. Any suggestions? Thanks.
Try
ParallelDo[ MemoryConstrained[Working code,Byte specification]]
Putting MemoryContrained around the outside is only constraining the control kernel’s memory use. Since the work is all being done on the compute kernels, I expect that is where you want to apply the limits.
Jon, Thanks for the suggestion. But, Mathematica quits the calculation if this memory constrained is reached. If I track my CPU Memory usage (Windows 10), it has a see-saw profile before finally quitting on me. How do I sustain the calculation? Can I place $HistoryLength somewhere? I would appreciate any suggestions. Thanks.
Yes, memory constrained can only abort the calculation.
Without seeing the code its hard to give advice, but in the end you need to make sure 1) make sure you are not accumulating data unecessarily ($HistoryLength=0 only helps in the control notebook. But making sure that you clear variables when finished with them, write data to disk if you need it later rather than accumulate in memory etc), 2) Make sure single calculations are not too big (you can use MemoryAvailable[] to plan the size of a calculation, although that is global so each kernel can only acquire a share of it).