Hi,

I have a looped function, which includes a Map operation, that I've optimized for serial computation whose computation time grows roughly proportional to (N^2+N)/2, where N is the length of the loop. The bottle neck happens at the Map operation, so I do

DistributeDefinitions[F, ..., from];
ParallelMap[F[...], from, Method->"CoarsestGrained"];

That works fine except every time I do a new ParallelMap the amount of memory the kernels are using grows until Mathematica crashes (and I have 64 GB of RAM...).

Sadly, I find the documentation on parallel processing in Mathematica to be the poorest part of the documentation, mostly because the examples are far too simple (they are also all numeric - I'm doing symbolic computations) but also because there is no documentation on "ParallelDeveloper" nor the functions it contains, and no discussion on managing subkernel memory.

At the beginning of my program I set HistoryLength = 0; and disable caching (which does not benefit what I'm doing - that's a different discussion). Those two things only effect the main kernel. So would "ParallelEvaluate[$HistoryLength = 0];" set the history length for the subkernels to 0 (if not we need an option somewhere to do it, assuming the subkernels have a history)?

I know I could prevent this problem by quitting the subkernels and relaunching them, but that has a huge overhead cost (~ N AbsoluteTiming[LaunchKernels[]]) and would wipe-out any gains from parallel processing. I have tried "ClearKernels[]" but that doesn't fix the memory problem. So, should I change my code to

ClearDistributedDefinitions[]; DistributeDefinitions[F, ..., from];
ParallelMap[F[...], from, Method->"CoarsestGrained"];

Is there anything else I should try?