I'm running a lot of loops on big data sets and need to conserve as much memory as possible. Each big loop creates a big file or results. I have set
$HistoryLength = 1;
but am wondering whether there is a way to clear all history of previous computations between runs of a loop without clearing variable definitions. Any tips?
I'm not yet a Wolfram Language expert, but if the Wolfram Language works like other languages that I've used, then you should might be able to use Block, With, and Module to scope your variables that require lots of memory.
This post might help.
Thank you for replying to my question.
I make regular use of Block, With, and Module. The slowdowns in processing many big files have to do with Mathematica accumulating a history of past computations. This happens even with Block or other scoping mechanisms if they contain loops.
A Wolfram tech support rep suggested that I throw in ClearSystemCache[ ], and that has resulted in a marked improvement in speed.