Failure running large batch run with Mathematica/SystemModeler

Questions
, , ,
1

I am doing a large batch run of SystemModeler simulations using Mathematica to map out a surface. For each simulation run I am pulling only 3 scalar values and storing them in a data file before going to the next run. I am using a Do loop structure to call the simulation, pull the data of interest and save to file. I need to run about 50000 simulations to map out a space, so the batch may take a long time to run.

I posted last week about one problem I was having at first with temporary file space, which I was able to fix with:

WSMLinkLibraryClearTemporaryFiles[-1]

However, now I having stability problems with Mathematic pertaining to dynamic updates. The batch run will chug along fine for a few hours, but then throw an error seemingly randomly about dynamic update taking too long, I then have to kill the kernal and start the run again where it left off. Or, in a few instances, a couple hours into the run Mathematica just is locked up and I have to kill the Windows process completly before I re-start Mathematic and start my run again.

I have also had this problem in the past (last year) doing batch runs of SystemModeler through Mathematica. I can not find any logical reason why these problems are happening, but it is very frustrating to have to keep re-starting my code.

Does anyone have any suggestions?

2

Here are a few suggestions/guesses:

  • Try setting the history length to 0:

    $HistoryLength = 0

  • Turn off dynamic updating for your notebook in Evaluation -> Dynamic Updating Enabled.

  • Turn off the dynamic progress monitor for simulations in your call to WSMSimulate:

      WSMSimulate["YourModel", WSMProgressMonitor -> False]

It would be interesting to know if any of these processes take a lot of memory and/or CPU when the hanging occurs:

  • WolframKernel
  • Mathematica
  • SessionManager (the link between Mathematica and SystemModeler)
  • WSMKernelX (the kernel for SystemModeler)
3

Thanks for the help, but it is still failing to run over large batches. I did all the suggested options, and I even migrated everything over into a wolfram language script, with no luck.

All I am doing is using a DO loop to run a few simulations to find a point of interest, then saving a couple of values from the simulation in a data file, then move to the next operating point of interest. I am not saving anything in memory (at least that I am aware of). However, after about an hour of runs it stops and says my computer is out of memory.

Any further suggestions?

4

Can you see which process is consuming all the memory?

5

If you are comfortable sharing your model and notebook, you could also send them to support@wolfram.com and we can take a closer look to narrow down what is eating up all the memory.

6

The process “Wolfram Kernel for Windows” is what is eating up memory. It grows about 10MB with every Do loop.

The process “WSMKernelX” stays at a level memory usage.

I can not share the model, but any suggestions would be appreciated.

7

I can also observe a small memory increase on running a simulation, but nothing on the order of 10MB.
Here are some more suggestions:

  • Clear the system cache in each iteration
  • Clear your variables in each iteration

So something like this:

    $HistoryLength = 0;
    Do[
     m1 = MemoryInUse[];
     s = WSMSimulate["IntroductoryExamples.HelloWorld"];
     v = s["x"];
     (* do something with the result v *)
     ClearAll[v, s];
     ClearSystemCache[];
     m2 = MemoryInUse[];
     Print[m2 - m1];
     , {i, 10}]

You can use the MemoryInUse and Print to monitor the memory usage.

If that does not improve anything, would you be willing to share your Do loop?