Message Boards Message Boards

Tips for preventing memory issues with Graphics3D

Quadro M1200
Intel i7
32 Gb RAM

I'm currently rendering some large 3D scenes for some projects. I'm aware I'm pushing my hardware and Mathematica to its limits. I previously asked some questions about frontend crashes when using Graphics3D and export and I now better understand why and can work around it.

I am managing to get the results I want by closely monitoring my system when pushing the settings just below failure. But I'm wondering if some of the issues I'm working around can be prevented. Therefore I'm posting my experience here and hoping to get some tips to improve.

But first the nice results before the boring evaluation.

enter image description here enter image description here

My laptop is a Intel i7 with 32 Gb RAM and and Nvidia Quadro M1200. for the graphs below red is memory usage and green is CPU usage both in %. For my scene I have 60 transparent iso-volumes and 4 solid ones. The streamlines I render are subsampled from around 500K lines i generate. For lines I select 12.5k lines with around 100 vertices each, for tubes I select 7.5 k lines. My $HistoryLength setting is always 0 since I'm working with sizable data.

When exporting my scene using lines this is my system memory. I don't display the scene in the frontend only export it to *.png files. The export is clearly handled by the frontend and CPU and at the moment of export the memory spikes and the GPU does some work. However after everything is done and i quit the kernel the Mathematica frontend is still using between 3-4 Gb of RAM. Exporting one PNG takes around 2-3 Gb.

enter image description here

When exporting tubes things get a bit more heavy and needs between 6-8 Gb of memory per image. after everything is done i have lost around 5 Gb of RAM that I can only get back by closing the frontend.

For a clean system and Mathematica start i use around 25% memory. After loading and processing i have 35% in use. After one round of exporting lines and tubes, closing Mathematica reopening loading and processing the data I'm at 50% memory usage so somewhere I lost 15%.

enter image description here

If I push everything to far the export will overflow the memory resulting in a frontend crash and sometimes a GPU driver crash. I now somewhat know the limits of what I can do but after working for a while exporting and optimizing visualization at some point my memory keeps growing. I now monitor this and when i come to the point of getting close to a crash the only thing i can do is restart everything to reset my memory.

POSTED BY: Martijn Froeling
9 Replies
Sort By:
Yuzhu Lu
Yuzhu Lu, Wolfram
Posted 11 months ago

In case of you have not try this yet, you can customize "TubePoints", CapForm and JoinForm to reduce the FrontEnd memory usage. By default value of "TubePoints" method option is {40, 2}, and the default "Round" CapForm and JoinForm involves more sampling. Since you have a lot of small tubes and reducing sampling won't affect too much your visual results, you definitely should give it a try. 

Graphics3D[{Tube[RandomReal[.1, {50, 50, 3}], 
   VertexColors -> RandomReal[1, {50, 50, 3}]]}]

Graphics3D[{JoinForm["Miter"], CapForm["Butt"], 
  Tube[RandomReal[.1, {50, 50, 3}], 
   VertexColors -> RandomReal[1, {50, 50, 3}]]}, 
 Method -> {"TubePoints" -> {5, 2}}]
POSTED BY: Yuzhu Lu

Hi Yuzhu,

I was actually already using these exact settings, even with "TubePoints"->{3,2} the visualization still looks OK for my application. I understand tube rendering takes much more memory and effort than rendering lines, and a few GB of memory are fine. The huge memory spike just before rendering is the thing that causes the FrontEnd crashes, without that happening i could probably easily increase the number of tubes to be rendered.

Thanks for looking into it!

PS from the my test notebook that can be found here, these are the settings i used for the tube rendering. enter image description here
POSTED BY: Martijn Froeling
Yuzhu Lu
Yuzhu Lu, Wolfram
Posted 11 months ago

A new method will be added to use less memory but slower tube rendering for the next release. Please contact us for details.
POSTED BY: Yuzhu Lu

Hi,

That would be awesome. I will patiently await the update. For now, I know that what I'm doing is indeed the optimal way and know where the limitations lie. Compared to a few years ago the visualizations have already come a long way for me. Thanks for the effort!
POSTED BY: Martijn Froeling

Hi Yuzhu,

Finally, I came around to testing the performance of my tube renders in v 14.0. As you promised the memory use has decreased substantially. Before I had crashes rendering scenes of 16MB with around 7.5K tubes I can now render way larger scenes. I have not seen where it crashes now but the render below is with 15k tubes which is more than I need now. enter image description here
POSTED BY: Martijn Froeling
Yuzhu Lu
Yuzhu Lu, Wolfram
Posted 5 months ago

Hi Martijn,

Thanks very much for confirming the improvement.
POSTED BY: Yuzhu Lu

If there is no way to reclaim memory in your main session, have you considered using the parallel computation framework to do these computations in a separate kernel/frontend?

For example, configure one parallel kernel and then do:

LaunchKernels[] (* to launch a single kernel *)
ParallelEvaluate[ (* run the code to do one rasterization/export *) ]
CloseKernels[] (* to free up the memory *)

Not perfect, but maybe allows better control over the memory issues you are running into?
POSTED BY: Arnoud Buzing

I was giving this some extra thought. But in this case it wont work since its the frontend that does the conversion of GraphicsPrimatives to actual 3D objects. During the entire process the kernel memory does not change and the kernels hardly do any work. The frontend is doing the heavy lifting (therefore it becomes unresponsive with a nice blue waiting circle during most of the processing time)
POSTED BY: Martijn Froeling

I made some examples where i can reproduce the memory thing happening with tubes by storing the graphics in Iconized form. The notebook is to large to upload here so you can find it here

The Graphics3D for both tube and line rendering are 16MB in the notebook. However to render the tubes the Mathematica frontend temporarily uses up to 12 Gb or memory. Displaying both and removing them from the frontend and then quitting kernel made 1.5 Gb of memory not recover.

enter image description here

enter image description here enter image description here

I also tried it in the cloud, the lines would render, the tubes would not.

POSTED BY: Martijn Froeling
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract