Get reasonable interactive 3D plots with ~500000 points?

Questions
, ,
1

I need to plot a dataset of about 500000 (x,y,z) points (scattered 3D plot, using ListPointPlot3D and simultaneously Plot3D to compare to a smooth surface). I want to be able to rotate it, etc. In the computers I have access to, the interface starts to get slow at about 50000 points. Example:

n = 50000;
xmax = 50.0;
ymax = 20.0;
x = RandomVariate[UniformDistribution[{0.0, xmax}], n];
y = RandomVariate[UniformDistribution[{0.0, ymax}], n];
z = 30.0 - 15.0*(x + y)/70.0 + RandomVariate[NormalDistribution[0.0, 0.5], n];
Show[{Plot3D[30.0 - 15.0*(x + y)/70.0, {x, 0.0, xmax}, {y, 0.0, ymax}, ImageSize -> 600], ListPointPlot3D[{x, y, z}\[Transpose], ImageSize -> 600]}]

For n=500000, it takes very long to produce a graph and it then freezes if I try to rotate it. I can trim the data (create bins and then take small samples from each bin), but I was looking for ways to optimize the plotting so that it can handle more points without trimming.

Thanks,

OL.

2

I’m unsure i’m new to 11.0 myself

obviously, turn off automatic things you didn’t above (Clipping, Axes, anything that might get evaluated while rotating)

this works pretty smoothly:

Graphics3D[
 Table[Sphere[{Random[], Random[], Random[]}, 0.01], {i, 0, 1000}]]

but it’s 1,000 objects - each object has many “texels” in 3d lingo (each ball has a whole plane of vertices). video games typically have high vertex count but low(er) object count, and objects at a distance are specially obstructed / blurred because: they aren’t fully being rendered.

50,000 objects is allot. I think there is some thing here where your drawing 50,000 objects not a single mesh model with 50,000 vertices (say, a mesh of some object like and airplane) and expecting 50,000 rendered shaded objects to be the same speed as 50,000 vertexes object who’s surface is shaded 1x. rotating the two would not be the same “job”.

i see the same thing you do but usure if that is “blazing fast” or “slow and un-optomized” (if it re-evaluates all per degree of rotation to avoid storing sets of points in memory?). i am pretty sure that any slowness is like mm adding things like Axes or other extras who’s viewing quality are preffered to flat out speed.

interesting question

3

Unsure if mathematica does this, but (true) OpenGL cards (likely any 3d card) support “sphere”. That’s not off topic because each sphere is a mesh generated by the video card (which infact can be generated per frame, or held as a set of points, even altered, and rotated). If you generate a ball mesh you’ll quickly see you need about 100 vertices to make it a ball shape - more like 1000 to make it nice sphere appearance.

In Mathematica 4.0 (excuse me i’m more familiar with it), I find Sphere… PlotPoints->50 a believable ball shape, 2400 vertexes, so 50k balls is more like over 2.4 x 5 x 10^7, vertexes minimum, well over 100 million.

Below is 50,000 objects that rotates like butter. (use triangle shapes for more speed is a 3d lesson)

Graphics3D[
Table[Pyramid[{{Random[], 0, 0}, {0, Random[], 0}, {-Random[], 0,
0}, {0, -Random, 0}, {0, 0, Random}} + i], {i, 1, 50000}]]

you’ll need to run it at 50 to see they are actaully pyramids

at any rate: rotating 50k objects will not be same as rotating a mesh with 50k points - be sure of that. (old true all-in-silicon opengl did just as well with polygon as pyramids, but gaming cards today are not optimized that way and do much better with pyramids, last i read about it)

4

meanwhile, rendering “points” is not a 3d card specialty last i read (is not a gl command or card feature). the points have to be made into (squares) facing toward you with shading turned off and etc so they all appear the same size at any distance in the scene. You can expect boxes to render faster, though you’d think they have more vertices. points may not be the “minimal drawning object” your thinking it is. pyramids are, or triangles facing the viewpoint - or such shape made from a low number of triangles that are box like seen from any viewpoint. they will be small or invisible when distant in the scene, of course, being 3d. the points are kind of special 2d plot in 3d if i am seeing correctly - and if i remember correctly that’s how Mathematica 4.0 did “point” and other 3D objects in Graphics3D - they were specially made to be “not 3d” and took extra processing to do so.

with my Pyramid example above I can do 100,000 but 500,000 crashed Mathematica. but you can’t see 100k pyramids in a scene anyway - they appear to be a line, and thus could be rendered easier as a line. my 2nd attempt succeeded (500k pyramids) though rendering as a few frames per second at best - but usable.

i really think you want to try something else (not points) - yet i know 5M of any 3D object (ie, box) wont be visible it’s an awful lot of objects (not vertices, objects) to render

A PICTURE OF WHAT YOUR SEEKING TO HAVE ROTATED REAL TIME would be helpful/necessary for any further talking points

(ie, something in google images that looks similar to what you’d like to see)

5

Even if you could smoothly rotate, or zoom, on a plot with 500,000 points it seems to me that the plot would be so dense that it would be difficult to extract information from it. One wouldn’t even be able to see the surface. And what, precisely, is the information that should be extracted?

One technique in a case like this is to use multiple plots, or better multiple presentations in a dynamic display. One thing you could do is to take smaller random samples of the 500,000 points and display those. (Like 10000 points.) If you are interested in detailed behavior around some particular {x,y} position you could just select all points in a small domain and display those, effectively zooming in. You could have an overall plot, with a random sample of the data, and a zoomed plot next to it with the domain size and location controlled by sliders. You could use a Button to generate new random samples.

You have only a limited number of pixels available. But multiple presentations can do wonders.

6

I’m running your example with 500K point and I can still rotate it albeit not fluid (I have a laptop).
Windows 10 MM10.2 executes in 27sec your code (with 500000 point)

7

Thanks to all that have responded. If you notice, I am trying to see just points ({x,y,z} triples), not extended objects. In my understanding that should be faster than trying to render the same amount of any extended objects that take more information than points, but that it only intuition; I am no expert in graphics algorithms.

You can see something similar to what I want to rotate if you just run the code I included in the original posting using n=1000. In my case, I need something closer to n=500000 (half a million). I tried the table of pyramids. It took a little while to produce, but rotated OK. My list of 50000 points also rotates more or less OK, it is with half a million points that it won’t rotate (takes about 19 sec. to create).

It is true that the half a million points form a dense cloud, but in my case I still should be able to see the surface because in reality, my surface is slightly below the cloud. Also my real surface has a little bit of ups and downs (it is not really flat) and I want to be able to judge visually how well these ups and downs correspond to the surface. I think my real cloud of points has a pretty sharply defined envelop.

I think the idea of creating bins (using, say, a 70 by 70 evenly spaced mesh) and taking the min and max in each bin can give me a good idea of what I am trying to see. Being able to see the whole thing would be nicer.

I am using Mathematica 11.0.0 Student edition on an hp Pavilion (64 bit) running ubuntu 15.10. It runs two cpu cores (I think it is an AMD i5 processor). I get similar results at work though, using version 9 on a different machine (also on Linux).

Thanks,

OL.

8

If your data is generated from a known random process what good is showing 500,000 sample points as opposed to 5,000? (I’m not suggesting there isn’t a reason for doing so. I’m only stating that it is not obvious.) If the data generating process is completely known, then displaying the surface that contains, say, 95% of the probability (with some appropriate restrictions such as including the (x,y,z) locations with the highest probability density values) might be desirable. (For example, in a two-dimensional case with a bivariate normal, then showing the elliptical contours containing various probabilities is common.)

So my question is why stick with just displaying the data points when with 500,000 sample points, you could estimate the distributional parameters and then display the underlying constant probability density surfaces?

9

Jim,

Thanks for your reply. I understand your argument. Trying to easily visualize the half-a-million points simply came up as something that would be helpful, and is really the tip of the iceberg.

The data do not quite represent a random process. They are the results of calculations of the behavior of a system using many different inputs for the starting conditions. The code I included in my post was really and example of a similar set of points. If our calculations really produced random results, then we would have a serious problem. In a sense, the details of the data generating process is not completely known (it is inside a complex code).

Questions come up when the surface has wiggles that do not seem real, or when the calculation of the system behavior seems to yield unreasonable low or high values for specific input values. As I said, I can get by with using the min’s and max’s in some bins, but the possibility of pushing the limit of the graphic interface seemed worth exploring.

There is nothing special for my problem about half-a-million. In some analyses a have significantly fewer points, in others, significantly more. Half-a-million seemed to be the point where the graph, the way I produce it, becomes too slow for interactive use. I thought there could be ways to push that limit. Thinking of RAM and CPU (at the current levels) half-a-million points does not seem like a whole lot. I think the key is the speed of interactively updating the rendering (user input, kernel, graphics card, display). But I am no expert in these matters, so I could be wrong. I can say that I have noticed the following about the speed of displaying an output:

  1. If one produces a very large textual output (a large matrix, or text, etc.), then Mathematica will be finished with the calculation long before the result is printed to the screen (of course, normally I do not print those to the screen).

  2. If one produces a 2D ListPlot with many data points (say ~15000), even if it is only one dataset, then using PlotMarkers->Automatic delays the display and even produces that dialog window that says that a very large output was produced and displays a part of it (although it displays the entire graph). The cause seems to be that with PlotMarkers->Automatic, Mathematica uses a different set of markers (they look slightly larger) than when no option is given.

In any event, it is clear that the ability to interactively rotate a graph must break at some point. I just thought that with a little more inside knowledge that limit could be pushed.

Thanks,

OL.

10

Good. I ask because much of the time data is used to estimate parameters of some specified model (even if the model is somewhat of a black box). Then one can make inferences about additional data that might be collected or even just to summarize the data.

Depending on the objective of the data collection process (and I’m not talking about the objective of the responsiveness of a 3D interactive figure with individual data points as opposed to the 3D figure) you might consider either 3D nonparametric density estimation and drawing surfaces of constant probability density or if there aren’t too many bumps in the data, 3D convex hulls. Again, these are showing 3D surfaces that might be more responsive as a 3D figure and still get across the message. (Or just plot the smaller number of points just outside a surface that contains a large fraction of the sample points. In other words the surface would help delineate the more “extreme” values.

11

As others said, are you sure that you need to visualize that many points interactively? Does that picture really tell you more than what you could see in data that has been reduced in some way?

However, if you really need many points, then it is generally a good idea to construct your own graphics objects with as few primitives as you can. Plotting functions are more flexible and make it easier to make the plot pretty. But won’t perform as well.

Thus with

pts = RandomVariate[NormalDistribution[], {500000, 3}];

the following may perform better:

 Graphics3D[{AbsolutePointSize[1], Point[pts]}]

than any of the following two:

Graphics3D[{AbsolutePointSize[1], Point /@ pts}]
ListPointPlot3D[pts]

This is just a general performance tip, not a full solution to your problem. The key is to put all coordinates in a single graphics primitive (Point, Line, etc.), when supported.

12

oh. then you want a visual of two 3d planes (in which case use a 3d mesh not points, 2D “ledger dots” are slower), or use 3d (triangles) for both planes

or: plot the difference between the two instead, perhaps coloring to show thickness, which will be much easier to view

what your trying to do is not supposed to be efficient, which i tried to say above in previous replies