I have a 2.4GB file containing xyz and RGB values from a laser scan survey. I want to bring it into Mathematica and plot in 3D, draw sections etc. I am aware that the data set is huge and can cut it down by sampling the data or by just plotting parts of the survey at one time. However, I first need to get the data into Mathematica and this is taking forever. It is currently in a .dat format. Would another format be quicker? or are there any tips to speed the process up? I can open the file in a text editor ok. Once I get it into Mathematica are there any tips as to how to deal with it e.g. what is the largest practical size data file for plotting at 3D etc.? Would the file take too long to sort? At the moment the data looks rather random which makes abstracting parts of it rather tedious. Any help would be welcome. I am running on an iMac with 16GB of Ram and a 3.4 GHz I7 processor
I have resampled again and got down to 50,000 points which it could handle, with some crashes. Strangely though, I was able to rotate the view quite well once it was formed. Thanks for your help.
Unfortunately, the x and y coordinates seem to be at random rather than on a regular grid.
Interpolating this to get a surface (that is what listplot3d does, but listpointplot3d doesn't) is quite complicated and takes a lot of time with millions of points, i'm not sure if Mathematica will be able to handle it. Interpolating your data on a rectangular grid will make plotting this possible, i guess you don't really need 48 million datapoints to visualize your surface? right? i gues 10^4-10^5 should be enough to get an idea of what the data looks like...
is your data on a regular grid? if yes: no do no supply the x and y data per point, rather just give an array of heights, and then DataRange to set the x and y values. If you supply x,y,z triplets it will do so-called delaunay triangulation on all your point which is very slow...
Sander. Thank you. However, when I tried to save the file, having read it in after a couple of hours, in binary, I got a message to say that it could not force the file to be binary. The file turned out to be 48 million records of 4 numbers, I resampled the data to get the number down below 1million, which it did quickly, and tried plotting that. ListPointPlot3D worked but when I tried ListPlot3D it eventually crashed. I would like to get a 3D view that I could the rotate and look at from different angles. Is there a more efficient way?
With large amount of datapoints I found that importing it as binary files is the fastest, probably mainly because it can directly read it from the files and put it directly in the memory.
For using the dat files you can read it in many different ways:
Going from SemanticImport, to Import, to ReadList you go from slow, normal, to fast. But they have their differences with respect to robustness.
In the end it really depends also on WHAT you read. a 2.4GB file is quite hefty, but the data might be much smaller when stored binary...