I work in microscopy I currently use Mathematica to process images small scale. However recently are data set have gotten really big, On the order 100s of Gigs to a few Tbs per run. I am looked into CUDA for palatalization of doing things like FFT, Convolutions, Wiener Filtering of images. I would like to interface with Mathematica and I know from doc about CUDA link.
So the real question here does mathematica have good set of tools for dealing with data sets too big to fit into memory. If so is there any guide on how do this properly. Ideal Mathematica would perform some basic calculations but a lot of the heavy lifting would be crunched in on GPUs through CUDA link. Visualization of the combined images would fit into memory after processing and be done by Mathematica. Any help or guidance would great.
Your problem seems to be a problem that will occur more and more. I heard during a wolfram event last year in the Netherlands a presenter say that Wolfram is looking into a streaming technique which would allow to compute on large image files. This would be introduced somewhere in v11. Perhaps if you don't get an answer here you could try the mathematica stackexchange site or log a call @ wolfram support?
I admit that I do not have any experience with this, but isn't Out-of-Core Image Processing (e.g. ImageFileApply and friends, introduced already in v9) to some extend what you are looking for?
Regards -- Henrik
I looked into out of core processing. I don't think it will do what I need. To give a little more detail these images are stacks of images say 20 to 50 frames per image (intermediate frames in a composite image) with the new cameras now each intermediate frame is 8k 8bit we have went from processing 10s or low hundreds of these stacks (at 2 or 4 k) to 1000's (at 8k). Great for data but presents serious problem in processing ha ha. In the these are filtered and added up to give 3d electron density maps. One of thing that makes it difficult is that the container file is not natively supported by mathematica. You have to read it in as byte code and then separate metadata from the body data and so on. So for instance I use to be able to import the a whole stack into memory no they are 60 or so gigs per stack. So i need good tools for dividing the data into chunks/so i can stream it through. Obviously this no trivial task when talk about this amount of data. Any input or guides to processing this much data wih mathematica would be great. The image processing it self I can do (Cuda C/C++) . However managing this much data takes tom expertise I don't have. I need to know where to look.
in principle, Henrik's tip is right on target. We implemented the commands ImageFileApply, ImageFileFilter, and ImageFileScan to allow for out-of-core image processing. The limiting factor for you may be the small set of supported image file types (e.g. TIFF).
Hence, the first idea would be to convert your data to one of the supported file types and then use the out-of-core capability of Mathematica.
If that does not work, the second approach would be an InputStream with which you could read and partition the incoming data into frames and process them one at a time. I believe 8k frame should easily fit into your memory. Mathematica has superb pattern matching capabilities to facilitate the parsing of any (binary) file format at hand.
Last but not least, I like to mention, that Mathematica has some built-in CUDA / OpenCL functions, such as CUDAFourier, CUDAImageConvolve, etc.. Hence, it is no alway necessary to link to you own CUDA / OpenCL kernels.
For further information, please look up the ref-pages of the functions mentioned above.