Message Boards Message Boards

Is there a more efficient way to import a large number of files?

Alex O'Brien
Posted 7 years ago

I'm working on an image classification machine learning project and have a somewhat large (~800 MiB) training data set. The data set is composed of individual PNG image files organized in directories. Upon trying to import them all

loadedData = ParallelMap[Import, dataFiles]

It takes an extremely long time (10+ minutes), and ends up using 21 GiB of memory. Obviously, something seems to be wrong. Is there a more efficient or more correct way to be loading all these images?
POSTED BY: Alex O'Brien
5 Replies
Sort By:

Also, if you want to train a neural net with those images you can perform an out-of-core training. See this example: http://www.wolfram.com/language/11/neural-networks/out-of-core-image-classification.html?product=language
POSTED BY: Shadi Ashnai
Alex O'Brien
Alex O'Brien
Posted 7 years ago

I'm aware of out-of-core training, but I had expected that it wouldn't be necessary for the size of my data.
POSTED BY: Alex O'Brien

PNG is usually (always?) compressed. When its imported into the WL, the size increases (sometimes quite dramatically), as its stored as an uncompressed array in memory. This is one major advantage of out-of-core training: you can store all your images in a compressed format on disk and only have small batches of uncompressed images in memory at any one time.

One last thing: JPG is a better format for out-of-core learning than PNG, as the Image NetEncoder is faster for this format.
POSTED BY: Sebastian Bodenstein

Try:

loadedData = ParallelMap[Import[#, IncludeMetaInformation -> None] &, dataFiles]
POSTED BY: Piotr Wendykier
Alex O'Brien
Alex O'Brien
Posted 7 years ago

This appears not to have made any (or much of a) difference. I'm not on a machine capable of loading 21 GiB of data at the moment, but it's still using at least 16 GiB before failing. I'm bewildered as to where all of this apparent data is coming from.
POSTED BY: Alex O'Brien
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract