Message Boards Message Boards

Best practices for dealing with very large files?

I am wondering what are the best practices in the Wolfram Language to deal with very large files?

My original question had to do with processing large FASTQ files, but it is more general.

Imagine a file that is way larger than available memory. I get the impression that the Import parsers cannot directly deal with such files. I see two kinds of solutions:

  1. I design a parser from scratch that can deal with large files, which seems like a total waste of time
  2. I do something around the lines proposed by Jason on my original question. That works but it is not totally elegant: it mostly seems like a very clever hack around an infrastructure that was not thought for very large files from the start

I was wondering if there are any best practices that do not require writing work arounds to the core libraries?

Many Thanks

POSTED BY: Tiago Antao
2 Replies

Hello Tiago,

have you tried ImageFileApply and friends? From the documentation:

ImageFileApply can be used to read, process, and write successive blocks of data from a very large image file whose size could exceed available memory.

Maybe this helps a bit, regards -- Henrik

POSTED BY: Henrik Schachner
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract