Best practices for dealing with very large files?

Questions
, ,
1

I am wondering what are the best practices in the Wolfram Language to deal with very large files?

My original question had to do with processing large FASTQ files , but it is more general.

Imagine a file that is way larger than available memory. I get the impression that the Import parsers cannot directly deal with such files. I see two kinds of solutions:

  1. I design a parser from scratch that can deal with large files, which seems like a total waste of time
  2. I do something around the lines proposed by Jason on my original question. That works but it is not totally elegant: it mostly seems like a very clever hack around an infrastructure that was not thought for very large files from the start

I was wondering if there are any best practices that do not require writing work arounds to the core libraries?

Many Thanks

2

Hello Tiago,

have you tried ImageFileApply and friends?
From the documentation:

ImageFileApply can be used to read, process, and write successive blocks of data from a very large image file whose size could exceed available memory.

Maybe this helps a bit, regards – Henrik

3

Have you seen this discussion on StackExchange?

https://mathematica.stackexchange.com/questions/15206/reading-periodic-elements-from-a-large-file