I am wondering what are the best practices in the Wolfram Language to deal with very large files?
My original question had to do with processing large FASTQ files, but it is more general.
Imagine a file that is way larger than available memory. I get the impression that the Import parsers cannot directly deal with such files. I see two kinds of solutions:
- I design a parser from scratch that can deal with large files, which seems like a total waste of time
- I do something around the lines proposed by Jason on my original question. That works but it is not totally elegant: it mostly seems like a very clever hack around an infrastructure that was not thought for very large files from the start
I was wondering if there are any best practices that do not require writing work arounds to the core libraries?
Many Thanks