Hi,
I’m running v13.1 on an Intel+Windows system with a reasonably fast SSD. My current application generates an ordered List of 250M to 300M integers which I would like to store for later use. The values are all in the range 0 to 30. In total I will have 30 to 100 of these files. Presently it takes 45 minutes of elapsed time to Export one of these lists to a .TSV File, which ends up being about 800MB in size. Read time is acceptable at ~12 seconds. What alternative Export format(s) might provide better storage performance?
Do you only need to access this from Mathematica, then *.mx using DumpSave is an option.
Much better!
Export .mx.gz wrote 133MB file in 1 minute.
Thank you.
Richard,
you could use NumericArray to reduce storage and then export the list with DumpSave.
byteArray =
NumericArray[RandomInteger[{0, 30}, 300 * 10^6], "UnsignedInteger8"]
ByteCount[byteArray]
Timing[DumpSave["test.mx", byteArray]]
On my machine the 300M Integers are written in no time to the HDD and the size of the mx file is less than 300 MByte (file size was corrected). See Richards table