The Write command works well, a good alternative to PutAppend.

Thanks, John.

Greg

Greg,

You should be able to handle 10 million x 30 arrays in Mathematica without breaking them up. I tried:

In[8]:= Timing[bigarray = RandomReal[1, {10000000, 30}];]

Out[8]= {2.75571, Null}

It only took 2.7 seconds to generate a random array of that size and Mathematica was fine with it. There are several things you can do to optimize this. 1. do not print the arrays -- use a semicolon so they are not displayed. 2. Stay away from Do and other looping constructs. Use the list functions such as Table, Map, etc. 3. Write the data (whether or not you break it up) into a binary file such as .mat or use the Binary read and write functions.

Note if you want to run your functions over subsets of the bigarray, you can do that by using the Part functionality and still keep the array as one big array for writing to your file later. For example

bigarray[[1 ;; 5]] = RandomReal[10, {5, 30}]

Will replace the first 5x30 array elements with new numbers ranging from 0 to 10. This is done in place so you still have one big array but can process it in "chunks".

I hope this helps.

Regards,

Neil

Hi Gregory,

You need a minor change in the code for achieving the desired result:

SeedRandom[666]
m1 = RandomReal[1, {4, 3}]
m2 = RandomChoice[CharacterRange["A", "Z"], {4, 3}]
m3 = RandomInteger[100, {4, 3}]
m = Sequence[m1, m2, m3];

tmpFile = FileNameJoin[{dumpPath, "tmp.m"}]

PutAppend[m, tmpFile] 

readData = ReadList[tmpFile]

Now readData contains already flattened array. Note that the correct extensions for files containing Mathematica expressions intended for loading using Get, ReadList or Import are .m and .wl, the extension .nb is for Mathematica notebooks.