One last suggestion (and then I will stop!):

What if you write an "externalSum" function in C. It would take your arrays and sum them up in C. I've done this a few times when Mathematica was "difficult".

(ie The last time I did this was to write a printf function in C so I could pass variables and a format string and get the exact printout that I wanted. )

Would that work for you? (While your knowledge of Mathematica is impressive, If you have not done C externals in Mathematica before I can send you an example)

Regards

Based on Devendra's explanation, I see the difference:

In[1]:= MaxMemoryUsed[total = Sum[{RandomInteger[10]}, {i, 100000}]]

Out[1]= 8801304

In[2]:= MaxMemoryUsed[total = Sum[RandomInteger[10], {i, 100000}]]

Out[2]= 7528

Sander, could you structure your problem as an array of sums instead of a sum of arrays?

Something like the first (avoiding the second):

In[3]:= MaxMemoryUsed[
 total = Table[Sum[RandomInteger[10], {i, 1000}], 100]]

Out[3]= 10112

In[4]:= MaxMemoryUsed[
 total = Sum[Table[RandomInteger[10], 100], {i, 1000}]]

Out[4]= 1751232

Hope this helps.

Regards

Sum allows very general inputs (numeric, symbolic, etc.) in its first argument. If it detects that the input is numeric, then a direct procedural method (with no storage) is used. At present, it does not check if the first argument is an array and hence does not apply procedural summation without storage of intermediate results, which leads to large memory usage in examples such as the one considered by you. Optimization in this situation would mean checking for arrays, which may be somewhat expensive particularly because Sum is not specially designed for such inputs.

Sum uses the procedural method (adding successive values to obtain the sum) by default for your example,

Unfortunately, at present, the procedural method in Sum is not optimized for adding arrays, and using a Do loop for doing the intermediate summations (as already done by you) is a nice way to work around the limitation.

Thank you for drawing our attention to this issue, and sorry for the inconvenience.

Interesting!

The memory usage is indeed much smaller. However:

Monitor[Sum[s[k], {k, 100}, Method -> "Procedural"], {k, MemoryInUse[]}]
Monitor[Sum[s[k], {k, 100}], {k, MemoryInUse[]}]

both show the same behaviour, is Procedural the default for 'small' k? What does "compute the sum procedurally" actually mean? Whatever the method, it will be done with some procedure...

Consider:

s[k_] := 11111111111^(11111111 + k)

Monitor[Table[{k, s[k]}, {k, 100}], {k, MemoryInUse[]}]

Monitor[Sum[s[k], {k, 100}, Method -> "Procedural"], {k, MemoryInUse[]}]