Group Abstract Group Abstract

Message Boards Message Boards

Decrease a Dataset size in RAM?

Andrea Barresi
Posted 8 years ago

Hi, I have a question about the size of Dataset in RAM. I have noticed that if i have a file .mx that contain a Dataset and the size on the HDD is for example 50 MB, when i import it in Mathematica the size in the RAM is about 10 times (500 MB)!

Do you know why there is so many difference between the two sizes? Is it normal?

I have noticed that if i save the same data as List and not as Dataset (without association) the size is just greater than the size in HDD. Is there any solution that allows using the dataset without this "waste" of memory", because it is a big problem when i have to deal with a large amount of data... but the datasets are more "comfortable" than pure List.

Thank you for your help!
POSTED BY: Andrea Barresi
3 Replies
Sort By:
Andrea Barresi
Andrea Barresi
Posted 8 years ago

It can't be the history, because i see this "anomalous" usage of memory as i import the data. Furthermore, if i apply the function ByteCount to the lists, it gives me two different results, as i wrote in the previous post, and it doesn't depend on the history...

There is also the usage of RAM during export that is different between the two cases.

If you want, i can attach an example file to explain better the problem...
POSTED BY: Andrea Barresi

Andrea,

Its likely your history. Mathematica stores all previous results (i.e. ...Out[12], Out[13], Out[14]...) Your intermediate text results are very large. Turn off History with 

$HistoryLength = 0;

or if you want to use % to refer to the previous result and not store it in a variable.

$HistoryLength = 1;

Now you may also have to clear old variables you no longer need to free up the memory.

You can see information about this in Memory Measurement and Optimization

Regards,

Neil
POSTED BY: Neil Singer
Andrea Barresi
Andrea Barresi
Posted 8 years ago
POSTED BY: Andrea Barresi
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard