Depending on how you did the calculation it might be a good idea to set the $HistoryLength:

$HistoryLength = 1

I don't think Mathematica will decide if it stores things in the regular memory or the virtual memory, that should be the task of the operating system.

Basically it is the storage of Out[...]. Remember that with % you can get the last answer and %% the penultimate and %%% the antepenultimate et cetera... This is only possible by having this 'history' in memory. $HistoryLength is by default set to infinity meaning it stores all these outputs. If you don't use it, it might take up lots of memory. However it depends a lot on how you wrote your script...