David K.'s post seems to be a better description to the problem.
I have googled combinations of a number of keywords from this discussion (e.g. memory problem, Mathamtica, pagefile, RAM, clear, memoryinuse, etc), and got some results resembling my sitatuation.
Some of the most relevant:http://forums.wolfram.com/mathgroup/archive/2009/Jun/msg00275.htmlhttp://www.tomshardware.co.uk/forum/15101-63-pagefile-hiberfil-gigantic-filling
So it seems that Mathematica indeed uses my virtual memory. And a clarifying sentence: "A paging file is an area on the hard disk that Windows uses as if it were RAM".
Let it be that Mathematica will use my Hard Drive, but then my question is, how to clear this trash? ..because it seems to keep it even after I reboot my laptop and until the system realizes that the memory is too low already.
I would love to clear this memory from inside Mathematica, if not simply by restarting the program.
Also, I have noticed, that if I press "Alt+." early I can skip these kind of functions (highly recursive, requiring big amount of memory...), but when Mthematica enters into such functions deeper, then I cannot break the operations. The system becomes very slow, and if I try to do smth, it simply says "Not responding"... I can only hope that when it responds again I can go to Evaluation -> QuitKernel.
I use Windows 7, x64, 4GB RAM.
Regarding the last link above, do I risk anything if I follow the specified paths and set Windows to "clear my pagefile at shudown"?
In the meantime I've found what risks I would face at: http://lifehacker.com/5426041/understanding-the-windows-pagefile-and-why-you-shouldnt-disable-it