Message Boards Message Boards

0
|
8658 Views
|
7 Replies
|
3 Total Likes
View groups...
Share
Share this post:

Memory Leak: MemoryUse[] returns values too high compared to my Data size

Posted 9 years ago

Hi everyone,

I have a code here that should in the end give an array containing chains (arrays of three dimensional points). I've tested it and have produced a small array of 32 chains of length 4 points.

The data should be small, and running the same code multiple times should not increase memory use... However, mine is always increasing, and MemoryUse[] returns values too high compared to the size of my data !

I can't find what's wrong ! Could someone help me ? I've given my code below, I left undetailed BooleanTuplesOneByOne, a basic subroutine I've written which gives one by one the elements of Tuples[{False,True},n] because I did not want to store Tuples[{False,True},n] in memory. I know for sure it's not causing the leak, I tested it and it is a simple alphabetical ordering algorithm... so nothing could go wrong there!

If needed I can explain the reasons behind all these routines, but it would be long and I was hoping someone could help me just by looking at the code structure rather than the why's and how's of my coding. I'm naturally more than happy to give any info I may have neglected to give!

Result = {{{0, 0, 0}}};
(*Liste des sauts premiers voisins d'un site octahédrique à un autre*)

NextSite = {{1/2, 1/2, 0}, {-(1/2), 1/2, 0}, {1/2, -(1/2), 
    0}, {-(1/2), -(1/2), 0}, {1/2, 0, 1/2}, {0, 1/2, 1/2}, {-(1/2), 0,
     1/2}, {0, -(1/2), 1/2}, {1/2, 0, -(1/2)}, {0, 1/
    2, -(1/2)}, {-(1/2), 0, -(1/2)}, {0, -(1/2), -(1/2)}};
NewResult = {};
While[Length[Result[[1]]] < 4,

  Print[Timing[

     NewResult = Fold[Function[{u, v},

        Fold[Function[{x, y},


          If[! MemberQ[v, v[[Length[v]]] + y] && ! 
             CheckIfAlreadyInChainArrayBySymmetryAndNewOrigin[x, 
              Append[v, v[[Length[v]]] + y]],

           Append[x, Append[v, v[[Length[v]]] + y]],

           x

           ]

          ], u

         , NextSite]

        ], NewResult

       , Result];

     Print["Time (s) :"];

     ][[1]]];

  Result = NewResult;
  NewResult = {};

  Print["Size of Table : "];
  Print[Result // Length];
  Print["MemoryInUse : "];
  Print[MemoryInUse[]];
  Print["Context Length : "];
  Print[Length[Names[$Context <> "*"]]];
  Print["-------------------"];

  ];

CheckIfAlreadyInChainArrayBySymmetryAndNewOrigin[ResultArray_, 
  NewChain_] := 
 Module[{BooleanResult, Index, f, BinaryBooleanArray, 
   TransformationNumber, TransformedNewChain, TransformationsArray},

  BooleanResult = False;

  f[1] = Function[u, (#*{-1, 1, 1}) & /@ u];
  f[2] = Function[u, (#*{1, -1, 1}) & /@ u];
  f[3] = Function[u, (#*{1, 1, -1}) & /@ u];
  f[4] = Function[u, (#.{{0, -1, 0}, {-1, 0, 0}, {0, 0, 1}}) & /@ u];
  f[5] = Function[u, (#.{{0, 1, 0}, {1, 0, 0}, {0, 0, 1}}) & /@ u];
  f[6] = Function[u, (#.{{1, 0, 0}, {0, 0, -1}, {0, -1, 0}}) & /@ u];
  f[7] = Function[u, (#.{{1, 0, 0}, {0, 0, 1}, {0, 1, 0}}) & /@ u];
  f[8] = Function[u, (#.{{0, 0, 1}, {0, 1, 0}, {1, 0, 0}}) & /@ u];
  f[9] = Function[u, (#.{{0, 0, -1}, {0, 1, 0}, {-1, 0, 0}}) & /@ u];

  TransformationNumber = 9;

  BinaryBooleanArray = Table[False, {i, 1, TransformationNumber}];

  Index = 1;
  TransformedNewChain = NewChain;
  While[BooleanResult == False && Index <= Length[TransformedNewChain],

   BooleanResult = 
    MemberQ[ResultArray, 
     Map[Function[u, u - TransformedNewChain[[Index]]], 
      TransformedNewChain]];

   Index = Index + 1;

   ];

  While[BooleanResult == False && 
    BinaryBooleanArray != Table[True, {i, 1, TransformationNumber}],

   BinaryBooleanArray = BooleanTuplesOneByOne[BinaryBooleanArray];
   TransformedNewChain = NewChain;

   TransformationsArray = MapIndexed[Function[{u, v},

      If[u,

       f[v[[1]]],

       # &

       ]

      ], BinaryBooleanArray];

   TransformedNewChain = Fold[Function[{u, v},

      v[u]

      ], TransformedNewChain

     , TransformationsArray];

   Index = 1;
   While[BooleanResult == False && 
     Index <= Length[TransformedNewChain],

    BooleanResult = 
     MemberQ[ResultArray, 
      Map[Function[u, u - TransformedNewChain[[Index]]], 
       TransformedNewChain]];

    Index = Index + 1;

    ];

   ];

  BooleanResult

  ]
7 Replies

Hi!

Looking through the contexts that were created in each iteration they seem to be related to the f[..] function definition inside your module:

{"f$10000", "f$10001", "f$10002", "f$10003", "f$10004", "f$10005", \
"f$10006", "f$10007", "f$10008", "f$10009", "f$10010", "f$10011", \
"f$10012" ... }

When moving the definition outside the module no new contexts where created for me. Would that be an alternative for you?

During evaluation of In[106]:= Time (s) :

During evaluation of In[106]:= 0.

During evaluation of In[106]:= Size of Table : 

During evaluation of In[106]:= 12

During evaluation of In[106]:= MemoryInUse : 

During evaluation of In[106]:= 52354064

During evaluation of In[106]:= Context Length : 

During evaluation of In[106]:= 4676

During evaluation of In[106]:= -------------------

During evaluation of In[106]:= Time (s) :

During evaluation of In[106]:= 0.0312002

During evaluation of In[106]:= Size of Table : 

During evaluation of In[106]:= 132

During evaluation of In[106]:= MemoryInUse : 

During evaluation of In[106]:= 52371488

During evaluation of In[106]:= Context Length : 

During evaluation of In[106]:= 4676

During evaluation of In[106]:= -------------------

During evaluation of In[106]:= Time (s) :

During evaluation of In[106]:= 0.982806

During evaluation of In[106]:= Size of Table : 

During evaluation of In[106]:= 1404

During evaluation of In[106]:= MemoryInUse : 

During evaluation of In[106]:= 52574336

During evaluation of In[106]:= Context Length : 

During evaluation of In[106]:= 4676

During evaluation of In[106]:= -------------------

During evaluation of In[106]:= New Contexts:

Out[122]= {}

I have no idea why it is doing that though, but maybe someone else can explain?

POSTED BY: Patrik Ekenberg

Appending Remove[f]; to the end of the module also seems to work:

CheckIfAlreadyInChainArrayBySymmetryAndNewOrigin[ResultArray_, 
  NewChain_] := 
 Module[{BooleanResult, Index, f, BinaryBooleanArray, 
   TransformationNumber, TransformedNewChain, TransformationsArray}, 
  BooleanResult = False;
  f[1] = Function[u, (#*{-1, 1, 1}) & /@ u];
  f[2] = Function[u, (#*{1, -1, 1}) & /@ u];
  f[3] = Function[u, (#*{1, 1, -1}) & /@ u];
  f[4] = Function[u, (#.{{0, -1, 0}, {-1, 0, 0}, {0, 0, 1}}) & /@ u];
  f[5] = Function[u, (#.{{0, 1, 0}, {1, 0, 0}, {0, 0, 1}}) & /@ u];
  f[6] = Function[u, (#.{{1, 0, 0}, {0, 0, -1}, {0, -1, 0}}) & /@ u];
  f[7] = Function[u, (#.{{1, 0, 0}, {0, 0, 1}, {0, 1, 0}}) & /@ u];
  f[8] = Function[u, (#.{{0, 0, 1}, {0, 1, 0}, {1, 0, 0}}) & /@ u];
  f[9] = Function[u, (#.{{0, 0, -1}, {0, 1, 0}, {-1, 0, 0}}) & /@ u];
  TransformationNumber = 9;
  BinaryBooleanArray = Table[False, {i, 1, TransformationNumber}];
  Index = 1;
  TransformedNewChain = NewChain;
  While[BooleanResult == False && 
    Index <= Length[TransformedNewChain], 
   BooleanResult = 
    MemberQ[ResultArray, 
     Map[Function[u, u - TransformedNewChain[[Index]]], 
      TransformedNewChain]];
   Index = Index + 1;];
  While[BooleanResult == False && 
    BinaryBooleanArray != Table[True, {i, 1, TransformationNumber}], 
   BinaryBooleanArray = BooleanTuplesOneByOne[BinaryBooleanArray];
   TransformedNewChain = NewChain;
   TransformationsArray = 
    MapIndexed[Function[{u, v}, If[u, f[v[[1]]], # &]], 
     BinaryBooleanArray];
   TransformedNewChain = 
    Fold[Function[{u, v}, v[u]], TransformedNewChain, 
     TransformationsArray];
   Index = 1;
   While[BooleanResult == False && 
     Index <= Length[TransformedNewChain], 
    BooleanResult = 
     MemberQ[ResultArray, 
      Map[Function[u, u - TransformedNewChain[[Index]]], 
       TransformedNewChain]];
    Index = Index + 1;];];
  Remove[f];
  BooleanResult]
POSTED BY: Patrik Ekenberg

No problem. It does seems like the problem they are discussing in that thread is the same as you are experiencing.

I saved the list of contexts before and after evaluation of the while loop:

Result = {{{0, 0, 0}}};
(*Liste des sauts premiers voisins d'un site octahédrique à un autre*)


preContext = Names[$Context <> "*"];
NextSite = {{1/2, 1/2, 0}, {-(1/2), 1/2, 0}, {1/2, -(1/2), 
    0}, {-(1/2), -(1/2), 0}, {1/2, 0, 1/2}, {0, 1/2, 1/2}, {-(1/2), 0,
     1/2}, {0, -(1/2), 1/2}, {1/2, 0, -(1/2)}, {0, 
    1/2, -(1/2)}, {-(1/2), 0, -(1/2)}, {0, -(1/2), -(1/2)}};
NewResult = {};
While[Length[Result[[1]]] < 4, 
 Print[Timing[
    NewResult = 
     Fold[Function[{u, v}, 
       Fold[Function[{x, y}, 
         If[! MemberQ[v, v[[Length[v]]] + y] && ! 
            CheckIfAlreadyInChainArrayBySymmetryAndNewOrigin[x, 
             Append[v, v[[Length[v]]] + y]], 
          Append[x, Append[v, v[[Length[v]]] + y]], x]], u, 
        NextSite]], NewResult, Result];
    Print["Time (s) :"];][[1]]];
 Result = NewResult;
 NewResult = {};
 Print["Size of Table : "];
 Print[Result // Length];
 Print["MemoryInUse : "];
 Print[MemoryInUse[]];
 Print["Context Length : "];
 Print[Length[Names[$Context <> "*"]]];
 Print["-------------------"];]; 
 postContext = Names[$Context <> "*"];

And then compared what new items had appeared in the new list:

Complement[postContext, preContext]

It is similar to what you were already doing when you compared the length of the context lists, but instead looking specifically on what had been added during evaluation!

POSTED BY: Patrik Ekenberg

Hey thanks !

I'll check it out tomorrow in the office. Could you lend me a few of your tricks that helped you detect the bug so that I can be more autonomous next time ?

I had looked for a lot of info, and here are a couple of things that were almost useful to me before I resorted to post my question here, maybe it'll help you find the answers you seek ?

http://stackoverflow.com/questions/6867892/reducing-memory-usage-in-an-extended-mathematica-session (This post's best answer shows some strange behaviours within Modules, and the Context gets polluted. Using Blocks fixes it somehow)

http://groups.google.com/group/comp.soft-sys.math.mathematica/browse_thread/thread/b599b45b9c312e97/3aece5f132c2de27 (The first link's reference link)

Does using Block rather than Module fix the issue for you ?

What version of Mathematica are you using ?

Because my context length is constant and equals 40...

So maybe the issue you pointed isn't affecting me ? I can only confirm this tomorrow morning in my time zone.

And your output for MemoryInUse[] doesn't increase as fast as mine, even though you have a much longer Context Length !

What version of Mathematica do you have ? Maybe I need an update. I have a rather recent one though...

Alright my problem is either the version of Mathematica or my setup (or both)

My platform is Linux x86 (64bit)

My memory consumption is off the charts in Mathematica 10.0.1.0, and I do not have the same behaviour as Patrik Ekenberg : my context is not littered with ghost variables. I have no idea where the memory is leaking.

Time (s) :

0.124007

Size of Table :

1

MemoryInUse :

88988904

Context Length :

38


Time (s) :

0.452028

Size of Table :

4

MemoryInUse :

91639808

Context Length :

38


Time (s) :

4.804301

Size of Table :

32

MemoryInUse :

125118880

Context Length :

38


I ran the same code in Mathemematica 1.3.0.0 and the memory consumption is low and the code behaves as it should, with the context not littered :

Time (s) :

0.792049

Size of Table :

1

MemoryInUse :

28239512

Context Length :

41


Time (s) :

0.888055

Size of Table :

4

MemoryInUse :

28240304

Context Length :

41


Time (s) :

4.7843

Size of Table :

32

MemoryInUse :

28245152

Context Length :

41


Here is the full code below :

FinalChainLength = 4;
(*The list Result will contain chains of length FinalChainLength, \
upon initialization it contains only one chain of length equals one*)

Result = {{{0, 0, 0}}};
(*The following table contains the next site we are going to hop to \
from Result[[i]][[Length[Result[[i]]]]] in order to extend the chain*)

NextSite = {{1/2, 1/2, 0}, {-(1/2), 1/2, 0}, {1/2, -(1/2), 
    0}, {-(1/2), -(1/2), 0}, {1/2, 0, 1/2}, {0, 1/2, 1/2}, {-(1/2), 0,
     1/2}, {0, -(1/2), 1/2}, {1/2, 0, -(1/2)}, {0, 1/
    2, -(1/2)}, {-(1/2), 0, -(1/2)}, {0, -(1/2), -(1/2)}};
NewResult = {};
While[Length[Result[[1]]] < FinalChainLength,

  Print[Timing[

     (*Double Fold is used to replace what would be in other \
languages a double For statement, scaning every element of result, 
     and scanning every element of NextSite*)

     NewResult = Fold[Function[{u, v},

        Fold[Function[{x, y},
          (*Only add the new extended chain to the new result array x \
if the next site is not already found in the currently studied chain \
v ; AND if the new extended chain isn't already in new result in a \
different form which can be found my applying symmetries and \
redefining the chain's origin point*)

          If[! MemberQ[v, v[[Length[v]]] + y] && ! 
             CheckIfAlreadyInChainArrayBySymmetryAndNewOrigin[x, 
              Append[v, v[[Length[v]]] + y]],

           Append[x, Append[v, v[[Length[v]]] + y]],

           x

           ]

          ], u

         , NextSite]

        ], NewResult

       , Result];

     Print["Time (s) :"];

     ][[1]]];

  Result = NewResult;
  NewResult = {};

  Print["Size of Table : "];
  Print[Result // Length];
  Print["MemoryInUse : "];
  Print[MemoryInUse[]];
  Print["Context Length : "];
  Print[Length[Names[$Context <> "*"]]];
  Print["-------------------"];

  ];
CheckIfAlreadyInChainArrayBySymmetryAndNewOrigin[ResultArray_, 
   NewChain_] := 
  Module[{BooleanResult, Index, f, BinaryBooleanArray, 
    TransformationNumber, TransformedNewChain, TransformationsArray},

   BooleanResult = False;

   f[1] = Function[u, (#*{-1, 1, 1}) & /@ u];
   f[2] = Function[u, (#*{1, -1, 1}) & /@ u];
   f[3] = Function[u, (#*{1, 1, -1}) & /@ u];
   f[4] = 
    Function[u, (#.{{0, -1, 0}, {-1, 0, 0}, {0, 0, 1}}) & /@ u];
   f[5] = Function[u, (#.{{0, 1, 0}, {1, 0, 0}, {0, 0, 1}}) & /@ u];
   f[6] = 
    Function[u, (#.{{1, 0, 0}, {0, 0, -1}, {0, -1, 0}}) & /@ u];
   f[7] = Function[u, (#.{{1, 0, 0}, {0, 0, 1}, {0, 1, 0}}) & /@ u];
   f[8] = Function[u, (#.{{0, 0, 1}, {0, 1, 0}, {1, 0, 0}}) & /@ u];
   f[9] = 
    Function[u, (#.{{0, 0, -1}, {0, 1, 0}, {-1, 0, 0}}) & /@ u];

   TransformationNumber = 9;
   (*We have a certain number of transformations possible, 
   we want to apply every possible combination of them*)

   TransformationsArray = Map[Function[u,
      (*This Fold will apply to NewChain every possible combination \
of f[i] (order is irrelevent) and store all the results in a (not so) \
large array of length Length[Tuples[{False,True},
      TransformationNumber]]*)
      Fold[Function[{x, y},

        y[x]

        ], NewChain

       ,
       (*This array here turns an array of Booleans into an array of \
transformations f[
       i] where True is found or #& (identity function) where False \
is found*)
       MapIndexed[Function[{x, y},

         If[x,

          f[y[[1]]],

          # &

          ]

         ], u]]

      ], Tuples[{False, True}, TransformationNumber]];

   (*Now we want to check if all the transformed versions of NewChain \
we found exist in ResultArray, 
   but because in every step of the main code the length of NewChain \
increases, (while TransformationsNumber remains constant), 
   I do not want this time to map a function and create a very large \
array, so I use a while recursion*)

   BinaryBooleanArray = Map[Function[u,

      Block[{LocalBooleanResult},

       LocalBooleanResult = False;
       Index = 1;
       While[LocalBooleanResult == False && Index <= Length[u],

        LocalBooleanResult = 
         MemberQ[ResultArray, Map[Function[v, v - u[[Index]]], u]];

        Index = Index + 1;

        ];
       LocalBooleanResult
       ]

      ], TransformationsArray];

   (*If any of the forms of NewChain was found in ResultArray then \
the final BooleanResult is True and NewChain must not be added in the \
main code*)
   BooleanResult = MemberQ[BinaryBooleanArray, True];

   BooleanResult

   ];
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract