Message Boards Message Boards

Why does the memory used keep going up and no output is given?

Posted 2 months ago

This actually eventually crashed my hard drive, after memory exceeded several Gigabytes. I wonder if it can be improved?

POSTED BY: Iuval Clejan
15 Replies
Posted 2 months ago

I ran with Memory constrain of 300 GBytes and still it was not enough, even without 0 for the smallest eigenvalue

POSTED BY: Updating Name

This should be a more concise way of setting up the problem.

mat = Table[m[i, j], {i, 1, 4}, {j, 1, 4}];
{eigs, vecs} = Eigensystem[mat];
c1 = Thread[vecs[[4, All]] >= 0];
c2 = Flatten[{eigs[[4]] == 0, Thread[Most[eigs] <= 0]}];
c3 = Thread[Flatten[mat] >= 0];
vars = Flatten[mat];
constraints = Join[c1, c2, c3]

So if I understand correctly you are seeking a matrix with all nonnegative entries such that three eigenvalues are nonpositive, one is zero, and a normalization of sorts is imposed wherein all fourth components of eigenvectors are nonnegative. (Or something got transposed and the intent was that the null vector have all nonnegative components). Assuming I have this much correct, the normalization can be dropped because it can be enforced after the fact, multiplying eigenvectors by -1 where needed. So I redo the constraints like so.

constraints = Join[(*c1,*)c2, c3];

This causes FindInstance to go a much faster route it seems. It finds the "obvious" solution.

Timing[inst = FindInstance[constraints, vars]]

(* Out[62]= {0.00738, {{m[1, 1] -> 0, m[1, 2] -> 0, m[1, 3] -> 0, 
   m[1, 4] -> 0, m[2, 1] -> 0, m[2, 2] -> 0, m[2, 3] -> 0, 
   m[2, 4] -> 0, m[3, 1] -> 0, m[3, 2] -> 0, m[3, 3] -> 0, 
   m[3, 4] -> 0, m[4, 1] -> 0, m[4, 2] -> 0, m[4, 3] -> 0, 
   m[4, 4] -> 0}}} *)

Requesting more solutions will hang it though.

Here is a different formulation. It might not perform any better but it does remove the Root objects from consideration, by setting up explicit vectors for the eigenvectors.

mat = Array[m, {4, 4}];
eigs = Array[e, 4];
vecs = Array[v, {4, 4}];
tvecs = Transpose[vecs];
eqns = Table[
   Thread[mat . vecs[[j]] == eigs[[j]]*tvecs[[j]]], {j, 4}];
c2 = Flatten[{eigs[[4]] == 0, Thread[Most[eigs] <= 0]}];
c3 = Thread[Flatten[mat] >= 0];
vars = Flatten[{mat, eigs, vecs}];
constraints = Flatten[{eqns, c2, c3}];

Again it is fast to find the trivial solution. I do not know if FindInstance or Reduce will go beyond that though.

POSTED BY: Daniel Lichtblau
Posted 2 months ago

No, you misunderstood, confusing the eigenvector matrix with its transpose. What I need is the fourth eigenvector (corresponding to the 0 eigenvalue) to be all positive components (the last component of any eigenvector seems to be positive by convention in Mathematica, so I only impose this on the first 3 components of the 4th eigenvector). Perhaps I should not set the 4th eigenvalue to 0, but to a small real number, so that this makes more sense. So c1 is not redundant.

POSTED BY: Iuval Clejan

I was wondering about that. In this case any nonzero component of the null eigenvector will force the corresponding column of the matrix to vanish (since you have a sum of products of nonnegatives equated to zero). Maybe that will help to simplify things, at least to a 3x3 case.

POSTED BY: Daniel Lichtblau
Posted 2 months ago

see my edit. Am I right about the last component of any eigenvector always being positive? Or maybe only the last component of the eigenvector associated with the smallest absolute eigenvalue?

Trying a slightly positive eigenvalue now for the 4th eigenvalue.

POSTED BY: Iuval Clejan
Posted 2 months ago

It worked (correcting by tevec->vec)! Thanks Now for the real problem of getting 2 eigenvalues to be near 0 and their corresponding eigenvectors to be non-negative.

mat = Array[m, {4, 4}];
eigs = Array[e, 4];
vecs = Array[v, {4, 4}];
c1 = Table[v[4, j] > 0, {j, 3}];
eqns = Table[Thread[mat . vecs[[j]] == eigs[[j]]*vecs[[j]]], {j, 4}];
c2 = Flatten[{eigs[[4]] == 0.01, Thread[Most[eigs] <= 0]}];
c3 = Thread[Flatten[mat] >= 0];
vars = Flatten[{mat, eigs, vecs}];
constraints = Flatten[{eqns, c1, c2, c3}];
Timing[inst = FindInstance[constraints, vars]]
POSTED BY: Iuval Clejan
Posted 2 months ago

I take it back. It didn't work. Another constraint is linear independence of all the eigenvectors...

POSTED BY: Iuval Clejan
  1. The problem doesn't look trivial, so large amounts of memory will be needed. You haven't specified how much memory you've got. If it managed to fill up 64GB, I'd get concerned, but less than that is par for the course. You've entered high performance computing area. Problems like that one are solved on HPC clusters usually.
  2. You'll need at least 16GB of RAM to get any progress at all, and 24GB to go past 20-30 minutes into the solution. I'd say 32GB is a minimum to have, 64GB will give you a decent margin for at least a couple of hours I'd hope. If you got less than that, you'll need to buy RAM or find a computer with more RAM. The problem doesn't fit into less memory it looks like.
  3. Simplify the elements of the eigenvectors matrix.
  4. Get rid of list4, instead tell FindInstance that you want non-negative reals as the domain of the problem.
  5. list5 is just a list of elements of M, no need to regenerate it - we already got elements of M, they are Flatten[M].
  6. Simplify the generator for list2 and list3, to make it explicit what the constraints are. Perhaps you made a mistake in them? For loops are hard to read in this case; Mathematica is not C - it can do better :). Table is easier to read.
  7. Add a memory resource constraint around FindInstance so that the computation is aborted before your computer runs out of memory. Limit to the amount of RAM in your computer minus about 2GB. Say 30GB for a 32GB system. If you want the computer to be more responsive (it's a tradeoff!) while the computations proceed, give a wider margin - say 3GB or 4GB of RAM to the system, so on a 32GB system you'd limit the computation to 28GB.
  8. It looks like you want the 4th eigenvalue to be zero. That may is a bit of a too tight of a constraint. Numerical eigenvalues are always sorted in descending order, so one was zero, it would be the 4th one. But symbolic eigenvalues can be in any order, so any one of them may be zero.
  9. Furthermore, the solution is found numerically, so a "zero" eigenvalue doesn't have to be zero, just much smaller than the other eigenvalues - say 10 orders of magnitude smaller would be reasonable for machine precision real numbers.
  10. Due to 6 and 7, the constraint for the "zero" eigenvalue may need to be re-phrased.
    simplifyMatrix[m_] := 
       ParallelMap[FullSimplify, Flatten[m]] // 
          ArrayReshape[#, Dimensions[m]] &

    Protect[m];
    M = Table[m[i, j], {i, 4}, {j, 4}];
    eig = Eigenvalues[M];
    V = Eigenvectors[M] // simplifyMatrix;

    list2 = Table[V[[4, j]] >= 0, {j, 3}];
    list3 = Table[eig[[j]] <= 0, {j, 3}]~Append~(eig[[4]] == 0);

    MemoryConstrained[
       FindInstance[Join[list2, list3], Flatten[M], NonNegativeReals],
       30*^10
    ]

The code above gets much further than before. It looks like FindInstance is able to rephrase the problem into a nonlinear optimization problem and uses a multi-threading parallelized library to work on it. You'll know the moment it got to this phase because the CPU fans on your machine will spin up significantly.

I didn't wait very long, only about 15 minutes - it wasn't done, but it was running just shy of 16GB memory use. So you need more than that. A minimum of 12 logical CPUs are required to take full advantage of parallelization inherent in the problem. More cores than that won't be used. On a 16 logical core machine, about 12 cores were used, the rest were idle. Parallelization in libraries that FindInstance uses is done with multithreading, so only one kernel is needed.

Posted 2 months ago

Thanks, I am trying it, memory still going up, but only up to 8 Gig. How do I know that all 8 of my cores are being used? Do I need to tell Mathematica somewhere to do this, or is it automatic?

POSTED BY: Iuval Clejan
Posted 2 months ago

It stabilized around 15Gig in memory, but after an hour it was using 100% CPU and no output so I killed it.

POSTED BY: Iuval Clejan
Posted 2 months ago

I reopened the file I aborted and this was displaying, with changing numbers (what does it mean?):

AnimatorBox[Dynamic[Progress`ProgressDump`clock$$], {2., DirectedInfinity[1], 1.},
AnimationRate->1,
AnimationRunTime->123.19397640228271`,
AnimationTimeIndex->123.19397640228271`,
AppearanceElements->None,
ImageSize->{1, 1}]
POSTED BY: Iuval Clejan
Posted 2 months ago

Tried it again and let it run for longer. This time it seems to have finished, but no output at all. Not even an empty list.

POSTED BY: Iuval Clejan
Posted 2 months ago

I noticed that it had six kernels running, but only one was using most of the CPU and most of the memory.

POSTED BY: Iuval Clejan
Posted 2 months ago

I see you upped your core and memory requirement. I can certainly increase memory up to 32Gigs, but I only have 8 cores. I will try setting the "0" eigenvalue to a small real number and see if it works.

POSTED BY: Iuval Clejan
Posted 2 months ago

Is it a typo: 30^10? Isn't 30 Gigabytes 30^9?

POSTED BY: Iuval Clejan
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract