Hi!
I am trying to perform a large scale optimization on Mathematica - however the speed is very poor - in fact, when the number of variables is high then it just makes computer really slow, and I have to restart. Code snippets are below:
Clear["Global`*"]
NoVariables = 10;
NoExamples = 5000;
allData = RandomReal[1, {NoExamples, NoVariables}];
X = Transpose[Prepend[Transpose[allData], Table[1, {NoExamples}]]];
Dimensions @ X (* X is 200 by 101 matrix, with first column of only \
1's *)
Thetas = Table[Subscript[\[Theta], i], {i, 0, NoVariables}];
Dimensions @ Thetas (* Thetas are 101 size vector, with Subscript[\
\[Theta], 0] representing the coefficient of the constant term *)
Y = RandomChoice[{0., 1.}, NoExamples ];
Dimensions @ Y (*y is a vector of size 200, being either 0 or 1 *)
This creates the main data variables: Y (vector), X(Matrix). Thetas is a vector of parameters I want to optimize. Below I create the objective function:
(* Convex Objective Function, row wise, is: y Log[1+\[ExponentialE]^-\
\[Theta].x]-(-1+y) Log[1+\[ExponentialE]^(\[Theta].x)]
We need to sum this over each row - there are 200 (NoExamples)
We will actually be scaling this by 1/NoExamples
We can vectorize this like follows to create the objective function \
element wise format: *)
ElementWiseObjective =
Y*Log[1 + Exp[-X.Thetas]] - (-1 + Y)*Log[1 + Exp[X.Thetas]];
Dimensions @ ElementWiseObjective
(* Perform summation and scaling to get final form objective \
function: *)
Objective = (1/NoExamples)*(Plus @@ ElementWiseObjective);
And then I run the optimization:
(* Perform Optimization *)
Timing @ FindMinimum[Objective, Thetas]
When NoVariables = 10 this takes about 7 seconds, and when NoVariables = 10, it takes about 40 seconds. When NoVariables = 100 then the computer becomes really slow, and it is impossible to abort or access other programs. Ideally I want NoVariables to be very large, perhaps even 500 or larger.
Is there a way I can tweak my code to tun this optimization, and then run it quicker? Any help would be much appreciated!
Thanks! Priyan.
Attachments: