(Cross-posted from Mathematica.SE.)
I wish to share some observations concerning current limitations (bugs?) of Mathematica's "LevenbergMarquardt" algorithm of FindMinimum which (as I hope) can help to improve it.
I've spent much time finding a minimal example demonstrating this problem. Normally one faces it when fitting very large and complicated functions, but the attached Notebook contains a simplified example which allows reproducing it. The essence of the problem seems to be that FindMinimum doesn't increase step size when it should. The outcome is a ridiculously slow minimization process where FindMinimum takes so little steps in the direction of the minimum that it virtually never achieves it. But with a simple workaround it is possible to get FindMinimum coming to the minimum in a few steps.
In the section "Demonstration of the problem" I give several numbered code samples. The case number 1 is an attempt to fit with the default settings. At first glance it works well because we get the desired minimum in 220 steps what is a good result, but look at the second case: simple multiplication of the residual vector by 10 allows obtaining the same much faster, in 51 steps! This is already suspicious because the correct implementation of the Levenberg-Marquardt algorithm shouldn't depend on the scale of residuals, at least in such extent. The third case emphasizes this: simple division of the residual vector by 2 makes it impossible to obtain the minimum with default MaxIterations, and we have to increase the latter, because now the minimization process requires 1005 steps!
The four code sample demonstrates a workaround: restricting MaxIterations to 100 and running the minimization again with starting values set to the minimum obtained after 100 iterations allows achieving the true minimum in 120 steps in total even when the residual vector is divided by 20.
This workaround demonstrates that it is undoubtedly possible to improve current algorithm of FindMinimum, and with relatively little effort. I also hope that it will help to fix the bug causing FindMinimum to depend on the scale of the residuals in such large extent.
Reported to support as [CASE:3967239].
Attachments:
