Nice visualizations! A short self-contained example for how you did the 3D optimizer plot would be useful to me.

On a personal note, I've spent quite a while trying to make second order optimizers work, latest attempt is here, and I've lost faith in black box approaches.

Black-box approach (ie, treating neural network as a generic function) makes it easy to improve over SGD in low dimensions, but not in high dimensions. The issue is that high dimensionality + mini-batching means that minibatch objective is well conditioned, meaning that gradient update is already high quality.

However, I think white-box approaches are promising. In fact, one could hypothesise that much of the progress in the last 5 years came from coming up with better update rules, except these tweaks are hiding in the architecture. IE, batch norm layer, skip connections, those could be seen as tweaks that make update more efficient, and could've been implemented as an optimizer tweak, rather than architecture tweak.

I'm not aware of second-order methods being successful for standard neural network architectures. Shampoo exists, but it's not clear that it outperforms Adam.

There's a theory that standard network architectures are already close to optimal for SGD, or they wouldn't be popular. It's an architecture/optimizer co-evolution issue.

The issue with non-convexity/saddlepoints has received a lot of attention, consider the 1500 [citations] of "Identifying and attacking the saddle point problem". Nothing of practical value appears to have come out of this line of work. High-dimensional setting breaks many common intuitions we have about optimization. Misha Belkin's "fit without fear" is a good read to get different perspective -- https://arxiv.org/abs/2105.14368 .

Things are more promising for second-order methods in domains where neural network design is constrained by the problem. KFAC appears to be an important component of getting some chemistry-related networks training, check out publications of http://davidpfau.com/ for the example .

I think advanced optimizers may be important for model compression. Wide models are easy to train with SGD, but narrow + deep are hard. For good performance on small devices (phones, hearing aids), you want small models, and may eventually want to train small models from scratch. SGD works badly on tiny models, so that could be a potential way to get impact from more advanced optimizers.

Thanks, indeed it might be difficult to get successful second order methods for training, but maybe it is still worth to search. For example the basic approach (cdOGR) in diagram below turned out similar to ADAM-like: also derived division by mean squared gradient coordinates. But additionally in nominator there is similar for position, what is natural when looking at shown "position vs gradient plot" - to find the g=0 minimum, we should divide gradient by width in g, but also multiply by width in theta.

Regarding other applications, I though about about continuous, metalearning - to hold beside the actual model, also model of Hessian there - e.g. in these diagonal/subspace approximations, to better exploit new scarce gradients.

Regarding saddle attraction, sure I know and cite saddle-free Newton, and use its |lambda|. However, there is this nice paper: https://arxiv.org/pdf/1902.02366 , calculating 3Mx3M Hessian, and suggesting asymmetry - different behavior for negative curvatures. Also, a few first dominant eigenvectors have rather positive curvature - so e.g. in subspace "s" OGR variants we can restrict to such directions.