Very long gradients on compact manifolds

[mateuszbaran]

In the PCA optimization example I'm currently experimenting with the objective function has a very steep gradient. For example for a reasonable starting location norm of the gradient is about 1e6, while the manifold I'm optimizing on (Grassmann, with some initial experiments with flag manifold) has injectivity radius of pi/2. Should maybe line searches take injectivity radius into account when determining step size? Quasi-Newton eventually finds the right step size but it takes way more steps than necessary. For compact manifolds maybe the initial step size could be something like min(1.0, injectivity_radius(M) / norm(M, p, gradient))?

[kellertuer]

We do not have a type or decorator for compactness, but sure, the initial stepsize in the Armijo is a keyword arg where you can specify an initial step size. The same should be available or is easy to realise for the two Wolfe searches.

[mateuszbaran]

OK, then I could add an is_compact function to ManifoldsBase.jl, and later (after #174 ) modify the initial guess 🙂.

[kellertuer]

That sounds like a good plan.

[mateuszbaran]

I've been thinking about it a bit and maybe the best solution would be introducing a max_stepsize(M, p) function in ManifoldsBase.jl? It would default to injectivity_radius(M, p). Instead of is_compact we would check isfinite(max_stepsize(M, p)) which would be more application-specific.

[kellertuer]

I think it would be enough to have a max_stepsize in Manopt.jl? The name might be too different from what ManifoldsBase is meant to represent?

Since I have a conditional file that gets loaded if Manifolds is present – one could then there define most max stepsizes.

[mateuszbaran]

Right, yes, that would also be fine.