A general QNOperator

[paraynaud]

Hello all, I want to submit to you an idea about quasi-Newton operators.
Currently, we have LBFGSOperator to build a convex quasi-Newton approximation and LSR1Operator to catch negative curvatures.
I wonder if we can build a limited-memory operator that could use both BFGS and SR1 equations.
Even if positive definitness is lost, we get more chance to satisfy numerical safeguards.
Contrary to a damped operator, it keeps the real curvature instead of a damped one.

Suppose a first LBFGS update

$$ B_{k+1} = B_k + \frac{y_k y_k^\top}{y_k^\top s_k} + \frac{B_k s_k s_k^\top B_k}{s_k B_k s_k}, $$

and then, a LSR1 update

$$ B_{k+1} = B_{k+1} v + \frac{r_{k+1} r_{k+1}^\top}{r_{k+1}^\top s_{k+1}}, $$

with $r_{k+1} = y_k{k+1} - B_{k+1} s_{k+1}$.
Then we get the product $B_{k+2} v$ as

$$ B_{k+2}v = B_k v + \frac{y_k^\top s_k}{y_k^\top s_k} y_k + \frac{s_{k}^\top B_k v}{s_k^\top B_k s_k} B_k s_k + \frac{r_{k+1}^\top v}{r_{k+1}^\top s} r_{k+1} $$

where $B_k$ may be initially set as a multiple of the identity.

This QNOperator could choose automatically which update perform based on the satisfaction of numerical safeguards.
It would keep the related vectors and adapt product accordingly to the update selected.
What do you think?

[dpo]

Yes it’s a good idea and I see where you would use it.