Add WolfeSearch: a strong Wolfe line search#233
Open
MashedP wants to merge 1 commit into
Open
Conversation
WolfeSearch implements the strong Wolfe conditions via the bracketing/zoom algorithm of Nocedal & Wright (Alg. 3.5-3.6). Since Optimistix calls AbstractSearch.step once per solver step, the textbook nested loops are expressed as a single per-call state machine dispatched by a `mode` flag. The curvature condition needs the gradient at each trial point. To avoid penalising searches that only need function values (e.g. BacktrackingArmijo), AbstractSearch gains an opt-in `requires_grad_eval` class flag; the gradient/ quasi-Newton solvers compute the trial-point gradient eagerly only when the search requests it, reusing it on acceptance. BFGS, DFP and LBFGS gain an optional `search=` argument (default unchanged: BacktrackingArmijo) so users can select WolfeSearch. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
WolfeSearch, a line search satisfying the strong Wolfe conditions, andmakes it selectable on the quasi-Newton minimisers. This addresses the
# TODO(raderj): switch out BacktrackingArmijo with a better line searchmarkers in
BFGS/DFP: a Wolfe search enforces a curvature condition, whichkeeps steps large enough for quasi-Newton Hessian updates to stay
well-conditioned, and in practice converges in fewer iterations.
For example, on Rosenbrock from
[-1.2, 1.0],LBFGSwith the defaultBacktrackingArmijostalls, whileLBFGS(search=WolfeSearch())converges to[1, 1].Design
WolfeSearchfollows the bracketing/zoom algorithm of Nocedal & Wright(Alg. 3.5,3.6), using bisection for the zoom interpolation. Because Optimistix
calls
AbstractSearch.steponce per solver step (the iteration loop belongs tothe solver), the textbook nested loops are expressed as a single per-call state
machine dispatched by a
modeflag, no nestedlax.while_loop.The curvature condition needs the gradient at each trial point, which the
solvers currently do not pass to the search. To avoid penalising searches that
only need function values (e.g.
BacktrackingArmijo), this adds an opt-in:AbstractSearch.requires_grad_eval: ClassVar[bool] = False.WolfeSearchsets it toTrue.AbstractGradientDescent.stepandAbstractQuasiNewton.stepcompute thetrial-point gradient eagerly only when
search.requires_grad_eval, andreuse it on acceptance (so there is no extra work overall, and no change for
the default path).
BFGS,DFPandLBFGSgain an optionalsearch=argument; the defaultremains
BacktrackingArmijo, so existing behaviour is unchanged.Tests
tests/test_wolfe_search.py: verifies the strong Wolfe conditions holddirectly on a 1D restriction, convergence of BFGS/LBFGS ×
use_inversewithWolfe,
jit, therequires_grad_evalflag, and that the default Armijo pathis unchanged.
tests/test_minimise.pycases still pass.ruffclean; beartype-strict typechecking passes.