Skip to content

Add WolfeSearch: a strong Wolfe line search#233

Open
MashedP wants to merge 1 commit into
patrick-kidger:mainfrom
MashedP:wolfe-search
Open

Add WolfeSearch: a strong Wolfe line search#233
MashedP wants to merge 1 commit into
patrick-kidger:mainfrom
MashedP:wolfe-search

Conversation

@MashedP

@MashedP MashedP commented Jun 9, 2026

Copy link
Copy Markdown

Summary

Adds WolfeSearch, a line search satisfying the strong Wolfe conditions, and
makes it selectable on the quasi-Newton minimisers. This addresses the
# TODO(raderj): switch out BacktrackingArmijo with a better line search
markers in BFGS/DFP: a Wolfe search enforces a curvature condition, which
keeps steps large enough for quasi-Newton Hessian updates to stay
well-conditioned, and in practice converges in fewer iterations.

For example, on Rosenbrock from [-1.2, 1.0], LBFGS with the default
BacktrackingArmijo stalls, while LBFGS(search=WolfeSearch()) converges to
[1, 1].

Design

WolfeSearch follows the bracketing/zoom algorithm of Nocedal & Wright
(Alg. 3.5,3.6), using bisection for the zoom interpolation. Because Optimistix
calls AbstractSearch.step once per solver step (the iteration loop belongs to
the solver), the textbook nested loops are expressed as a single per-call state
machine dispatched by a mode flag, no nested lax.while_loop.

The curvature condition needs the gradient at each trial point, which the
solvers currently do not pass to the search. To avoid penalising searches that
only need function values (e.g. BacktrackingArmijo), this adds an opt-in:

  • AbstractSearch.requires_grad_eval: ClassVar[bool] = False.
  • WolfeSearch sets it to True.
  • AbstractGradientDescent.step and AbstractQuasiNewton.step compute the
    trial-point gradient eagerly only when search.requires_grad_eval, and
    reuse it on acceptance (so there is no extra work overall, and no change for
    the default path).

BFGS, DFP and LBFGS gain an optional search= argument; the default
remains BacktrackingArmijo, so existing behaviour is unchanged.

Tests

  • New tests/test_wolfe_search.py: verifies the strong Wolfe conditions hold
    directly on a 1D restriction, convergence of BFGS/LBFGS × use_inverse with
    Wolfe, jit, the requires_grad_eval flag, and that the default Armijo path
    is unchanged.
  • All 493 existing tests/test_minimise.py cases still pass.
  • ruff clean; beartype-strict typechecking passes.

WolfeSearch implements the strong Wolfe conditions via the bracketing/zoom
algorithm of Nocedal & Wright (Alg. 3.5-3.6). Since Optimistix calls
AbstractSearch.step once per solver step, the textbook nested loops are
expressed as a single per-call state machine dispatched by a `mode` flag.

The curvature condition needs the gradient at each trial point. To avoid
penalising searches that only need function values (e.g. BacktrackingArmijo),
AbstractSearch gains an opt-in `requires_grad_eval` class flag; the gradient/
quasi-Newton solvers compute the trial-point gradient eagerly only when the
search requests it, reusing it on acceptance.

BFGS, DFP and LBFGS gain an optional `search=` argument (default unchanged:
BacktrackingArmijo) so users can select WolfeSearch.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant