updated regularization schemes by adding Tikhonov and Marquardt

[leedh0124]

1. Adds :Tikhonov and :Marquardt options to regularize_linear_solve for the interior-point solver’s linear solve step.
2. Uses normal equations for the new regularization modes:

• :Tikhonov: solves with ∇F'∇F + ηI
• :Marquardt: solves with ∇F'∇F + ηD, where D is a diagonal scaling from ∇F

[dfridovi]

These changes look good to me! Please make sure to update the tests to exercise all the different regularization options. Ideally, please also run the QP benchmarks and report runtimes in the PR description - it will be useful documentation for the release notes when we make a new release. Will approve merge when this is all in.

[leedh0124]

No problem :)

[leedh0124]

Updates after review

Addressed reviewer feedback by adding test coverage for the newly introduced linear solve regularization schemes and running the QP benchmarks.

Test coverage

Updated test/runtests.jl so the QP tests exercise the direct solver regularization options:

• :none
• :identity
• :Tikhonov
• :Marquardt

QP Benchmark Results

Ran the QP benchmark with 100 random samples for each regularization scheme:

data = SolverBenchmarks.benchmark(
    SolverBenchmarks.QuadraticProgramBenchmark();
    num_samples = 100,
    ip_kwargs = (regularize_linear_solve = scheme, tol = 1e-6),
)

SolverBenchmarks.summary_statistics(data)

IP regularization	IP success	IP mean runtime	IP std	PATH success	PATH mean runtime	PATH std	IP / PATH
:none	100%	0.03585 s	0.00420 s	100%	0.00519 s	0.00149 s	690.4%
:identity	100%	0.02398 s	0.00364 s	100%	0.00526 s	0.00161 s	455.7%
:Tikhonov	100%	0.10464 s	0.06258 s	100%	0.00614 s	0.00821 s	1705.2%
:Marquardt	100%	0.11147 s	0.07580 s	100%	0.00518 s	0.00128 s	2150.6%

It's actually running slower than PATH. Is there something I'm missing when running the QP benchmarks?

[dfridovi]

Yeah that is troubling. I am not sure when the solver slowed down. I wonder if it has to do with the default regularization level (at least for :internal and :identity). I expected it for the new ones, because you are explicitly forming the Jacobian transpose Jacobian product etc. It is worth spending some time with Claude to see if there are easy ways to optimize that. If it requires more substantial code changes though just document it here in the PR description, maybe add a warning when a user selects the new regularization modes, and we can merge.