Numerical Optimization Toolbox for Reduced OCPs

Overview¶

This lecture is a numerical optimization toolbox for the reduced formulation

The same algorithms will later be reused for PDE-constrained optimization, where $f(u)=J(S(u),u)$ and each gradient evaluation typically requires state/adjoint solves.

Main goals:

1. put GD, nonlinear CG, BFGS, and trust-region in one common framework;

2. isolate the role of Armijo/Wolfe conditions;

3. compare methods by cost per iteration and robustness;

4. prepare the transition to reduced PDE-constrained optimization.

Unified Iterative Template¶

Most unconstrained methods fit, at a high level, the same iterative template: Choose $u_0\in\mathbb{R}^n$ and for $k=0,1,2,\ldots$ compute

with four design choices:

• model of local “geometry”, i.e., what we know about the “landscape” of $f$ in the surrounding of $u_k$ (gradient only, Hessian, Hessian approximation);

• search direction $p_k$;

• step acceptance/globalisation rule;

• stopping criteria.

Typical stopping checks (often combined):

Line Search and Armijo¶

Given a descent direction $p_k$ with $\nabla f(u_k)^T p_k<0$, line search picks $\alpha_k>0$ to ensure robust decrease.

The ideal choice is the exact line-search minimizer

Since

an interior minimizer satisfies $\phi'(\alpha_k^\star)=0$. Using the second-order expansion at $\alpha=0$,

with $g_k=\nabla f(u_k)$ and $H_k=\nabla^2 f(u_k)$, the minimizer of this local model is

For a quadratic $f(u)=\frac12 u^T H u-b^T u$ (constant Hessian), this is exact:

and for steepest descent ($p_k=-g_k$):

In general nonlinear problems, exact line search is usually too expensive; backtracking is the standard practical alternative.

Armijo backtracking¶

Choose $\alpha_0>0$, $c_1\in(0,1)$, $\beta\in(0,1)$. Set $\alpha\leftarrow \alpha_0$ and reduce $\alpha\leftarrow \beta\alpha$ until

Interpretation:

• right-hand side is a conservative linear prediction of decrease;

• if condition fails, the step is too optimistic for local curvature;

• backtracking is cheap and usually enough in practice.

Wolfe conditions (for $0<c_1<c_2<1$):

Strong Wolfe replaces the curvature inequality with

Geometric interpretation along the line $\phi(\alpha):=f(u_k+\alpha p_k)$:

• Armijo enforces sufficient decrease in function value (step not too large);

• Wolfe curvature condition enforces $\phi'(\alpha)$ to be much less negative than $\phi'(0)$, so the step is not stopped too early;

• strong Wolfe additionally controls oscillations by requiring small absolute slope at the accepted point.

Gradient Descent (GD)¶

Direction:

Strengths:

• minimal per-iteration cost;

• robust with Armijo backtracking;

• simple baseline for any new model.

Limitations:

• sensitive to conditioning;

• slow on narrow valleys (zig-zag behavior);

• linear convergence under strong convexity.

Use case in this course: reference solver for reduced OCP prototypes.

Nonlinear Conjugate Gradient (CG)¶

Replace steepest descent with

with reset $p_k=-g_k$ when needed.

Popular $\beta_k$ formulas:

• Fletcher-Reeves: $\beta_k^{FR}=\frac{g_k^T g_k}{g_{k-1}^T g_{k-1}}$;

• Polak-Ribiere+: $\beta_k^{PR+}=\max\!\left(0,\frac{g_k^T(g_k-g_{k-1})}{g_{k-1}^T g_{k-1}}\right)$.

Key facts:

• for SPD quadratics with exact line search, linear CG terminates in at most $n$ steps;

• nonlinear CG is memory-light (O(n) storage) and often faster than GD;

• practical performance depends strongly on line-search quality and resets.

BFGS Quasi-Newton¶

BFGS stands for Broyden-Fletcher-Goldfarb-Shanno (the four authors who proposed this update family).

Core idea: mimic Newton’s method without computing the exact Hessian. Instead, build a sequence of positive-definite inverse-Hessian approximations from gradient differences, so directions keep curvature information while each iteration only needs function/gradient evaluations.

Approximate inverse Hessian $H_k\approx \nabla^2 f(u_k)^{-1}$ and use

With

BFGS update is

Equivalent expanded form:

So the inverse-Hessian approximation is updated through rank-1 outer-product terms (overall a rank-2 correction per iteration).

Practical points:

• much faster than GD on ill-conditioned smooth problems;

• good local behavior (often superlinear with standard assumptions);

• more memory and linear algebra than GD/CG.

For large scale: use L-BFGS (limited-memory variant): keep only the last $m$ pairs $(s_i,y_i)$ (typically $m\in[5,20]$), do not form $H_k$ explicitly, and compute $p_k=-H_k g_k$ via the two-loop recursion. Memory drops from $O(n^2)$ to $O(mn)$ and matrix factorizations are avoided.

Trust-Region (TR) Methods¶

Instead of fixing a direction then line-searching, solve a local model:

Accept/reject with

Rule of thumb:

• $\rho_k$ small: model inaccurate -> shrink radius $\Delta_k$;

• $\rho_k$ good: accept step, maybe enlarge $\Delta_k$.

Advantages:

• very robust under nonconvexity or poor Hessian quality;

• natural handling of negative curvature.

Typical subproblem solvers: Cauchy step, dogleg, truncated CG.

Method Selection Cheat Sheet¶

• GD + Armijo: safest baseline, cheapest iteration, slowest asymptotically.

• Nonlinear CG: cheap memory, often better than GD, more tuning-sensitive.

• BFGS: strong default for smooth medium-size problems.

• Trust-region: robust when curvature is difficult/nonconvex.

In reduced PDE-constrained optimization:

• dominant cost = gradient/Hessian-vector evaluation (state/adjoint solves);

• iteration count matters less than “PDE solves per accepted step”;

• robust globalisation is critical.

Finite-Dimensional Conceptual Experiments (Notebook)¶

In jupyterbook/codes/lecture03/optimization_toolbox.ipynb:

1. GD + Armijo on an anisotropic quadratic.

2. Nonlinear CG (FR and PR+) on Rosenbrock.

3. BFGS vs GD comparison on Rosenbrock.

4. Trust-region (dogleg) on a shifted nonconvex quadratic.

Each experiment reports:

• objective history;

• gradient norm decay;

• trajectory in 2D (when applicable).

To start separating reusable code from lecture-specific examples, the repository now also includes jupyterbook/codes/common/optim.py and the small script jupyterbook/codes/lecture03/gradient_descent_demo.py. The notebook remains the main teaching artifact; the Python script is a lighter-weight example that reuses code meant to be shared by later lectures.