...

- My notes since my PhD study (2017). Email me if you catch a mistake/typo.
- I update this page often, refresh browser to avoid showing the old page.
- Some quotes
- “In maths you don't understand things. You just get used to them.''
**John von Neumann** - “I learned very early the difference between knowing the name of something and knowing something”
**Richard Feynman**

...

**COMP1215 Foundations of Computer Science 2023-2024 Uni. Southampton****CO327 Deterministic OR Models 2022-spring, Uni. Waterloo****MARO201**

...

...

1st-order Taylor approximation of convex function and Bregman divergence

proj operator is firmly non-expensive & obtuse angle criterion

Augmented objective function & methods for constrained optimization

Prox of norm, Moreau's decomposition, conjugate of norm = indicator function of dual norm

Projection onto nonnegative orthant, rectangular box and polyhedron |

ball | ball | simplex | nonnegative unimodal vector, intersection of convex sets | -Lipschitz matrix | Spectraplex

...

...

Sufficient Descent Lemma of gradient descent on L-smooth function

Subdifferential and subgradient

Subgradient method

Proximal bundle method

...

Moreau-Yosida Envelope & Proximal map

Proximal gradient & convergence rate on

-smooth convex functionSolving

-regularized least squares by reweighted least squaresAdaptive restarts

...

Nesterov's optimal convergence rate of 1st-order method on convex smooth functions

Nesterov's Estimate Sequence, part 1: what is it & how to make one

Nesterov's Estimate Sequence, part 2: optimal 1st-order scheme

Convergence rate of

**gradient descent**on convex smooth functionConvergence rate of

**gradient descent**on strongly-convex smooth functionConvergence rate of

**projected gradient method**on -Lipschitz convex functionConvergence rate of

**Nesterov’s accelerated gradient method**on -smooth convex functionConvergence rate of

**Nesterov’s accelerated gradient method**on α-strongly convex -smooth functionConvergence of

**proximal gradient with Nesterov's acceleration / FISTA**

...

...

Convergence analysis of 1st-order method on a quadratic programming problem using dynamic system

Convergence of 4th-order Runge-Kutta update on least square problem

Principle of least action & Euler-Lagrange equation of motion

...

Properties of conjugate

Duality | KKT | Slater’s constraint qualifications

Fast primal-dual proximal gradient algorithm and preconditioning

convergence rate of ADMM ergodic convergence rate of ADMM

...

...

Cubic regularization

Anderson Acceleration

...

...

Convergence of MM

BSUM (only convergence, no rate)

TiTAN

...

Convergence of randomized block coordinate gradient descent on β-smooth convex function

Greedy coordinate descent

Accelerating coordinate descents

...

Kurdyka-Łojasiewicz property

Convergence of PALM on non-convex problem, part 2 : generated sequence converges to a critical point

Inertial Proximal Alternating Linearized Method (iPALM)

Xu-Yin's Block Coordinate Descent Method for Regularized Multiconvex Optimization, hand written

...

...

...

Penalty method is not effective for NNLS – using iteratively reweighed least squares

NNLS by projected gradient descent: acceleration & restart | NNLS PGD mfile | NNLS PGD mfile 2

If A and its inverse are both nonnegative, then A is the permutation of a positive diagonal matrix

Convergence analysis of the multiplicative update NMF algorithm

NMF via projected gradient |

PGD algorithm mfile | APG algorithm mfileNMF via HALS: column-wise exact block coordinate descent |

HALS mfile

...

Nuclear norm is tight convex relaxation of rank function only within the unit ball

Characterization of nuclear norm : nuclear norm is the dual norm of the spectral norm

Understanding the uniqueness of the sol. of the nuclear norm minimization

Singular value thresholding solves

, by von Neumann trace inequality

...

...

...

...

...

...

...