COMP6BBB Optimisation for Machine Learning, 2025 Spring

• under construction

AICE1004 Mathematics for Artificial Intelligence and Computer Engineering, 2024 Fall

• under construction

COMP1215 Foundations of Computer Science, Uni. Southampton, 2023 Fall

• Combinatorics | Graph | Probability | Statistics notes
• Key concepts & book | Madbook

CO327 Deterministic OR Models, Uni. Waterloo, 2022-spring

MARO201 UMONS, Belgium, 2019-2020

• Gradient descent solving quadratic problems | video

• Convex, strongly conve, Lipschitz, Lipschitz gradient, Lipschitz Hessian, lower semicontinuous function

• 1st-order Taylor approximation of convex function and Bregman divergence

• Sandwich theorem for β-smooth function

• Monotonictiy of gradient of strongly-convex smooth function

• 1st-order optimality condition for constrained optimization

• Common glossaries in optimization

• Coercive function

• proj operator is firmly non-expensive & obtuse angle criterion

• Augmented objective function & methods for constrained optimization

• , and

• Prox of norm, Moreau's decomposition, conjugate of norm = indicator function of dual norm

• Characterizing subdifferential of norm

• Projection onto nonnegative orthant, rectangular box and polyhedron | ball | ball | simplex | nonnegative unimodal vector, intersection of convex sets | -Lipschitz matrix | Spectraplex

• Linear Optimization

• Solving piece-wise linear equation

• Motivation of mirror descent 1: Gradient vector lives in the dual space

• Motivation of mirror descent 2: a geometry example

• Mirror descent algorithm

• Gradient flow gives gradient update, proximal point update, extra-gradient update and 4th order Runge-Kutta update

• Convergence of gradient flow in continuous time

• Convergence analysis of 1st-order method on a quadratic programming problem using dynamic system

• Convergence of 4th-order Runge-Kutta update on least square problem

• Deriving ODE model of Nesterov's accelerated gradient

• Principle of least action & Euler-Lagrange equation of motion

• Lagrangian Mutiplier

• What is conjugate: ?

• Properties of conjugate

• Deriving the dual problem using conjugate

• Duality | KKT | Slater’s constraint qualifications

• Fast dual proximal gradient method

• Fast primal-dual proximal gradient algorithm and preconditioning

• How ADMM was developed

• Convergence of ADMM

• convergence rate of ADMM

• ergodic convergence rate of ADMM

• Where does operator splitting come from

• Fixed point of average of non-expansive operator

• Convergence of a proximal quasi-Newton's method

• Cubic regularization

• Sol to line search of Anderson Acceleration subproblem

• Anderson Acceleration

• Functions that has Lipschitz Hessian

• Big finite sum

• Confusing naming in the stochastic methods

• Stochastic Gradient Method (SG)

• MM: the technique of surrogate

• Solving total variation norm regularised least square by MM

• Convergence of MM

• BSUM (only convergence, no rate)

• TiTAN

• Coordinate Descent - an introduction

• Solving nonegative least sqaures with L1-regularization

• Convergence of randomized block coordinate gradient descent on β-smooth convex function

• Greedy coordinate descent

• Accelerating coordinate descents

• Polyak-Łojasiewicz inequality

• Proximal Polyak-Łojasiewicz inequality

• Kurdyka-Łojasiewicz property

• Proximal Alternating Linearized Method (PALM)

• Convergence of PALM on nonconvex problem, part 1 : generated sequence is non-increasing and the trajectory has finite length

• Convergence of PALM on non-convex problem, part 2 : generated sequence converges to a critical point

• Inertial Proximal Alternating Linearized Method (iPALM)

• Xu-Yin's Block Coordinate Descent Method for Regularized Multiconvex Optimization, hand written

• Convergence of TRM on unconstrained smooth optimization

• Continuous total variation in infinite dimensional space

• Discrete total variation in finite dimensional space

• Solving total variation norm regularised least square by Proximal gradient method with ADMM

• Solving total variation norm regularised least square by MM

• Error computation in matrix factorization

• Hyperspectral unmixing & NMF

• Short-time Fourier transform

• ABC of tensors

• Canonical Polyadic Decomposition: basic notations

• Clifford algebra in