CSCI 6960/4140 Machine Learning and Optimization, Spring 2026

Overview

Modern machine learning formulates goals as optimization problems and applies suitable algorithms to solve them. The first part of the course introduces convex optimization for analyzing deterministic and randomized optimization methods; the second part introduces a selection of modern ML architectures. Homeworks give hands-on algorithmic experience and assess understanding of theoretical aspects. A project gives the students experience with proper research methodology, critically reading the research literature, and effective technical communication.

We will cover topics from two broad areas:

Course Logistics

The syllabus is available as an archival pdf, and is more authoritative than this website.

Instructor: Alex Gittens (gittea at rpi dot edu)

TA: Huzaifa Arif ( arifh at rpi dot edu)

Lectures: Tue/Fri 12pm-2pm ET in JEC 5119

Questions, Discussions, and Course Material: Piazza

Office Hours: Tue 2pm-3pm ET, Friday 11am-12pm ET in Lally 316

Grading Criteria:

Rubrics are available for the project component of the course.

Letter grades will be computed from the semester average. Lower-bound cutoffs for the undergraduates are: A, B, C and D grades are 90%, 80%, 70%, and 60%, respectively. Lower-bound cutoffs for the graduates are: A, B, C grades are 90%, 80%, 70%. These bounds may be moved lower at the instructor's discretion.

Course Materials

Course discussions and announcements are available in Piazza. You are responsible for keeping up with the Piazza for announcements.

Lecture Notes

Each lecture has two PDFs: Handout (no progressive reveals, for self-study) and Presentation (with reveals, for following along in class).

Part 1: Optimization Foundations
L1 Course logistics & ERM ▼

Empirical risk minimization, examples of ML as optimization, closed-form solutions vs gradient descent.

L2 Hypothesis spaces & SVMs ▼

Hypothesis spaces, loss functions, regularization, soft-margin SVMs.

L3 MLE & MAP estimation ▼

Justifying ERM via law of large numbers, learning parameterized models via maximum likelihood and maximum a posteriori estimation.

L4 Classification & cross-entropy ▼

Binary and multi-class classification, cross-entropy loss, KL divergence.

L5 Gradient descent & convexity ▼

Oracle methods, gradient descent algorithm, convexity definitions and characterizations of convex functions.

L6 Convergence theory for GD ▼

Smoothness, descent lemma, strong convexity, condition number, sublinear convergence for smooth convex, exponential convergence for smooth strongly convex.

L7 Stochastic gradient descent ▼

SGD algorithm, per-iteration speedup, progress vs noise, noise floor, Robbins–Monro conditions, O(1/T) and O(1/√T) rates, Jensen’s inequality, iterate averaging, minibatch SGD, Polyak averaging.

L8 Subdifferentials & subgradients ▼

Extended-value functions, lower semicontinuity, extreme value theorem, subdifferentials, subgradients, computation examples.

L9 Subgradient methods & projections ▼

Subdifferential calculus, convex indicator functions, normal cones, Fermat’s condition, subgradient descent, projections, projected subgradient descent, total variation denoising.

L10 Newton’s method ▼

Local quadratic convergence, globalization via line search and trust region, Armijo’s condition, backtracking, damped Newton, trust region Newton, hybrid GD+Newton.

L11 Momentum & adaptive methods ▼

Momentum SGD, Nesterov acceleration, preconditioning, AdaGrad, RMSProp, Adam, AdamW, element-wise vs spectral preconditioning.

Part 2: Neural Network Architectures
L12 Neural networks & MLPs ▼

NNs as computational DAGs, feature learning, fully connected feedforward NNs, MLPs, necessity of nonlinear activations, universal approximation, depth separation.

L13 Nonconvex optimization for NNs ▼

History of NNs, enabling factors for modern deep learning, ε-stationarity, GD/SGD for L-smooth nonconvex functions, noise floor, saddle points, overparameterization, double descent, implicit regularization.

L14 Backpropagation & automatic differentiation ▼

Gradients of compositions, topological orderings, Jacobians, multivariate chain rule, reverse-mode AD, backpropagation for MLPs, time/memory cost, PyTorch as AD engine.

L15 Convolutional neural networks ▼

Inductive biases (locality, translation equivariance, parameter efficiency), convolution, padding/striding/pooling, multichannel architectures, im2col, receptive fields, transfer learning, LeNet-5.

L16 Training deep networks ▼

Vanishing/exploding gradients, Xavier/He initialization, dead neurons, batch & layer normalization, skip connections, auxiliary losses.

L17 Sequence models & attention ▼

RNNs, tokenization, Elman RNNs, BPTT, LSTMs, encoder–decoder architectures, information bottleneck, QKV attention, BPE tokenization.

L18 Transformers ▼

Self-attention, positional embeddings, multi-headed self-attention, position-wise FFNs, encoder/decoder blocks, cross-attention, causal masking, teacher forcing, encoder-only / decoder-only / full encoder-decoder architectures.

L19 Modern LLM architectures ▼

Dominance of decoder-only architectures, bottlenecks to scaling, RMSNorm, SwiGLU FFNs, RoPE attention, Grouped-Query Attention, the modern LLM template.

L20 Training and aligning LLMs ▼

Full LLM training pipeline: pretraining, supervised fine-tuning (SFT), and alignment. KL-constrained reward maximization and DPO for alignment. LoRA for parameter-efficient fine-tuning.

L21-22 Variational Autoencoders ▼

Latent variable models, the evidence lower bound (ELBO), the reparameterization trick, probabilistic encoders and decoders, VAE architecture.

L23 Diffusion Models I — Score Matching ▼

Score functions, denoising score matching, forward and reverse diffusion processes, noise schedules, signal-to-noise ratio, denoising score matching objective, diffusion training algorithm.

L24 Diffusion Models II — Fast Sampling, Guidance, Scalable Architectures ▼

fast sampling with DDIM, classifier-free guidance, latent diffusion (VAE + U-Net), DiT (Diffusion Transformers), the full Stable Diffusion pipeline.

L25 Flow Matching: Learning the Transport Directly ▼

Transport viewpoint of generation, continuity equation for density-level mass conservation, conditional flow matching (CFM) training and the regression principle, CFM theorem and ODE sampling, comparison with DDPM and DDIM.

Homeworks

Homework submission links: (Jupyter notebooks only, 5MB limit)

Late assignments will not be accepted, unless you contact the instructor at least two days before the due date to receive a deferral. Deferrals will be granted at the instructor’s discretion.

Project

Each student will complete a final project. Graduate students will complete a research project related to the subject matter of the course, and undergraduate students have the choice to either complete a research project or a pedagogical project. Projects are completed in small groups and culminate in a written report and presentation in ICLR-style format. See the project page for more details.

Supplementary Materials

If you want a refresher on the mathematics assumed in this course, see the free text Mathematics for Machine Learning.