CSCI 6960/4140 Machine Learning and Optimization, Spring 2026
Overview
Modern machine learning formulates goals as optimization problems and applies suitable algorithms to solve them. The first part of the course introduces convex optimization for analyzing deterministic and randomized optimization methods; the second part introduces a selection of modern ML architectures. Homeworks give hands-on algorithmic experience and assess understanding of theoretical aspects. A project gives the students experience with proper research methodology, critically reading the research literature, and effective technical communication.
We will cover topics from two broad areas:
- Convex Optimization Foundations: Convex analysis, optimality conditions, gradient-based methods, proximal algorithms, augmented Lagrangians, ADMM, Frank-Wolfe methods, kernel methods, strong convexity, convergence theory, Newton's method, stochastic gradient descent, and adaptive gradient methods (momentum, Nesterov, AdaGrad, RMSProp, ADAM).
- Modern ML Architectures: Neural networks as computational graphs, MLPs, universal approximation, autoencoders, backpropagation, convolutional neural networks, U-Net and Inception architectures, batch and layer normalization, residual networks, dropout, recurrent neural networks, LSTMs, Transformers, BERT, GPT models, generative adversarial networks, variational autoencoders, and diffusion models.
Course Logistics
The syllabus is available as an archival pdf, and is more authoritative than this website.
Instructor: Alex Gittens (gittea at rpi dot edu)
TA: Huzaifa Arif ( arifh at rpi dot
Lectures: Tue/Fri 12pm-2pm ET in JEC 5119
Questions, Discussions, and Course Material: Piazza
Office Hours: Tue 2pm-3pm ET, Friday 11am-12pm ET in Lally 316
Grading Criteria:
- Midterm, 25% (undergraduate) / 30% (graduate)
- Final Exam, 25% (undergraduate) / 35% (graduate)
- Homework, 25% (undergraduate) / 15% (graduate)
- Project, 25% (undergraduate) / 20% (graduate)
Letter grades will be computed from the semester average. Lower-bound cutoffs for the undergraduates are: A, B, C and D grades are 90%, 80%, 70%, and 60%, respectively. Lower-bound cutoffs for the graduates are: A, B, C grades are 90%, 80%, 70%. These bounds may be moved lower at the instructor's discretion.
Course Materials
Course discussions and announcements are available in Piazza. You are responsible for keeping up with the Piazza for announcements.
Lecture Notes
Each lecture has two PDFs: Handout (no progressive reveals, for self-study) and Presentation (with reveals, for following along in class).
Vanishing/exploding gradients, Xavier/He initialization, dead neurons, batch & layer normalization, skip connections, auxiliary losses.
Latent variable models, the evidence lower bound (ELBO), the reparameterization trick, probabilistic encoders and decoders, VAE architecture.
Score functions, denoising score matching, forward and reverse diffusion processes, noise schedules, signal-to-noise ratio, denoising score matching objective, diffusion training algorithm.
Homeworks
Homework submission links: (Jupyter notebooks only, 5MB limit)
Late assignments will not be accepted, unless you contact the instructor at least two days before the due date to receive a deferral. Deferrals will be granted at the instructor’s discretion.
- Homework 1. Assigned Friday January 23, 2026. Due 11:59pm ET Friday February 6, 2026.
- Homework 2. Assigned Tuesday February 10, 2026. Due 11:59pm ET Tuesday February 24, 2026.
- Homework 3. Assigned Tuesday February 24, 2026. Due 11:59pm ET Tuesday March 17, 2026.
- Homework 4. Assigned Friday March 20, 2026. Due 11:59pm ET Friday April 3, 2026.
- Homework 5. Not assigned, here if you want to take a look.
Project
Each student will complete a final project. Graduate students will complete a research project related to the subject matter of the course, and undergraduate students have the choice to either complete a research project or a pedagogical project. Projects are completed in small groups and culminate in a written report and presentation in ICLR-style format. See the project page for more details.
Supplementary Materials
If you want a refresher on the mathematics assumed in this course, see the free text Mathematics for Machine Learning.