Stanford ME 343 / CME 216 homepage
Machine Learning for Computational Engineering
These is the web site for ME 343/CME 216 Machine Learning in Computational Engineering. This material was created by Eric Darve, with the help of course staff and students.
Syllabus
Policy for late assignments
Extensions can be requested in advance for exceptional circumstances (e.g., travel, sickness, injury, COVID related issues) and for OAE-approved accommodations.
Submissions after the deadline and late by at most 2 days (+48 hours after the deadline) will be accepted with a 10% penalty. No submissions will be accepted 2 days after the deadline.
See Gradescope for all the current assignments and their due dates. Post on Slack if you cannot access the Gradescope class page. We will send you the 6-letter code.
Class modules and learning material
Python tutorials
- Python setup guide
- Google colab guide
- Introduction to Python notebook
- Numpy demo notebook
- Python inheritance demo notebook
- All the files including required text input files
Introduction to ML and SVM
Module 1-Part 1
- 1.1 Brief introduction to ML; Video; Slides
- 1.2 Examples of machine learning; Video; Slides
- 1.3 Supervised learning; Video; Slides
- 1.4 Machine learning in engineering; Video; Slides
- 1.5 Introduction to SVM; Video; Slides
- Reading Assignment 1
Module 1-Part 2
- Python setup guide
- SVM Python notebook
- 1.6 scikit-learn; Video; Slides
- 1.7 Soft-margin; Video; Slides
- 1.8 Over-fitting; Video; Slides
- 1.9 Training and validation sets; Video; Slides
- 1.10 Kernel trick; Video; Slides
- Reading Assignment 2
- Homework 1
Deep Learning
Module 2
- TensorFlow Python notebook DNN regression demo
- 2.1 Perceptron; Video; Slides
- 2.2 MLP; Video; Slides
- 2.3 TensorFlow/Keras; Video; Slides
- 2.4 Sequential API; Video; Slides
- 2.5 Functional API; Video; Slides
- Subclassing Python notebook inheritance demo
- 2.6 Subclassing; Video; Slides
- Reading Assignment 3
Module 3-Part 1
- Python notebook DNN regularization demo
- 3.1 Loss function; Video; Slides
- 3.2 Cross-entropy; Video; Slides
- 3.3 TensorFlow loss functions; Video; Slides
- 3.4 Backpropagation; Video; Slides
- 3.5 Backpropagation formula; Video; Slides
- 3.6 Learning rate for training; Video; Slides
- 3.7 Empirical method for learning rate; Video; Slides
- 3.8 Overfitting; Video; Slides
- 3.9 DNN initializers; Video; Slides
- 3.10 Regularization; Video; Slides
- Reading Assignment 4
- Homework 2
Module 3-Part 2
- Saddle point demo notebook
- ADAGRAD notebook
- 3.11 Stochastic Gradient Descent; Video; Slides
- 3.12 Saddle points; Video; Slides
- 3.13 Momentum; Video; Slides
- 3.14 Adagrad; Video; Slides
- 3.15 RMSProp and Adam; Video; Slides
- Reading Assignment 5
- Homework 3
Module 3-Part 3
- 3.16 Trust region; Video; Slides
- 3.17 BFGS; Video; Slides
- 3.18 L-BFGS; Video; Slides
- Reading Assignment 6
- Homework 4
Physics Informed Machine Learning
Module 4
- Automatic Differentiation in TensorFlow notebook
- Physics Informed Learning with TensorFlow notebook
- 4.1 Physics Informed Machine Learning (PIML); Video; Slides
- 4.2 TensorFlow Automatic Differentiation (TF_AD); Video; Slides
- 4.3 Physics Informed ML using TensorFlow (TF_PIML); Video; Slides
- Reading Assignment 7
- Homework 5
Module 5
Automatic differentiation, physics informed learning using ADCME
The slides are assembled into a single PDF file. Each lecture video covers one section in the PDF.
- Module 5, Automatic Differentiation for Computational Engineering; Videos; Slides
- Reading Assignment 8
Module 6
Inverse modeling using ADCME
- Module 6, Inverse Modeling using ADCME; Videos; Slides
- Reading Assignment 9
- Homework 6
Final project
Reading material
Books
- Deep learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville
- Deep learning with Python by François Chollet
- Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow : concepts, tools, and techniques to build intelligent systems by Aurélien Géron
- Numerical optimization by Jorge Nocedal and Stephen Wright
- Fundamentals of deep learning : designing next-generation machine intelligence algorithms by Nikhil Buduma
- Elements of statistical learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman
- Deep learning: an introduction for applied mathematicians by Catherine Higham and Desmond Higham
- Machine learning: a probabilistic perspective by Kevin Murphy (in searchworks)
- Deep learning illustrated: a visual, interactive guide to artificial intelligence by Jon Krohn
- Neural networks and deep learning by Michael Nielsen
- Foundations of machine learning by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar
- Neural networks and learning machines by Simon Haykin
- The matrix cookbook by Kaare Petersen and Michael Pedersen
Video tutorials
- Introduction to deep learning: concepts and fundamentals by Laura Graesser
- Introduction to deep learning models with TensorFlow: learn how to work with TensorFlow to create and run a TensorFlow graph, and build a deep learning model by Lucas Adams
- Deep learning with TensorFlow: applications of deep neural networks to machine learning tasks by Jon Krohn
Review papers
- LeCun, Bengio and Hinton, Deep learning, Nature, 521:436-444, 2015
- Schmidhuber, Deep learning in neural networks: an overview, Neural Networks, 61:85-117, 2015
- Automatic differentiation in machine learning: a survey by Atılım Günes Baydin, Barak Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind
- A review of the adjoint-state method for computing the gradient of a functional with geophysical applications by R.-E. Plessix
Online classes and tutorials
- Introduction to Deep Learning, MIT
- fast.ai
- Machine Learning 2014-2015, Oxford, by Nando de Freitas
Links
- ADCME (wiki page) developed by Kailai Xu and Prof. Darve
- DeepXDE developed by Lu Lu and Prof. Karniadakis
- List of books and tutorials on ML
- Online courses
- TensorFlow notebooks by Jon Krohn
- TF2 notebooks by Jon Krohn
- Deep learning illustrated notebooks by Jon Krohn
- TensorFlow playground
- MNIST visualization by Adam Harley
- Distill, a journal for machine learning visualizations