Probabilistic Graphical Models
Overview
Many of the problems in artificial intelligence, statistics, computer systems, computer vision, natural language processing, and computational biology, among many other fields, can be viewed as the search for a coherent global conclusion from local information. The probabilistic graphical model's framework provides a unified view of this wide range of problems, enabling efficient inference, decisionmaking and learning in problems with a very large number of attributes and huge datasets. This graduatelevel course will provide you with a strong foundation for both applying graphical models to complex problems and for addressing core research topics in graphical models. The class will cover three aspects: The core representation, including Bayesian and Markov networks, and dynamic Bayesian networks; probabilistic inference algorithms, both exact and approximate; and, learning methods for both the parameters and the structure of graphical models. Students entering the class should have a preexisting working knowledge of probability, statistics, and algorithms, though the class has been designed to allow students with a strong numerate background to catch up and fully participate. It is expected that after taking this class, the students should have obtained sufficient working knowledge of multivariate probabilistic modeling and inference for practical applications, should be able to formulate and solve a wide range of problems in their own domain using GM and can advance into more specialized technical literature by themselves. Students are required to have successfully completed 10701 or 10715, or an equivalent class.
Where and When
 Time: Tuesday, Thursday 12:00  1:20 pm
 Location: GatesHillman Center 4307
 Piazza Link: https://piazza.com/cmu/spring2018/10708/home
 GradeScope Link: https://gradescope.com/courses/14138
People
Instructor:
 Kayhan Batmanghelich
Office Hour: GatesHillman Center 8228 on Tuesdays 1:30  2:30 pm
Administrative Assistant:
 Noreen Doloughty
Teaching Assistants:

Yifeng Tao (yifengt@andrew.cmu.edu)

Xiongtao Ruan (xruan@andrew.cmu.edu)

Yuanning Li (yuanninl@andrew.cmu.edu)
If you wish to email only the instructors, the email is 10708Spring18@gmail.com.
Announcements
 If you have any questions about class policies or course material, please use piazza or you can email all of the instructors at 10708Spring18@gmail.com. Please use this list instead of individual email addresses to ensure a prompt response.
 The Homework 0 is out. The deadline is 7:00 pm Jan 23. No late HW will be accepted.
 Please find the scribe template here.
 The project proposal will be due by 7 PM on Friday, February 16th, and should be submitted via Gradescope.
 Homework 1 is released. You can view it on the course page: https://piazza.com/cmu/
spring2018/10708/resources . The due date is Feb 18 (11:59 pm).  Homework 2 is released. You can view it on the course page: https://piazza.com/cmu/
spring2018/10708/resources . The due date is Mar 21 (11:59 pm).  The project midway report will be due 11:59 pm, April 13th. See Piazza for more detail.
 Homework 3 is released. You can view it on the course page: https://piazza.com/cmu/
spring2018/10708/resources . The due date is April 20 (11:59 pm).
Lectures
Date  Lecture  Scribe  Readings  Announcements  

Jan 16, 2018 
Lecture 1 (Kayhan)  Slide, Video Probabilistic Graphical Model: 
None  None  HW0 is out.  
Module 1: Representation  
Jan 18, 2018 
Lecture 2 (Kayhan)  Slide, Annotated, Video Directed GMs: Bayesian Networks 
Notes (by Sumedha Singla)  1. Koller and Friedman Textbook, Ch. 3 2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 3 

Jan 23, 2018 
Lecture 3 (Kayhan)  Slide, Annotated, Video Undirected Graphic Model 
Notes (by Arjun Sharma)  1. Koller and Friedman Textbook, Ch. 4 2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 4 
HW 0 is due today.  
Jan 25, 2018 
Lecture 4 (Kun Zhang)  Slide, Annotated, Video Causal Graphic Model 
Notes (by M. Malik and N. Shajarisales)  None  The reading summary is due in a week.  
Module 2: Classical Methods of Inference & Learning  
Jan 30, 2018 
Lecture 5 (Kayhan)  Slide, Annotated, Video Algorithms for Exact Inference 
Notes (by MF. Chang and D. Rajagopal) 
David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6 Other resources: (1) 

Feb 1, 2018 
Lecture 6 (Kayhan)  Slide, Annotated, Video Factor graph, message passing, and Junction Tree 
Notes (S. Greg) 
David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6 Other resources: (1) 

Feb 6, 2018 
Lecture 7 (Kayhan)  Slide, Annotated, My Notes, Class Notes, Video Exponential families and friends: Learning the parameters of a fully observed BN 
Notes (A. Kamath) 
Kevin Murphy, Machine Learning A Probabilistic Perspective, Ch. 9 or Jordan Textbook, Ch. 8, Ch. 9 

Feb 8, 2018 
Lecture 8 (Kayhan)  Slide, Annotated, My Notes, Class Notes, Video Learning the parameters of UGM 
Notes (C. Zhou and C. Zhang)  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9, Section 6  
Feb 13, 2018 
Lecture 9 (Kayhan)  Slide, Video EM and partially observed GM 
None 
Jordan Textbook, Ch. 10 or David Barber, Bayesian Reasoning, and Machine Learning, Ch. 11 
The reading summary is due in a week.  
Module 3: Graphical Model in Application & Learning  
Feb 15, 2018 
Lecture 10 (Kayhan)  Slide, Annotated, Video HHM and CRF 
Notes (B. Lengerich and M. Kleyman) 
H. Wallach, Conditional Random Fields: An Introduction J. Lafferty et al., Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data 

Feb 20, 2018 
Lecture 11 (Kayhan)  Slides, Annotated, Video CRF (Cont'd) + Intro to Topic Models 
Notes (A. Yang and J. He) 
D. Blei et al., Latent Dirichlet Allocation (Sections 14) 

Feb 22, 2018 
Lecture 12 (Kayhan)  Slides, Annotated, Class Notes, Video Intro to Topic Models (Cont'd), Factor Analysis and (maybe State Space) 
Notes (G. Plumb and A. Rumack) 
Jordan Textbook, Ch. 14 Jordan Textbook, Ch. 15 (if we finished State Space) 

Feb 27, 2018 
No Class 
HW 2 is out.  
Mar 1, 2018 
Lecture 13 (Kun Zhang)  Slides, Video Learning Structure of a Graphical Model 
None 
and David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9.5 
The reading summary is due in a week.  
Module 4: Approximate Inference & Learning  
Mar 6, 2018 
Lecture 14 (Kayhan)  Slides, Annotated, Video Loopy Belief Propagation 
Notes (K. Xiong, C. Malaviya) 
M. Wainwright and M. Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 4.1 

Mar 8, 2018 
Lecture 15 (Kayhan)  Slides, Annotated, Video Mean field Approximation 
Notes (Y. Hung, H. Tsai)  D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 126  
Mar 20, 2018 
Lecture 16 (Kayhan)  Slides, Annotated, Video Stochastic Gradient Descent, SVI, and scalability 
Notes (Y. Feng, J. Wu)  D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 126  
Mar 22, 2018 
Lecture 17 (Kayhan)  Slides, Jupyternotebook, Video Approximate Inference Monte Carlo Methods 
Notes (B. Paria, P. Chikersal)  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27  
Mar 27, 2018 
Lecture 18 (Kayhan)  Slides, Annotated, My Notes, Video MCMC and Gibbs sampling 
None 
David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27 

Mar 29, 2018 
Lecture 19 (Kayhan)  Slides, Annotated, Video Hamiltonian Monte Carlo 
Notes (B. Lyu) 
Slice Sampling: David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Section 29.7 
The reading summary is due in a week.  
Module 5: Deep Learning and Graphical Models & Learning  
Apr 3, 2018 
Lecture 20 (Kayhan)  Slides, Annotated, Video Introduction to Deep Learning 
Notes (A. Alavi, Yu Chen) 
Deep Learning Book, Ch. 6.25, 20.34 Optional:


Apr 5, 2018 
No Class 

Apr 10, 2018 
Lecture 21 (Kayhan)  Slides, Annotated, Video A Hybrid: Deep Learning and Graphical Models 
Notes (P. Liang, A. Rayasam) 
Diederik P. Kingma, Variational Inference & Deep Learning: A New Synthesis, Ch. 2 Generative Adversarial Nets: Goodfellow et al., 2014. 

Apr 12, 2018 
Lecture 22 (Kayhan, Mingming Gong)  Slides, Annotated, Video A Hybrid DL and GM (cont’d) + 
Notes (S. Bai, C.K. Yeh)  None  
Apr 17, 2018 
Lecture 23 (Mingming Gong, Kayhan)  Slides, Video Applications in Computer Vision (cont’d) + Gaussian Process 
None  None  The reading summary is due in a week.  
Module 6: Spectral and nonparametric view & Learning  
Apr 24, 2018 
Lecture 24 (Kayhan)  Slides, JupyterNotebook, Annotated, Video Gaussian Process 
Notes (A. Siddhant, S. Ghosh, C. Nagpal) 
David Barber, Bayesian Reasoning, and Machine Learning, Ch. 19 Optional: L. Song. Learning via Hilbert space embedding of distributions, Sec. 2.1, 2.2, 3.1, 3.2 

Apr 26, 2018 
Lecture 25 (Forough Arabshahi)  Slides, Annotated, Video Spectral Methods 
None  None 