10-708 (CMU) Probabilistic Graphical Models

Probabilistic Graphical Models

Overview


Many of the problems in artificial intelligence, statistics, computer systems, computer vision, natural language processing, and computational biology, among many other fields, can be viewed as the search for a coherent global conclusion from local information. The probabilistic graphical model's framework provides a unified view of this wide range of problems, enabling efficient inference, decision-making and learning in problems with a very large number of attributes and huge datasets. This graduate-level course will provide you with a strong foundation for both applying graphical models to complex problems and for addressing core research topics in graphical models.  The class will cover three aspects: The core representation, including Bayesian and Markov networks, and dynamic Bayesian networks; probabilistic inference algorithms, both exact and approximate; and, learning methods for both the parameters and the structure of graphical models. Students entering the class should have a pre-existing working knowledge of probability, statistics, and algorithms, though the class has been designed to allow students with a strong numerate background to catch up and fully participate.  It is expected that after taking this class, the students should have obtained sufficient working knowledge of multivariate probabilistic modeling and inference for practical applications, should be able to formulate and solve a wide range of problems in their own domain using GM and can advance into more specialized technical literature by themselves.  Students are required to have successfully completed 10701 or 10715, or an equivalent class.

Where and When

 

People


Instructor:

  • Kayhan Batmanghelich
    Office Hour: Gates-Hillman Center 8228 on Tuesdays  1:30 - 2:30 pm

Administrative Assistant:

  • Noreen Doloughty

Teaching Assistants:

If you wish to email only the instructors, the email is 10708Spring18@gmail.com.

 

Announcements


  • If you have any questions about class policies or course material, please use piazza or you can email all of the instructors at 10708Spring18@gmail.com. Please use this list instead of individual email addresses to ensure a prompt response.
  • The Homework 0 is out. The deadline is 7:00 pm Jan 23. No late HW will be accepted. 
  • Please find the scribe template here.
  • The project proposal will be due by 7 PM on Friday, February 16th, and should be submitted via Gradescope.
  • Homework 1 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is Feb 18 (11:59 pm).
  • Homework 2 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is Mar 21 (11:59 pm).
  • The project midway report will be due 11:59 pm, April 13th. See Piazza for more detail.
  • Homework 3 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is April 20 (11:59 pm).

 

Lectures


 

Date Lecture Scribe Readings Announcements
Jan 16, 2018

Lecture 1 (Kayhan) - SlideVideo

Probabilistic Graphical Model:
A view from moon

None None HW0 is out.
Module 1: Representation
Jan 18, 2018

Lecture 2 (Kayhan) - Slide, Annotated, Video

Directed GMs: Bayesian Networks

Notes (by Sumedha Singla) 1. Koller and Friedman Textbook, Ch. 3
2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 3 
 
Jan 23, 2018

Lecture 3 (Kayhan) - Slide, Annotated, Video

Undirected Graphic Model

Notes (by Arjun Sharma) 1. Koller and Friedman Textbook, Ch. 4
2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 4
HW 0 is due today.
Jan 25, 2018

Lecture 4 (Kun Zhang) - Slide, Annotated, Video

Causal Graphic Model

Notes (by M. Malik and N. Shajarisales) None The reading summary is due in a week.
Module 2: Classical Methods of Inference & Learning
Jan 30, 2018

Lecture 5 (Kayhan) - Slide, Annotated, Video

Algorithms for Exact Inference

Notes (by MF. Chang and D. Rajagopal)

David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6

Other resources: (1)

 
Feb 1, 2018

Lecture 6 (Kayhan) - Slide, Annotated, Video

Factor graph, message passing, and Junction Tree

Notes (S. Greg)

David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6

Other resources: (1)

 
Feb 6, 2018

Lecture 7 (Kayhan) - Slide, Annotated, My Notes, Class Notes, Video

Exponential families and friends: Learning the parameters of a fully observed BN

Notes (A. Kamath)

Kevin Murphy, Machine Learning A Probabilistic Perspective, Ch. 9

or

Jordan Textbook, Ch. 8, Ch. 9

 
Feb 8, 2018

Lecture 8 (Kayhan) - Slide, Annotated, My Notes, Class Notes, Video

Learning the parameters of UGM

Notes (C. Zhou and C. Zhang) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9, Section 6  
Feb 13, 2018

Lecture 9 (Kayhan) - Slide, Video

EM and partially observed GM

None

Jordan Textbook, Ch. 10

or  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 11

The reading summary is due in a week.
Module 3: Graphical Model in Application & Learning
Feb 15, 2018

Lecture 10 (Kayhan) - Slide, Annotated, Video

HHM and CRF

Notes (B. Lengerich and M. Kleyman)

H. Wallach, Conditional Random Fields: An Introduction

J. Lafferty et al., Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

 
Feb 20, 2018

Lecture 11 (Kayhan) - Slides, Annotated, Video

CRF (Cont'd) + Intro to Topic Models

Notes (A. Yang and J. He)

D. Blei et al., Latent Dirichlet Allocation (Sections 1-4)

 
Feb 22, 2018

Lecture 12 (Kayhan) - Slides, Annotated, Class Notes, Video

Intro to Topic Models (Cont'd), Factor Analysis and (maybe State Space)

Notes (G. Plumb and A. Rumack)

Jordan Textbook, Ch. 14

Jordan Textbook, Ch. 15 (if we finished State Space)

 
Feb 27, 2018

 No Class

    HW 2 is out.
Mar 1, 2018

Lecture 13 (Kun Zhang) - Slides, Video

Learning Structure of a Graphical Model 

None

This short overview paper

and 

David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9.5

The reading summary is due in a week.
Module 4: Approximate Inference & Learning
Mar 6, 2018

Lecture 14 (Kayhan) - Slides, AnnotatedVideo

Loopy Belief Propagation

Notes (K. Xiong, C. Malaviya)

M. Wainwright and M. Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 4.1

 
Mar 8, 2018

Lecture 15 (Kayhan) - Slides, Annotated, Video

Mean field Approximation

Notes (Y. Hung, H. Tsai) D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 1-26  
Mar 20, 2018

Lecture 16 (Kayhan) - Slides, Annotated, Video

Stochastic Gradient Descent, SVI, and scalability

Notes (Y. Feng, J. Wu) D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 1-26  
Mar 22, 2018

Lecture 17 (Kayhan) - Slides, Jupyter-notebook, Video

Approximate Inference Monte Carlo Methods

Notes (B. Paria, P. Chikersal) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27  
Mar 27, 2018

Lecture 18 (Kayhan) - Slides, AnnotatedMy Notes, Video

MCMC and Gibbs sampling

None

David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27

Convergence Diagnostics For MCMC

 
Mar 29, 2018

Lecture 19 (Kayhan) - Slides, Annotated, Video

Hamiltonian Monte Carlo

Notes (B. Lyu)

Slice Sampling: David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Section 29.7

HMC: A Conceptual Introduction to Hamiltonian Monte Carlo

The reading summary is due in a week.
Module 5: Deep Learning and Graphical Models & Learning
Apr 3, 2018

Lecture 20 (Kayhan) - Slides, Annotated, Video

Introduction to Deep Learning

Notes (A. Alavi, Yu Chen)

Deep Learning Book, Ch. 6.2-5, 20.3-4

Optional:

 
Apr 5, 2018

 No Class

     
Apr 10, 2018

Lecture 21 (Kayhan) - Slides, Annotated, Video

A Hybrid: Deep Learning and Graphical Models

Notes (P. Liang, A. Rayasam)

Diederik P. Kingma, Variational Inference & Deep Learning: A New Synthesis,  Ch. 2

Generative Adversarial Nets: Goodfellow et al., 2014.

 
Apr 12, 2018

Lecture 22 (Kayhan, Mingming Gong) - Slides, Annotated, Video

A Hybrid DL and GM (cont’d) +
Applications in Computer Vision

Notes (S. Bai, C.-K. Yeh) None  
Apr 17, 2018

Lecture 23 (Mingming Gong, Kayhan) - SlidesVideo

Applications in Computer Vision (cont’d) + Gaussian Process

None None The reading summary is due in a week.
Module 6: Spectral and non-parametric view & Learning
Apr 24, 2018

Lecture 24 (Kayhan) - Slides, Jupyter-Notebook, Annotated, Video

Gaussian Process

Notes (A. Siddhant, S. Ghosh, C. Nagpal)

David Barber, Bayesian Reasoning, and Machine Learning, Ch. 19

Optional: L. Song. Learning via Hilbert space embedding of distributions, Sec. 2.1, 2.2, 3.1, 3.2

 
Apr 26, 2018

Lecture 25 (Forough Arabshahi) - Slides, Annotated, Video

Spectral Methods

None None