Lectures

In this session we will discuss the IBM word alignment models 1 and 2. We will view them as constrained mixture models. After motivating the models intuitively, we will develop them formally and devise an EM parameter estimation algorithm for them. We will also try to understand why the resulting optimisation problem contains multiple optima.

• Illustration

• Videopgm

• Philip Schulz has made available a nice tutorial which is the basis for his lecture.

• Wilker Aziz has made available some notes on the derivation of the M-step for IBM1-2.

• In order to get a more concrete view of implementation, including pseudo-codes for the EM procedure, read M. Collins’s lecture notes.

• The original IBM paper is very dense, but if you want to go around teaching IBM models you should probably read it a few times.

• R. Moore (2004) discusses a number of problems with IBM model 1 involving 1) null words, 2) rare words, and 3) initialisation for EM. If you can understand the problems he raises and the solutions he proposes, you are really getting it about IBM1. Also, implementing some of his improvements will be worth some extra points in project 1.

• Toutanova et al. (2011) discuss the non-strict convexity of IBM1. This is related to the non-identifiability problem discussed in class.

• Vogel et al. 1996 revisit the assumption that alignments links are independent.

• Schulz et al, 2016 revisit NULL alignments in IBM1-2.

In this session we will continue the exposition of IBM 1 and 2 models for word alignment. We will view them as constrained mixture models. After motivating the models intuitively, we will develop them formally and devise an EM parameter estimation algorithm for them. We will also try to understand why the resulting optimisation problem contains multiple optima.

• Illustration

• Videoibm

• Philip Schulz has made available a nice tutorial which is the basis for his lecture.

• Wilker Aziz has made available some notes on the derivation of the M-step for IBM1-2.

• In order to get a more concrete view of implementation, including pseudo-codes for the EM procedure, read M. Collins’s lecture notes.

• The original IBM paper is very dense, but if you want to go around teaching IBM models you should probably read it a few times.

• R. Moore (2004) discusses a number of problems with IBM model 1 involving 1) null words, 2) rare words, and 3) initialisation for EM. If you can understand the problems he raises and the solutions he proposes, you are really getting it about IBM1. Also, implementing some of his improvements will be worth some extra points in project 1.

• Toutanova et al. (2011) discuss the non-strict convexity of IBM1. This is related to the non-identifiability problem discussed in class.

• Vogel et al. 1996 revisit the assumption that alignments links are independent.

• Schulz et al, 2016 revisit NULL alignments in IBM1-2.

We present a Bayesian version of IBM1 which has a conjugate Dirichlet prior on the translation parameters. We exploit the conjugacy of the resulting model to develop an efficient variational inference algorithm. During the lecture we will explicitly introduce the logic of variational inference. Students should acquaint themselves with basic properties of exponential families beforehand.

• IBM1-2

• VB for IBM1

• Videovb

• The lecture will very closely follow this script. Students should read it before the lecture. We may decide to skip some details from the script in the lecture.

• The same model but with a Gibbs sampler for inference is presented in Mermer and Saraclar (2011).

• An outstanding account of variational methods with lots of references to recent developments is given in David Blei’s tutorial.

• Chapter 2 of Matthew Beal’s PhD thesis gives a very clear development of variational inference for Bayesian models.

• Bayesian estimation for NLP

• For Bayesian inference without VI, check this MCMC tutorial

• Dyer et al. 2013 revisit the parameterisation of IBM model 2. Their model, called ‘fast_align’ achieves state-of-the-art performance and a robust and efficient implementation is available.

• Bayesian Word Alignment for Statistical Machine Translation

In this lecture, we will show a different parameterisation of categorical distributions in terms of logistic distributions.

• Videonn

• Kocisky et al., (2014).

• Berg-Kirkpatrick et al., (2010).

This lecture presents probabilistic modelling with neural networks, we will start with maximum likelihood estimation based on fully observed data, then we will discuss the challenges of learning distributions over observed and unobserved (latent) data. We will then talk about variational inference and how it may help us estimate deep generative models.

• Videovi

• Neural Variational Inference and Learning in Belief Networks

• Variational Inference:

• Chapter 2 of Matthew Beal’s PhD thesis gives a very clear development of variational inference for Bayesian models.

• An outstanding account of variational methods with lots of references to recent developments is given in David Blei’s tutorial.

This lecture will present the variational auto-encoder, an important deep generative model that makes a building block for many others. We will also learn about the reparameterised gradient, a major development for efficient estimation of DGMs.

• Videovae

• Gaussian distribution

• The original: Kingma and Welling (2014) and Rezend et al (2014)

• Variational auto-encoders:

• A tutorial (disclaimer: not my favourite!)

• More on reparameterisation (advanced!): Titsias and Lazaro-Gredilla (2014) and Ruiz et al (2016)

• Automatic Differentiation Variational Inference

This lecture will present deep latent variable models of word representation

• Videoembed

• CBOW and Skip-gram: Distributed Representations of Words and Phrases and their Compositionality

• Skip-gram is a binary classifier: word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method

• Generative modelling via lexical alignment: Learning Bilingual Word Representations by Marginalizing Alignments

• Embedding Words as Distributions with a Bayesian Skip-gram Model

• Deep Generative Model for Joint Alignment and Word Representation

• Auto-encoding variational inference for topic models

• Semantic Parsing with Semi-Supervised Sequential Autoencoders

This lecture will present generative models over natural language sentences.

• Videolm

• Recurrent neural network based language model

• Generating Sentences from a Continuous Space

• Variational neural machine translation

• Videotopic

• Topic models

• Neural Variational Inference for Text Processing

• Discovering Discrete Latent Topics with Neural Variational Inference

• Autoencoding Variational Inference For Topic Models

• Videonmt

• Neural Machine Translation by Jointly Learning to Align and Translate

• Neural Machine Translation Slides

• Videonli

• A large annotated corpus for learning natural language inference

• Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

• Enhanced LSTM for Natural Language Inference

This lecture will present models of whose latent variables are discrete and cannot be exactly marginalised.

• Videodiscrete

• VI without reparameterised gradients: Neural Variational Inference and Learning in Belief Networks

• Control variates: MuProp: Unbiased Backpropagation for Stochastic Neural Networks

• Investigation of induced latent variables: Do latent tree learning models identify meaningful structure in sentences?

• No latent variables, but generative training: Recurrent Neural Network Grammars

• More on gradient estimation: Simple Statistical Gradient-Flowing Algorithms for Connectionist Reinforcement Learning

• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs

• Learning to Compose Words into Sentences with Reinforcement Learning

• Videogcn

• Graph Convolutional Encoders for Syntax-aware Neural Machine Translation

• Modeling Latent Sentence Structure in Neural Machine Translation