In this introductory lecture we will discuss the nature of translation data (parallel corpora) and the opportunities around this particular source of annotation.

In this session we will discuss the IBM word alignment models 1 and 2. We will view them as constrained mixture models. After motivating the models intuitively, we will develop them formally and devise an EM parameter estimation algorithm for them. We will also try to understand why the resulting optimisation problem contains multiple optima.

• Illustration

• Philip Schulz has made available a nice tutorial which is the basis for his lecture.

• Wilker Aziz has made available some notes on the derivation of the M-step for IBM1-2.

• In order to get a more concrete view of implementation, including pseudo-codes for the EM procedure, read M. Collins’s lecture notes.

• The original IBM paper is very dense, but if you want to go around teaching IBM models you should probably read it a few times.

• R. Moore (2004) discusses a number of problems with IBM model 1 involving 1) null words, 2) rare words, and 3) initialisation for EM. If you can understand the problems he raises and the solutions he proposes, you are really getting it about IBM1. Also, implementing some of his improvements will be worth some extra points in project 1.

• Toutanova et al. (2011) discuss the non-strict convexity of IBM1. This is related to the non-identifiability problem discussed in class.

• Vogel et al. 1996 revisit the assumption that alignments links are independent.

• Schulz et al, 2016 revisit NULL alignments in IBM1-2.

In this session we will continue the exposition of IBM 1 and 2 models for word alignment. We will view them as constrained mixture models. After motivating the models intuitively, we will develop them formally and devise an EM parameter estimation algorithm for them. We will also try to understand why the resulting optimisation problem contains multiple optima.

• Illustration

• Philip Schulz has made available a nice tutorial which is the basis for his lecture.

• Wilker Aziz has made available some notes on the derivation of the M-step for IBM1-2.

• In order to get a more concrete view of implementation, including pseudo-codes for the EM procedure, read M. Collins’s lecture notes.

• The original IBM paper is very dense, but if you want to go around teaching IBM models you should probably read it a few times.

• R. Moore (2004) discusses a number of problems with IBM model 1 involving 1) null words, 2) rare words, and 3) initialisation for EM. If you can understand the problems he raises and the solutions he proposes, you are really getting it about IBM1. Also, implementing some of his improvements will be worth some extra points in project 1.

• Toutanova et al. (2011) discuss the non-strict convexity of IBM1. This is related to the non-identifiability problem discussed in class.

• Vogel et al. 1996 revisit the assumption that alignments links are independent.

• Schulz et al, 2016 revisit NULL alignments in IBM1-2.

We present a Bayesian version of IBM1 which has a conjugate Dirichlet prior on the translation parameters. We exploit the conjugacy of the resulting model to develop an efficient variational inference algorithm. During the lecture we will explicitly introduce the logic of variational inference. Students should acquaint themselves with basic properties of exponential families beforehand.

• IBM1-2

• VB for IBM1

• The lecture will very closely follow this script. Students should read it before the lecture. We may decide to skip some details from the script in the lecture.

• The same model but with a Gibbs sampler for inference is presented in Mermer and Saraclar (2011).

• An outstanding account of variational methods with lots of references to recent developments is given in David Blei’s tutorial.

• Chapter 2 of Matthew Beal’s PhD thesis gives a very clear development of variational inference for Bayesian models.

• Bayesian estimation for NLP

• For Bayesian inference without VI, check this MCMC tutorial

• Dyer et al. 2013 revisit the parameterisation of IBM model 2. Their model, called ‘fast_align’ achieves state-of-the-art performance and a robust and efficient implementation is available.

• Bayesian Word Alignment for Statistical Machine Translation

This lecture covers the necessary background for Inversion Transduction Grammars and possibly also tree-structured CRFs (parameter estimation and prediction).

• Bitext parsing

• ITGs

• David Chiang has a nice tutorial on synchronous grammar. I will not stick too much to it though.

• Dyer (2010) shows that a cascade of monolingual parsers is a very neat and general way to perform bitext parsing.

• Laura Kallmeyer’s book (i.e. Parsing Beyond Context-Free Grammars) is a really good reference. Particularly, Chapter 3 on parsing as deduction.

In this lecture we will talk about word order difference in terms of a permutation of the integers. We will learn factorization of permutations into prime permutations and permutation trees (PETs). A word-aligned parallel corpus can be factorized into (almost) a parallel treebank of synchronous trees called PETs. Given a parallel treebank we will present work on inducing reordering grammar for preordering and translation.

• Stanojevic ́ and Sima’an. 2015. Reordering Grammar Induction

• Part 1: PBSMT

• Part 2: HPBSMT

• Adam Lopez’s SMT survey, in particular, sections 1 to 3.

• Part 1: PBSMT

• Part 2: HPBSMT

• Adam Lopez’s SMT survey, in particular, sections 1 to 3.

Evaluation in Machine Translation is dealing with automatic assesment of the translation quality. Unlike many machine learning tasks, here there is no unique gold truth to which we can compare the translation – there are many equally good translations. Also translations can be good or bad in many dimensions: morphology, syntax or semantics. Evaluation is important not only for assesing the quality of the final system but also for automatic tuning of the systems.

• A good survey of evaluation methods can be found in Chapter 8 of Koehn’s book. You can also refer to statmt’s survey page

• BLEU is the most widely used automatic evaluation metric, it is mostly a precision metric.

• BEER is a trainable metric meant for approximating human ranking of translations.

• METEOR tries to balance precision and recall.

• Read (Lopez, 2012) for a formal view of evaluation of automatic evaluation metrics (you read it right!).

• Hypothesis testing for SMT or whether two systems differ.

• Bengio, Yoshua, et al. A neural probabilistic language model. Journal of machine learning research 3.Feb (2003): 1137-1155.

• Mikolov, Tomas, et al. Recurrent neural network based language model. Interspeech. Vol. 2. 2010.

• Goldberg NN for NLP primer

Starting from Neural language models, we will learn how to train a model end-to-end to perform translation.

• Sequence to sequence learning

• Neural Machine Translation by Jointly Learning to Align and Translate

• Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. “On the difficulty of training recurrent neural networks.” ICML (3) 28 (2013): 1310-1318.