Lectures

“In this introductory lecture we will discuss the distributional models for learning word representations from text.”

“In this lecture we will discuss the skip-gram model and its properties, and then move on to compositional semantics, concerned with modelling phrase meaning.”

• The following paper provides a nice explanation of skip-gram with negative sampling:

• Yoav Goldberg and Omer Levy. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method

In this session we will discuss three recent papers on dependency-based word embeddings, specialisation and retro-fitting.

• In this session we will discuss the following papers:

• Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In proceedings of ACL 2014.

• Manaal Faruqui, Jesse Dodge, Sujay K. Jauhar, Chris Dyer, Eduard Hovy and Noah A. Smith. 2015. Retrofitting Word Vectors to Semantic Lexicons. In Proceedings of NAACL 2015.

• Douwe Kiela, Felix Hill and Stephen Clark. 2015. Specializing Word Embeddings for Similarity or Relatedness. In Proceedings of EMNLP 2015.

“In this session we will discuss learning multilingual word representations and representations for phrases and sentences.”

• In this session we will discuss the following papers:

• Jocelyn Coulmance, Jean-Marc Marty, Guillaume Wenzek and Amine Benhalloum. 2015. Trans-gram, Fast Cross-lingual Word-embeddings. In Proceedings of EMNLP 2015.

• Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual Models for Compositional Distributed Semantics. In Proceedings of ACL 2014.

• Richard Socher, Brody Huval, Christopher D. Manning and Andrew Y. Ng. 2012. Semantic Compositionality through Recursive Matrix-Vector Spaces. In Proceedings of EMNLP 2012.

In this lecture I will present probabilistic modelling with neural networks, we will start with maximum likelihood estimation based on fully observed data, then we will discuss the challenges of learning distributions over observed and unobserved (latent) data. We will then talk about variational inference and how it may help us estimate deep generative models.

• The wake-sleep algorithm for unsupervised neural networks

• Neural Variational Inference and Learning in Belief Networks

• Variational Inference:

• Chapter 2 of Matthew Beal’s PhD thesis gives a very clear development of variational inference for Bayesian models.

• An outstanding account of variational methods with lots of references to recent developments is given in David Blei’s tutorial.

• Stochastic optimisation:

• Practical notes by Leon Bottou (in particular section 5.2 discusses learning rate schedules)

• A handbook chapter

• The original advanced!

In this lecture I will present the variational auto-encoder, an important deep generative model that makes a building block for many others. We will also learn about the reparameterised gradient, a major development for efficient estimation of DGMs.

• Gaussian distribution

• The original: Kingma and Welling (2014) and Rezend et al (2014)

• Variational auto-encoders:

• A tutorial (disclaimer: not my favourite!)

• More on reparameterisation (advanced!): Titsias and Lazaro-Gredilla (2014) and Ruiz et al (2016)

• Automatic Differentiation Variational Inference

In this lecture I will present two deep latent variable models of word representation: a discriminative and a generative one.

• CBOW and Skip-gram: Distributed Representations of Words and Phrases and their Compositionality

• Skip-gram is a binary classifier: word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method

• Generative modelling via lexical alignment: Learning Bilingual Word Representations by Marginalizing Alignments

• Embedding Words as Distributions with a Bayesian Skip-gram Model

• Deep Generative Model for Joint Alignment and Word Representation

• Auto-encoding variational inference for topic models

• Semantic Parsing with Semi-Supervised Sequential Autoencoders

In this lecture I will present generative models over natural language sentences.

• Recurrent neural network based language model

• Generating Sentences from a Continuous Space

• Variational neural machine translation

In this session we will give an introductory lecture on the relationship of representation learning and neurocognition of language, and discuss two recent papers on the topic.

• From distributional semantics to feature norms: grounding semantic models in human perceptual data

• A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain

• Natural speech reveals the semantic maps that tile human cerebral cortex

• (http://gallantlab.org/huth2016/)

• In this session we will discuss the following papers:

• Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science, 320(5880):1191–1195.

• Samira Abnar, Rasyan Ahmed, Max Mijnheer, Willem Zuidema. 2018. Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity. Proceedings Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018).

In this session I will give an introductory lecture on lexical ambiguity and metaphor, and how these phenomena can be modelled using semantic representations. We will then discuss two recent papers on the topic.

• Ekaterina Shutova. 2015. Design and Evaluation of Metaphor Processing Systems. Computational Linguistics, 41(4): 579-623.

• In this session we will discuss the following papers:

• Dario Gutierrez, Ekaterina Shutova, Tyler Marghetis and Benjamin Bergen. 2016. Literal and Metaphorical Senses in Compositional Distributional Semantic Models. In Proceedings of ACL 2016, Berlin, Germany.

• Marek Rei, Luana Bulat, Douwe Kiela and Ekaterina Shutova. 2017. Grasping the Finer Point: A Supervised Similarity Network for Metaphor Detection. In Proceedings of EMNLP 2017, Copenhagen, Denmark.

In this session we will give an introductory lecture on multimodal semantics, i.e. learning semantic representations from both linguistic and visual data. We will then discuss their application in multimodal dialogue systems.

• In this session we will discuss the following papers:

• Douwe Kiela and Leon Bottou 2014. Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics. In Proceedings of EMNLP 2014, Doha, Qatar.

• Angeliki Lazaridou, Nghia The Pham, Marco Baroni. 2015. Combining Language and Vision with a Multimodal Skip-gram Model. In Proceedings of NAACL 2015.

In this session we will continue our discussion of multimodal semantics. At the end, I will give you an overview of the third practical and what we expect to see in your final report.

• In this session we will discuss the following papers:

• A.J. Anderson, D. Kiela, S. Clark and M. Poesio. Visually Grounded and Textual Semantic Models Differentially Decode Brain Activity Associated with Concrete and Abstract Nouns. Transactions of the Association for Computational Linguistics (TACL).

• Luana Bulat, Stephen Clark and Ekaterina Shutova. 2017. Speaking, Seeing, Understanding: Correlating semantics models with conceptual representation in the brain. In Proceedings of EMNLP 2017, Copenhagen, Denmark.

• Alexis Conneau and Douwe Kiela. SentEval: An Evaluation Toolkit for Universal Sentence Representations.

In this lecture I will present models of graph induction.

• Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling

• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

• Categorical Reparameterization with Gumbel-Softmax

• Learning Structured Text Representations

• Neural Machine Translation with Source-Side Latent Graph Parsing

• Structured Attention Networks

• Learning to Compose Task-Specific Tree Structures

In this lecture I will present models of whose latent variables are discrete and cannot be exactly marginalised.

• VI without reparameterised gradients: Neural Variational Inference and Learning in Belief Networks

• Control variates: MuProp: Unbiased Backpropagation for Stochastic Neural Networks

• Investigation of induced latent variables: Do latent tree learning models identify meaningful structure in sentences?

• No latent variables, but generative training: Recurrent Neural Network Grammars

• More on gradient estimation: Simple Statistical Gradient-Flowing Algorithms for Connectionist Reinforcement Learning

• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs

• Learning to Compose Words into Sentences with Reinforcement Learning

In this guest lecture, Dr Jelle Zuidema will talk about his work on deep learning and logical inference.

Here we list topics and papers that you can use to drive your studies beyond what we presented in this course.

• Deep generative models

• Beyond Gaussians: Automatic Differentiation Variational Inference

• Beyond known densities: Variational Inference with Normalizing Flows and Improved Variational Inference with Inverse Autoregressive Flow

• Advanced control variates: Backpropagation through the Void: Optimizing control variates for black-box gradient estimation

• Non-parametric priors: Nonparametric Variational Auto-encoders for Hierarchical Representation Learning

• Implicit likelihoods: Generative Adversarial Nets and Hierarchical Implicit Models and Likelihood-Free Variational Inference