iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page
  • Key Points
  • Notes/Questions

Was this helpful?

  1. 2018
  2. 5

Neural Machine Translation by Jointly Learning to Align and Translate

PreviousMemory NetworksNextConvolutional Sequence to Sequence Learning

Last updated 5 years ago

Was this helpful?

TL;DR 本文是将注意力机制 Attention Mechanism 引入机器翻译的第一文, 文中称为对齐 Alignment. 在编码器 encoder 与解码器 decoder 之间增加了一个对齐模型 alignment model, 使得翻译不再是将 source 编码成固定长度的上下文向量 context vector 再解码成 target, 而是维护一个可变的 context, 每翻译一个词, 都从 source 中搜索最相关的部分与之对齐. 由于使用的 soft-alignment, 梯度可通过 alignment model 反向传播, 因此同时学习翻译与对齐.

Key Points

  • 指出传统 encoder-decoder 框架的不足之处. 随着序列的增长, 将输入的所有信息压缩进一个固定长度的向量越来越困难, 模型性能将急剧下降.

  • 为解决上述问题, 引入了 aligment model, decoder 每生成一个词的翻译, 都会对输入序列进行搜索, 找出与当前要翻译的内容最相关的部分; 结合之前的输出, 当前 decoder 的状态和查询结果, 进行更好的翻译.

  • 给出了 alignment model 的一个范式:

    • 综合各要素计算下一个翻译的概率: $p(yi|y_1, \dots, y{i-1}, x)=g(y_{i-1}, s_i, c_i)$; $s_i$, $c_i$ 分别是 decoder 第 i 时刻的 state 与查询得的 context.

    • 根据之前的输出, 状态和当前 context 计算当前状态: $si=f(s{i-1}, y_{i-1}, c_i)$;

    • context vector 是加权求和的结果: $ci=\Sigma{j=1}^{Tx} \alpha{ij}hj$; $h_j$ 是 endoder 在第 i 时刻的状态, $\alpha{ij}$ 是对应的权值.

    • 使用 decoder 的上一个状态计算和 encoder 某时刻状态的相似度, 即对齐程度: $e{ij}=a(s{i-1}, h_j)$; (按照原文的表示, 此处为小写的 A)

    • 概率和为 1, 因此使用 softmax, soft-alignment 由此得名: $\alpha{ij}=\frac{exp(e{ij})}{\Sigma{k=1}^{T_x} exp(e{ik})}$. (此处为希腊字母 alpha, 表征概率)

  • 对齐时, 在整个输入序列上搜索相关部分, 每个词都以一定概率和翻译结果对齐, 好处是可以考虑更多的上下文信息, 并很自然地解决了词组的翻译问题, 带来的问题是, 对于 $T_x$ 长度的 source 和 $T_y$ 长度的 target, 要进行 $T_x \times T_y$ 对齐运算.

  • 具体地, 文中使用感知机来计算相似度, 即 $a(s{i-1}, h_j)=v_a^T tanh(W_a s{i-1}+U_a h_j)$

Notes/Questions

  • 从概率的角度看, translation 可以理解为 conditional language modeling. 即找出最优的翻译, 最大化 $p(target|source)$ 的概率.

  • 用 $si=f(s{i-1}, y_{i-1}, c_i)$ 来计算状态, 抛弃了 decoder RNN 的状态, 总觉的挺别扭的.

  • 给出了 Attention 的一个范式, 但计算方法上还比较粗糙, 这也为后面的各种论文提供了发挥空间.

Alignment Illustration