iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page
  • TL;DR
  • Key Points
  • Notes/Questions

Was this helpful?

  1. 2018
  2. 5

Self-Attention with Relative Position Representations

PreviousTransforming Auto-encodersNext1

Last updated 5 years ago

Was this helpful?

TL;DR

本文可以说是 的姊妹篇. 不同于前一篇使用 Positional Encoding 将时序编码进序列, 本文将输入序列看成 a labeled, directed, full-connected graph. 然后学习边的表示, 作为相对位置 relative position 的表示. 实验发现, 在翻译任务上, 相比绝对位置, 即使用 positional encoding, 相对位置能带来性能提升.

Key Points

  • Positional encoding 基于不同频率的正/余弦函数对位置进行编码, 对(绝对)位置进行编码的同时, 对于相对位置关系也具有一定的表示能力, 因为距离为 k 的两个位置的编码之间是线性关系

  • 本文将输入序列视为带标签的, 有向, 全连接图, 学习边的表示, 显式地对相对位置进行编码, 并将边向量加入 self-attention 的计算过程, 使得 self-attention 对相对位置是可知的.

  • 文中学习了两种不同的边向量, 分别在不同时候加入计算过程, 以下是边向量对 self-attention计算的改造:

    1. $zi=\Sigma{j=1}^n \alpha{ij}(x_jW^V)$ $\rightarrow$ $z_i=\Sigma{j=1}^n \alpha{ij}(x_jW^V+a{ij}^V)$ (按照文中的符号, 括号外的是阿尔法, 括号内的是小写的 A);

    2. $e{ij}=\frac{(x_i W^Q)(x_j W^K)^T}{\sqrt{d_z}}$ $rightarrow$ $e{ij}=\frac{(xi W^Q)(x_j W^K +a{ij}^K)^T}{\sqrt{d_z}}$ (此处是小写的 A).

  • 两种 self-attention 的拓展都只是简单地加了边向量, 目的是不影响模型的计算并行性.

  • 基于过长的相对位置并不能带来有用信息, 文中限制了两个位置的最大距离, 即边长. 如下, k 是最大边长长度:

    • $a{ij}^K=w{clip(j-i, k)^K}$;

    • $a{ij}^V=w{clip(j-i, k)^V}$;

    • $clip(x, k)=max(-k, min(k, x))$;

  • 考虑到空间复杂性 space complexity, 本文在每个头之间, 序列之间都共享了相对位置表示 (只是扩展了 self-attention, 继续沿用 Transformer 的 Multi-head Attention).

    * 对于后一种拓展, 不同位置对之间的相对位置表示不同, 无法通过单次矩阵乘法得到所有位置对的表示, 同时为避免相对位置表示的广播, 实际拆成了两部分进行计算: $e{ij}=\frac{x_i W^Q(x_j W^K)^T + x_i W^Q(a{ij}^K)^T}{\sqrt{dz}}$. 前半部份就是为扩展时的 $e{ij}$, 后半部分通过形变成能计算矩阵乘法的形状, 然后进行矩阵乘法, 最后将两部分将两部分相加. (这部分内容, 我看不太明白)

  • 文中实验发现

    • 同时使用 positional encoding 和相对位置表示并没有带来进一步的性能提升;

    • 边长大于等于 2 时, BLEU 已经趋于不变;

    • 使用第二种扩展, 即在计算对齐程度度时, 加入相对位置编码已经能足够了; (再使用第一种扩展并没有带来性能提升).

Notes/Questions

  • 暂缺

Attention Is All You Need
relative_position_as_edge.png