iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page
  • TL;DR
  • Key Points
  • Notes/Questions

Was this helpful?

  1. 2018
  2. 5

Distance-based Self-Attention Network for Natural Language Inference

PreviousEffective Approaches to Attention-based Neural Machine TranslationNextDeep Residual Learning for Image Recognition

Last updated 5 years ago

Was this helpful?

论文地址:

TL;DR

本文在 DiSAN 上做了微创新, 使用 distance mask, 对相对位置进行了建模, 相距越远的单词, mask matrix 中对应位置将是一个越大的负值, 从而一定程度上抑制了远距离单词间的依赖, 换言之, 强调了单词对邻近单词的依赖, 从而更好地分配 attention.

Key Points

  • 文章将提出的模型应用于自然语言推理 NLI, 沿用了传统框架 (如下). 创新点体现在 sentence encoder 上

  • sentence encoder 基于 self-attention 对句子进行编码 (如下), 可以看到中间那一部分像极了 Transformer 的 encoder. 不同点在于, mutl-head attention 带上了 mask, 后一层的 add 变成了 gate.

  • 可以看到, 模型从 forward 和 backward 两个方向分别进行了学习, 因此, 即使使用了 distance mask, 也没有抛弃 DiSAN 中提出的 directional mask. Masked Attention 的计算如下: $Masked(Q, K, V)=softmax(\frac{QK^T}{\sqrt{dk}} + M{dir} + \alpha M_{dis})V$. 式中 $\alpha$ 起到调控作用.

  • Distance mask 中每个元素代表句中两个单词间绝对距离的负. 由于 $exp(-\inf)=0$, 因此距离越远, 负值越大, 单词间的依赖程度越低.

  • Distance mask 强化单词对邻近单词的依赖, 作用类似于 CNN 的 filter提取局部特征. 不同点在于, 前者是对整个句子的 mask, 而后者仅仅局部像素的 mask.

  • Masked multi-head attention 之后是一个 Fusion gate, 控制 attention 输出和 word embedding 的比例: $Gate(S, H)=F\bigodot S^F+(1-F)\bigodot H^F, where F=sigmoid(S^F+H^F+b^F)$ (S 是 word embedding 的矩阵, H 是 attention 的矩阵).

  • 最后使用 MaxPooling 或 Multi-dimensional attention 或两者一起, 将拼接结果矩阵压缩为向量.

  • 实验证明:

    • 相对位置很重要, 使用 distance mask 的实验组比不使用 distance mask 的实验组对句子长度具有更强的鲁棒性;

    • distance mask 强化了单词对邻近单词的依赖, 但真正具有强依赖关系的单词, 在远距离的情况下也能保持依赖. 换言之, 在保证局部依赖的同时, 又不是全局依赖.

    • Fusion gate 具有调节输出的作用, 关键词将更多地从 attention 输出, 非关键词更多地走 shortcut connection, 保持 word embedding.

    • Multi-dimensional attention 与 max pooling 的行为很相似, 都更关注关键词.

Notes/Questions

  • 本文对 DiSAN 做了一个很小的改动, 站在巨人的肩膀上, 在 SNLI 上取得了 SOTA 的测试结果

https://arxiv.org/abs/1712.02047
overall_architecture_distance_self_attention_network.png
sentence_encoder_in_distance_self_attention_network.png
distance_mask_distance_self_attention_network.png