iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page

Was this helpful?

  1. 2019
  2. 01

Dynamic Meta-Embeddings for Improved Sentence Representations

PreviousIntrinsic Evaluation of Word Vectors Fails to Predict Extrinsic PerformanceNext2018

Last updated 5 years ago

Was this helpful?

论文地址:

要点

本文和之前的 word meta-embedding, 想法一致, 利用多个预训练的 word embeddings 来提高性能, 不同的是, 本文并没有导出一个 word embedding, 而是通过 self-attention 来调整不同 word embedding 的比例, 提高 sentence representation. 就像冷水和热水混合, 调出一个合适的水温, 而控制器把握在洗澡的人手中, 而根据他的感受(performance)来调整.

本文的做法有几点好处:

  1. task-specific, 直接在具体任务上调整不同 word-embedding 的权值, end-to-end;

  2. 直接通过具体任务的性能来反映方法的优劣, 可以说更直观;

  3. 一般使用 attention 都逃不脱的权值分布图, 可以提供哪个 word-embedding 在哪个任务上更被需要等视角;

根据是否利用 context, 文章提供了两种思路: DME(Dynamic Meta-Embedding) 和 Contextualized DME.

前者首先将所有使用到的 word embeddings 映射到同一个向量空间, 然后对每一维特征都计算加权平均, 就这样.

w' 是映射后(同一个单词的)不同的 word vector 组成的矩阵, $alpha$则是对应每个位置的权值, 由 self-attention 计算得, 公式如下, 其中 a 是可学习的参数. 该方法被称为 self-attention 是因为, w' 被用了两次(还有一次是加权平均时).

CDME 与 DME 的不同之处在于权值的计算上. CDME 将映射后的 word vector 输入了一个 BiLSTM, 然后用 BiLSTM 的状态来计算权值, 如下. 要注意的一点是, 这里的 BiLSTM 是专门用于 self-attention 的权值的, 并不是后面 DME/CDME 作为输入的 BiLSTM (文中没有交代清楚这一点, 我一开始对 h 出现在此处还感到奇怪来着, 看了代码才弄明白).

至于$phi$, 论文配套的代码提供了两个可选的函数, 一个是常规的 softmax, 一个是用 sigmoid 来作门控(gating).

之后将使用 DME/CDME 得到的 $w_j$ 当作普通的 word vector, 后面按正常的操作来就行了.

https://aclweb.org/anthology/D18-1176
dme.png
self_attention_dme.png
self_attention_cdme.png