iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page
  • TL;DR
  • Key Points
  • Notes/Questions

Was this helpful?

  1. 2018
  2. 6

Attention-over-Attention Neural Networks for Reading Comprehension

PreviousConsensus Attention-based Neural Networks for Chinese Reading ComprehensionNextBaseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

Last updated 5 years ago

Was this helpful?

论文地址:

TL;DR

本文算是 CASReader 的姊妹篇吧. 在机器阅读的常用的 document-level attention 之外, 又引入了一层 attention, 考量 attention 的重要性. 在候选答案时, 提出了 N-best Re-Ranking 策略, 对 N 个最优的候选项进行 double-check, 从中选择最后答案.

Key Points

  • 机器阅读的模型关注了 document 的表示 (对 document 中的词进行 attention learning), 但很少关注到 query 的表示. 本文在 document-level attention 之上引入了另一个 attention, 来表示每个 attention 的重要性, 即题目所谓的 attention-ovoer-attention.

  • 同 CASReader, 文中保留了 document 和 query 中每个单词的表示.

  • 计算 query 中单词对 document 中单词的依赖程度: $M(i, j)=h{doc}(i)^T\cdot h{query}(j)$, 得到的 $M\in \mathbb{R}^{|D|*|Q|}$ (|D| 表示 document 的长度, 即单词数, |Q| 表示 query 的长度). 然后对矩阵按列进行 softmax, 这样每一列表示 query 中一个单词与整个 document 中所有单词的匹配程度, 称为 query-to-document attention $\alpha$.

  • 再计算 document-to-query attention. 由于 cosine similariy 是对称的, 之前得到的 M 可复用, 只是这次按行进行 softmax, 就得到了 document 中单词对 query 单词的 attention, 称为 document-to-query attention $beta$.

  • 为计算 attention-over-attention, 文中先对 $\beta$ 按列求均值: $\beta=\frac{1}{n}\Sigma_{t=1}^{|D|}beta(t)$, 然后计算 $\alpha$ 和 $\beta$ 的点积: $s=\alpha^T \beta$. 这样, query 中每个单词的贡献/重要性都得到了考虑.

  • 预测时, 文章依旧采用 ASReader 的方法, 将 unique word 的所有作为 answer 的概率相加, 最高者为 answer.

  • 文中提出了 N-best Re-ranking 策略, 即生成 N 个最可能的候选答案, 代回 query 中, double-check 答案的正确性.

  • 此处文章使用了 3 种 Language Modeling 模型来评估答案, 分别是 Global N-gram LM, Word-class LM, Local N-gram LM. 前两者能捕获全局信息, 在训练集上训练得到, 后者则用于捕获局部信息, 在对应的 document 上训练.

  • 使用 K-best MIRA 算法学习三个 LM 的权值, 然后加权求和从 N-best 答案中选出最终答案.

  • 实验发现, Local N-gram LM 和另两个 LM 更有适合的场合. 对于 common noun 类型的答案, global features 帮助更大; 对于实体名作为答案的情况, local LM 可以提供更多有帮助的信息.

Notes/Questions

  • (回忆一下, 在 CASReader 中, 使用了 merging function, 函数一经定义就不可修改, 属于静态方法. 相比而言, AoAReader 的处理更灵活)

  • 文章通过利用 document 与 query 的交互信息来提升模型性能, 关于为什么要先求均值, 然后求点积, 一笔带过. 虽然结果很好, 个人感觉没有解释清楚.

  • N-best Re-Ranking 有点类似 LM 使用的 Beam Search, 摒弃了贪婪方法, 提升结果的可靠性.

https://arxiv.org/abs/1607.04423
AoAReader.png