iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page
  • TL;DR
  • Key Points
  • Notes/Questions

Was this helpful?

  1. 2018
  2. 6

Consensus Attention-based Neural Networks for Chinese Reading Comprehension

PreviousSemi-supervised sequence tagging with bidirectional language modelsNextAttention-over-Attention Neural Networks for Reading Comprehension

Last updated 5 years ago

Was this helpful?

论文地址:

TL;DR

本文在 ASReader 上做了微创新, 对 query 中每个单词对应的 attention 做一个 merging, 得到最终的 document 中各单词作为 answer 的概率.

Key Points

  • 文章公开了一份中文阅读理解数据集. 数据集构造方法如下 (欲自建数据集的同学可了解下):

    • 以人民日报的文章为语料, 分词+词性标注;

    • 给定一篇文章, 随机选择一个名词作为 answer (要求至少出现 2 次以上);

    • 以包含 answer 的句子作为 query, 并将 answer 挖掉;

  • 文章还构造了额外两个测试集, 以童话为语料, 而不是新闻, 这就更考验模型的性能了. 其中一个仍按上述方法构造, 另一个是手工构造的, 后者使得机器阅读的难度更大了.

  • 文章提出了 Consensus Attention Sum Reader, CASReader. 可以看出是基于 ASReader 的.

  • 还是简述下模型结构:

    • Document 与 query 共享 embedding, 好处是相比分离地学习两套 embedding, query 能获得更好的表示 (没见过两套 embedding 的);

    • 两套 Bi-GRU, 分别计算 document 和 query 的表示, 前向和反向 hidden state 做拼接处理;

    • 对 query 中的每个单词, 计算它与 document 中所有单词的匹配程度: $\alpha(t)=softmax(h{doc}\bigodot h{query}(t))$ (t 表示时序, 即 query 的第 t 个位置).

  • (敲黑板). 所谓 consensus, 是要在 query 的每个单词上达到某种共识; 所谓 consensus attention, 就是在单词对应的 attention 之间取得共识: $s=f(\alpha(1), \dots, \alpha(m))$ (m 表示 query 的长度).

  • 文中提出了三种 merging function f:

    • max (此时的共识就是推举出最大的 attention, 下面的类似): $s\propto softmax(max_{t=1\dots m}\alpha(t))$;

    • sum: $s\propto softmax(\Sigma_{t=1}^m \alpha(t))$;

    • avg: $s\propto softmax(\frac{1}{m}\Sigma_{t=1}^m \alpha(t))$;

  • 最后, 和 ASReader 一样, 对 document 中 unique word, 计算所有位置上的概率和: $P(w|D, Q)=\Sigma_{i\in I(w, D)s_i,w \in V}$ ($I(w, D)$ 是指示函数, 表明单词 w 出现在 D 的位置).

  • 文中对于 unknown words 的处理 mark 一下. 将 200K 的 vocab 砍到 100K, 所有的 unknown words 映射到不同的 10 个符号, 而不是 就完了. 该方法出自另一篇论文, 余以为 10 个符号的含义可以自由发挥, 不必拘泥.

Notes/Questions

  • 文中说, 以他们的方法构造数据集, document 可复用, 即 document: (query, answer) 是 1:N 的关系, 从而能构造大型数据集 (之前的方法, 一种是使用自动生成的摘要作为 query, 一种是以 document 的后一句作为 query). 不过, 我感觉这样得到的数据集质量会很差, query+answer 都不能保证能推理吧.

https://arxiv.org/abs/1607.02250
CASReader.png