iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page
  • Key Points
  • Notes/Questions

Was this helpful?

  1. 2018
  2. 5

Hierarchical Attention Networks for Document Classification

PreviousA Structured Self-attentive Sentence EmbeddingNextGrammar as a Foreign Language

Last updated 5 years ago

Was this helpful?

TL;DR 本文将 attention mechanism 引入 document classification 任务, 根据文档的分层特点 (word, sentence, document), 提出了 hierarchical attention network, HAN. 文中使用了 word attention 和 sentence attention, 用对 word vectors 加权求和的方式来表示 sentence vector, 并用 sentence vectors 加权求和的方式表示 document vector, 最后执行分类.

Key Points

  • HAN 提出的依据是: 文档是具有层次结构的, 单词组成句子, 句子构成文档; 文档各部分的重要性是不相同的, 不同单词和句子能表达不同的信息; 单词之间具有相关性, 不能完全独立地处理单词; 单词和句子的重要性与高度依赖于 context.

  • 不同于 NMT, document classification 不存在 target, HAN 也没用 encoder-decoder 结构, 而是采用了 word encoder-> word attention-> sentence encoder-> sentence attention 的结构. 两个 encoder-> attention 的结构相同, 输入不同.

  • HAN 使用一个可学习的 context vector, 类似于 attention-based NMT 中 decoder 的状态. 以 sentence attention 为例进行说明:

    1. 将 RNN 的状态送入 MLP 得到用于后续计算的 vector: $u_i=tanh(W_s h_i+b_s)$;

    2. 使用 $ai=\frac{exp(u_i^T u_s)}{\Sigma_i exp(u_i^T us)}$, 计算各 sentence vectors 的 attention weights ($u_s$ 即 sentence context vector);

    3. 通过加权求和 $v=\Sigma_i a_i h_i$ 得到 document vector.

Notes/Questions

  • 本文的方法很直观, 就是利用文档的分层结构特点, 但效果也很显著. 实验证明了利用分层结构带来的裨益.

  • 为什么要在 RNN 之后加一层 MLP, 让人有点匪夷所思 (有时间的话, 我会对比一下加不加 MLP 的效果)

  • 文章提到 length-adjustment 能带来 3 倍速的训练速度提升. 所谓 length-adjustment 就是将长度相近的文档放进同一个 batch 中, 训练时维持每个 batch 保持差不多等量的 token.

HAN for document classification