iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page
  • TL;DR
  • Key Points
  • Notes/Questions

Was this helpful?

  1. 2018
  2. 6

Semi-supervised sequence tagging with bidirectional language models

PreviousUniversal Language Model Fine-tuning for Text ClassificationNextConsensus Attention-based Neural Networks for Chinese Reading Comprehension

Last updated 5 years ago

Was this helpful?

论文地址:

TL;DR

这篇算是 NAACL2018 最佳论文 (ELMo) 的前传, 那篇文章的主要思想在本文都有所提及. 只是论文的目标不一样罢了. 文中引入了 LM embedding, 在 word embedding 之外迁移了从大型语料中习得的上下文特征, 提升了其他任务的性能, 在小型训练集上效果更显著.

Key Points

  • 开篇, 本文就提出了一个尖锐的问题: 许多任务, 单单使用单词本身的语义还不够, 还需要单词的上下文中的语义. 这是本文要解决的问题.

  • 对以上问题最简单的解决办法就是用 CNN/RNN 来处理序列, 从而对单词的上下文进行表示. 本文的解决之道就是在大型语料上训练 language model, 学习 LM embedding, 再迁移到其他任务上.

  • 以序列标注为例, 本文提出的方法如下:

    • 在大型语料上预训练 word embedding 和 LM embeddidng;

    • 真正进行标注学习时, 计算每个 token 的 word embedding 和 LM embedding, 拼接 (回忆 ELMo, 前者捕捉上下文无关语义, 后者捕捉上下文依赖语义);

    • 序列标注的任意模型 (同昨天的模型类似, 本文采用的是 RNN+CRF).

  • 实验中, 本文使用的已经是 biLM 了, 再具体一点, 弥补下 ELMo 一张图都没有的缺陷:

  • 本文对学习率的调整, 关注一下: 先固定学习率学习, 直到在验证集上取得最好的性能, 之后每 5 个 epoch 将学习率缩小 10 倍, 然后重复这个缩小的过程一遍, 结束.

  • 当数据量较小时, 计算模型性能的方差很重要.

  • 实验表明, 不必在任务相关的语料上预训练 LM (语料可能比较小), 在大型任务不相关的语料上训练的 LM 足够捕捉上下文信息了 (这和 word embedding 多像呀).

  • 实验分析:

    • 本文使用的标注模型使用两层 bi-RNN (见上图左侧), 将 LM embedding 注入到第一层 RNN 之后的性能最好, 这可能是因为第二层 RNN 能捕捉任务相关 (第一层 RNN) 的上下文与任务无关的上下文信息之间的交互;

    • LM 的尺寸对性能有影响, 甚至可能会减弱性能, 文章对此的分析是, LM 的帮助来自它从大型语料习得有效的组合函数;

    • 对于小型训练集, 从其他大型语料中习得的 LM 对其性能提升最大.

Notes/Questions

  • 题目中所说的 semi-supervised, 含义在于, 在大型语料上学习的 LM embedding 能迁移到其他任务上. 和 ImageNet 上预训练的模型用于其他任务, 以及 word embedding 类似.

https://arxiv.org/abs/1705.00108