iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page
  • TL;DR
  • Key Points
  • Notes/Questions

Was this helpful?

  1. 2018
  2. 5

Memory Networks

TL;DR

本文提出了记忆网络 Memory Network, 其核心思想是利用一段可读写的记忆组件来辅助推理, 其他组件包括: 1) 对输入的处理; 2) 对记忆的更新; 3) 对以记忆搜索; 4) 对外响应. 方法很直白, 就是一个袖珍计算机模型.

Key Points

  • 记忆网络由用数组表示的记忆和以下 4 组件组成:

    1. I - 输入特征映射, 将输入转换为内部特征表示;

    2. G - 泛化组件, 在给定输入的情况下, 更新旧的记忆; (这个命名居然是为了未来考虑的)

    3. O - 输出特征映射, 给定新的输入与当前记忆状态, 产生输出;

    4. R - 响应, 将输出转换成期望的响应格式.

  • 记忆网络的工作流程按 I->G->O->R 的依次进行, 按照文中描述, 记忆只会在有新的输入的情况下被更新, 即 O, R 都不会对 G 产生影响. 测试时, 记忆依旧可以使用和被更新, 但 IGOR 的参数不再更新.

  • 记忆网络的实现可以很灵活, 应用场景也很广泛, 只要满足上述范式即可, 文中只是给出了一种实现方案, 并证明了在机器阅读上的有效性.

  • 最简单的 G 就是, 只保存不修改. 但记忆过大时, "回忆"是一个问题; 而当有限的记忆被塞满时, 必须实现"遗忘", 选择哪一个记忆单元重写记忆是一个问题. 这两个问题都可以归结为寻址问题, 回忆下内存的工作机制, 对理解应有帮助.

  • 根据文章的描述, O 与 R 的命名也不太准确. O 的输出只是输出到 R, 而 R 产生真实的对外的响应, 此二者承担起推理的主要职责.

  • 文章循序渐进地介绍了将记忆网络应用于机器阅读的过程, 记录要点:

    • O 产生输出特征时, 根据设定的超参数寻觅出 k 个记忆, 后一个记忆的寻址取决于输入与之前的记忆, 这个过程类似于 Language Modeling, 这里将 vocabularity 替换成了训练过程中填充好的记忆 (数组);

    • R 的过程与 O 类似, 这次是真的在 vocabularity 中寻找与输入+记忆最相关的词了;

    • 为解决记忆过大, 寻址开销大的问题, 文章使用了哈希的技巧: 将输入哈希进一个或多个存储单元. 记忆寻址时, 只对同一存储单元中的记忆计算依赖程度;

    • 文章提供了一种对写入时间建模的方法: 假设记忆单元的索引与写入时间相关, 则将对索引进行编码, 作为额外的特征;

    • 机器阅读中遇到的生词, 使用上下文词袋来学习. 训练时, 模仿 dropout, 以 d 的概率使用已知词的上下文来表示它;

Notes/Questions

  • 可能是我看过 FAIR 最差的论文了, 原来我以为 FAIR 出品, 必是精品. 很多表达都不清不楚的, 将记忆网络应用于机器阅读的过程, 也过于简化了, 或者说, 铺垫太多, 重点没有突出, 实验也很潦草的样子.

  • 因为之前看了 TCN 那篇, 本文还是有所启发的:

    1. 记忆不是被更新的，而是被组合的: 只要存储空间足够大, 每个时刻的记忆都应记录的，都是唯一的，我们的记忆可能是从不同时刻选择了记忆，对它们做了组合。Soft-Attention 做了类似的工作, 只是每一次, 它的记忆是序列各时刻的状态, 没有一个长期保存的记忆。

    2. 对于记忆的修改会造成信息丢失. 但是大记忆的寻址是一个很严重的问题. 文中的哈希是一种方法, 我觉得分层也是一种方法. 既然在 word embeddig 中, 科学家们使用了各种办法来近似 softmax 的分母, 那些方法估计也值得考虑.

PreviousU-Net: Convolutional Networks for Biomedical Image SegmentationNextNeural Machine Translation by Jointly Learning to Align and Translate

Last updated 5 years ago

Was this helpful?