iamkissg
  • PaperHighlights
  • 2019
    • 03
      • Not All Contexts Are Created Equal Better Word Representations with Variable Attention
      • Learning Context-Sensitive Word Embeddings with Neural Tensor Skip-Gram Model
      • Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
      • pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
      • Contextual Word Representations: A Contextual Introduction
      • Not All Neural Embeddings are Born Equal
      • High-risk learning: acquiring new word vectors from tiny data
      • Learning word embeddings from dictionary definitions only
      • Dependency-Based Word Embeddings
    • 02
      • Improving Word Embedding Compositionality using Lexicographic Definitions
      • From Word Embeddings To Document Distances
      • Progressive Growing of GANs for Improved Quality, Stability, and Variation
      • Retrofitting Word Vectors to Semantic Lexicons
      • Bag of Tricks for Image Classification with Convolutional Neural Networks
      • Multi-Task Deep Neural Networks for Natural Language Understanding
      • Snapshot Ensembles: Train 1, get M for free
      • EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
      • Counter-fitting Word Vectors to Linguistic Constraints
      • AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
      • Learning semantic similarity in a continuous space
      • Progressive Neural Networks
      • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      • Language Models are Unsupervised Multitask Learners
    • 01
      • Querying Word Embeddings for Similarity and Relatedness
      • Data Distillation: Towards Omni-Supervised Learning
      • A Rank-Based Similarity Metric for Word Embeddings
      • Dict2vec: Learning Word Embeddings using Lexical Dictionaries
      • Graph Convolutional Networks for Text Classification
      • Improving Distributional Similarity with Lessons Learned from Word Embeddings
      • Real-time Personalization using Embeddings for Search Ranking at Airbnb
      • Glyce: Glyph-vectors for Chinese Character Representations
      • Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
      • Distilling the Knowledge in a Neural Network
      • Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrin
      • The (Too Many) Problems of Analogical Reasoning with Word Vectors
      • Linear Ensembles of Word Embedding Models
      • Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
      • Dynamic Meta-Embeddings for Improved Sentence Representations
  • 2018
    • 11
      • Think Globally, Embed Locally — Locally Linear Meta-embedding of Words
      • Learning linear transformations between counting-based and prediction-based word embeddings
      • Learning Word Meta-Embeddings by Autoencoding
      • Learning Word Meta-Embeddings
      • Frustratingly Easy Meta-Embedding – Computing Meta-Embeddings by Averaging Source Word Embeddings
    • 6
      • Universal Language Model Fine-tuning for Text Classification
      • Semi-supervised sequence tagging with bidirectional language models
      • Consensus Attention-based Neural Networks for Chinese Reading Comprehension
      • Attention-over-Attention Neural Networks for Reading Comprehension
      • Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
      • Convolutional Neural Networks for Sentence Classification
      • Deep contextualized word representations
      • Neural Architectures for Named Entity Recognition
      • Improving Language Understanding by Generative Pre-Training
      • A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence C
      • Teaching Machines to Read and Comprehend
    • 5
      • Text Understanding with the Attention Sum Reader Network
      • Effective Approaches to Attention-based Neural Machine Translation
      • Distance-based Self-Attention Network for Natural Language Inference
      • Deep Residual Learning for Image Recognition
      • U-Net: Convolutional Networks for Biomedical Image Segmentation
      • Memory Networks
      • Neural Machine Translation by Jointly Learning to Align and Translate
      • Convolutional Sequence to Sequence Learning
      • An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
      • Graph Attention Networks
      • Attention is All You Need
      • DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
      • A Structured Self-attentive Sentence Embedding
      • Hierarchical Attention Networks for Document Classification
      • Grammar as a Foreign Language
      • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
      • Transforming Auto-encoders
      • Self-Attention with Relative Position Representations
    • 1
      • 20180108-20180114
  • 2017
    • 12
      • 20171218-2017124 论文笔记
    • 11
      • 20171127-20171203 论文笔记 1
      • 20171106-20171126 论文笔记
      • 20171030-20171105 论文笔记 1
  • Paper Title as Note Title
Powered by GitBook
On this page

Was this helpful?

  1. 2019
  2. 02

Learning semantic similarity in a continuous space

PreviousAdaScale: Towards Real-time Video Object Detection Using Adaptive ScalingNextProgressive Neural Networks

Last updated 5 years ago

Was this helpful?

论文地址:

要点

本文的思想清奇: 将文档表示成一个正态分布! 于是, 文档间的不相似度就变成了从一个分布迁移到另一个分布的量. 该(不)相似度的指标类似于 WMD, 只是 WMD 考虑的是从一个文档的单词迁移到另一个文档的单词的迁移量的累计和. 如果将 WMD 的迁移看作是离散的, 那么本文提出的方法从分布迁移到另一个分布的方法/指标就是连续的, 文题由此而来.

文章提出了两步走的学习方法:

  1. 学习一个深度生成模型(VAE), 用于将句子表示成正态分布;

  2. (在第一步的学习完成之后, )用 Siamese Framework, 比较两个句子的正态分布表示, 而非向量表示.

下图是将句子表示成分布表示的示意图.

通过 VAE(变分自编码器) 来得到句子的正态分布表示, 此时句子不再被表示成向量空间中的一个点, 而是由均值$\mu$和方差$\sigma$构成的后验分布. 这样做的好处是, 分布的不确定性意味着它表征了句子所有可能性句意/意图的可能性. 模型示意图如下所示. 用了一个单层的 BiLSTM 网络作为 encoder layer, 用 BiLSTM 的输出(以什么为输出比较自由, 可简单地实验得到, 最大值/均值/最后一个时间步的hidden state)构造正态分布的均值与方差: 均值由简单的线性变换得到, 同时线性变换得到对角协方差矩阵的对数表示. Decoding 阶段, 采用 teacher-forcing, decoder 以正确的句子作为标准输出.

图中的 N(0,1) 表示使用了reparameterization 的技巧: $z~\mu(s)+\sigma(s)N(0,1)$. Repeat 和 Reformulate 则分别意味着, 图中的 s' 可以是输入(repeat), 也可以是另一个相同意思的句子(reformulate).

至于用 Siamese network 来比较两个句子得到的 distribution represetation, 和以向量为输入的模型无太大差别, 只是必须要考虑到分布表示由两个变量刻画得到. 模型示意图如下. 图中已经很明确地标出了如何计算 wasserstein2 tensor, 并以此作为 MLP 的输入, 最后输出句子对的相似度.

https://papers.nips.cc/paper/7377-learning-semantic-similarity-in-a-continuous-space
illustration_sentence_distribution_representation.png
vae4sentence_representation.png
variational_siamese_network.png