What does word2vec actually learn?

word2vec 实际上在学习什么？

What does word2vec actually learn?

word2vec 实际上在学习什么？

And how to train embeddings from similarity functions

以及如何从相似度函数训练嵌入

Representing discrete objects by continuous vectors, the so-called embeddings, has been at the heart of many successful machine learning solutions. The superiority comes from the fact that, unlike the original discrete objects, the embedding vectors offer a compact representation that captures the similarity between the original objects.

将离散对象表示为连续向量（即所谓的嵌入）一直是许多成功机器学习解决方案的核心。其优越性在于，与原始的离散对象不同，嵌入向量提供了一种紧凑的表示形式，能够捕捉原始对象之间的相似性。

In this article, we consider the famous word2vec algorithm. Word2vec is simple and intuitive. At a high level, it says that words that appear frequently close to each other should have a similar vector representation. In particular, the example

在本文中，我们将讨论著名的 word2vec 算法。Word2vec 简单直观。从高层次来看，它认为频繁出现在彼此附近的词语应该具有相似的向量表示。特别是，下面这个例子

embedding(man) - embedding(king) ~ embedding(woman) - embedding(queen)

embedding(man) - embedding(king) ≈ embedding(woman) - embedding(queen)

has become the poster child for the ability of word embeddings to capture word semantics.

已成为词嵌入能够捕捉词语语义能力的典型代表。

However, the optimization objective cannot be presented by a well-defined quantity. For comparison, consider learning word embeddings using matrix factorization. Let D be a text corpus consisting of m documents and a vocabulary of n unique words. We compute the n-times-m word, document matrix M, where M[u,v] records how many times word u occurs in document v, see Figure 1.

然而，其优化目标无法用一个明确定义的量来表示。作为对比，考虑使用矩阵分解来学习词嵌入。设 D 是一个包含 m 篇文档和 n 个唯一单词的文本语料库。我们计算 n×m 的词-文档矩阵 M，其中 M[u,v] 记录单词 u 在文档 v 中出现的次数，见图1。

The matrix factorization is defined as

矩阵分解定义如下：

In the following, we will slightly abuse notation and denote by u both a word u and its embedding vector.

在下文中，我们将略微滥用符号，用 u 同时表示单词 u 及其嵌入向量。

In this case, we know that for a word embedding u, and a document embedding v the inner product between u and v preserves the information how many times the word u occurs in the document v. The larger the embedding dimensionality d, the better the approximation. Unfortunately, there is no such clear formulation of the optimization objective for the word2vec model. What exactly does the inner product of two word vectors in word2vec preserve? And do the embeddings necessarily become better by increasing the dimensionality d?

在这种情况下，我们知道对于词嵌入 u 和文档嵌入 v，u 与 v 的内积保留了单词 u 在文档 v 中出现次数的信息。嵌入维度 d 越大，近似效果越好。不幸的是，对于 word2vec 模型的优化目标并没有这样清晰的表述。word2vec 中两个词向量的内积究竟保留了什么？增加维度 d 是否必然使嵌入变得更好？

A research paper by Levy and Goldberg answers exactly this question [1]. In this article I present the theoretical results from [1] and later show how they can be used to design a more general class of embeddings.

Levy 和 Goldberg 的一篇研究论文恰好回答了这个问题 [1]。在本文中，我介绍了 [1] 中的理论结果，并随后展示它们如何用于设计更通用的嵌入类别。

Training word embeddings: word2vec

训练词嵌入：word2vec

Let us briefly consider how word2vec with negative sampling works. For a more comprehensive description, we refer to this article. Let D be the corpus consisting of word, context pairs. In word2vec the context of word w is defined as the k words surrounding w where k is usually a small constant varying between 5 and 15. We want to learn word embeddings such that if two words frequently co-occur in the corpus, their inner product is large.

让我们简要回顾一下带负采样的 word2vec 是如何工作的。更全面的描述请参考这篇文章。设 D 为由词-上下文对组成的语料库。在 word2vec 中，词 w 的上下文被定义为围绕 w 的 k 个词，其中 k 通常是一个介于 5 到 15 之间的小常数。我们希望学习词嵌入，使得如果两个词在语料库中频繁共现，它们的内积就很大。

Consider a word w and let c be a word in its context. For word pairs (w,c) occurring together in the corpus, we want the inner product of the embeddings to maximize the probability that (w,c) indeed appears in the corpus (denoted as D=1). The probability is modeled by a sigmoid function:

考虑一个词 w，设 c 是它上下文中的一个词。对于在语料库中共同出现的词对 (w,c)，我们希望嵌入的内积能够最大化 (w,c) 确实出现在语料库中（记为 D=1）的概率。该概率由 sigmoid 函数建模：

The above has a trivial solution, we can simply make all inner products arbitrary large. Thus, we also introduce negative pairs, i.e. pairs that do not co-occur in the corpus, for which the objective is:

上式有一个平凡解，我们可以简单地将所有内积设得任意大。因此，我们还引入负样本对，即不在语料库中共现的对，其目标为：

The algorithm can be summarized as follows:

该算法可总结如下：

We run the above algorithm for several epochs over the corpus in order to guarantee that the learning process converges in an optimum.

我们在语料库上运行上述算法多个周期，以保证学习过程收敛到最优解。

The theoretical analysis

理论分析

Fix a word w and consider the objective for all pairs in which w appears. Let #(w,c) be the number of appearances of the pair (w,c) in the corpus. We can write the objective as

固定一个词 w，考虑所有包含 w 的对的目标函数。设 #(w,c) 为词对 (w,c) 在语料库中出现的次数。我们可以将目标函数写为：

where the second product is over the negative pairs we generate.

其中第二个乘积是针对我们生成的负样本对的。

By taking the logarithm of the objective and observing that each negative word cN has a chance to be sampled, we obtain:

通过对目标函数取对数，并观察到每个负样本词 cN 都有被采样的机会，我们得到：

Let us explain the above. The word w is fixed, we consider all word context pairs (w,c) that appear in the corpus and we sample k negative pairs (w, cN) such that each word c is sampled with probability #(c)/|D|. We want for positive pairs the inner product to be a large positive number. For negative pairs we want the inner product to be a negative number with large absolute value.

让我们解释一下上式。词 w 是固定的，我们考虑语料库中出现的所有词-上下文对 (w,c)，并采样 k 个负样本对 (w,cN)，使得每个词 c 以概率 #(c)/|D| 被采样。对于正样本对，我们希望内积是一个较大的正数；对于负样本对，我们希望内积是一个绝对值较大的负数。

Observe that |D|, the number of pairs in the corpus, is constant. Thus, by dividing the above expression by |D| the objective becomes

注意到 |D|，即语料库中的对数，是一个常数。因此，将上式除以 |D|，目标函数变为：

This already provides us with some intuition. The objective is to optimize the embeddings such that they reflect the probability for a positive pair to be sampled as opposed to a pair being sampled at random. Positive pairs are generated with probability

这已经为我们提供了一些直觉。该目标是优化嵌入，使其反映正样本对被采样的概率与随机采样一对的概率之间的对比。正样本对以概率：

And for negative pairs, two words are sampled independently at random, each with probability

生成。而对于负样本对，两个词独立随机采样，每个词以概率：

By setting the inner product as an unknown parameter and solving the corresponding optimization problem, we can find the optimal value for the inner product:

被采样。通过将内积设为未知参数并求解相应的优化问题，我们可以找到内积的最优值：

In the above P(w,c) is the probability of occurrence of the pair (w,c), and P(w) is the marginal probability of occurrence of word w in the corpus. The above turns out to be a widely used word association measure in natural language processing, the pointwise-mutual information (PMI) measure.

其中 P(w,c) 是词对 (w,c) 出现的概率，P(w) 是词 w 在语料库中出现的边缘概率。上式恰好是自然语言处理中广泛使用的词关联度量——点互信息（PMI）度量。

This is pretty amazing! It turns out that word2vec is essentially equivalent to matrix factorization where the matrix entries are the PMI scores between word pairs. And PMI as a distance measure was used for NLP-related tasks since the 80s [2], long before the emergence of the concept of word embeddings.

这相当惊人！事实证明，word2vec 本质上等价于矩阵分解，其中矩阵条目是词对之间的 PMI 分数。而 PMI 作为距离度量自 80 年代以来就用于 NLP 相关任务 [2]，远早于词嵌入概念的出现。

Embeddings based on arbitrary similarity functions

基于任意相似度函数的嵌入

Now it is easy to see that we can simply replace the probability for sampling positive and negative pairs. We only need to update the second and third steps in the word2vec algorithm presented above:

现在很容易看出，我们可以简单地替换采样正样本对和负样本对的概率。我们只需要更新上述 word2vec 算法中的第二步和第三步：

Why is this helpful? This gives us more freedom to assign importance to pairs. We can become creative and consider different similarity measures. For example, the Jaccard similarity between words is defined as follows:

这为什么有用？这给了我们更大的自由度来为词对分配重要性。我们可以发挥创造力，考虑不同的相似度度量。例如，词之间的 Jaccard 相似度定义如下：

Thus, we can learn embeddings that optimize the objective that words w and c are similar to each other if the presence of w implies that it is likely that c also appears in the document, and vice versa. In this case, the pair (“keira”, “knightley”) will likely have a higher score than (“data”, “science”). The objective becomes:

因此，我们可以学习嵌入来优化这样的目标：如果词 w 的出现意味着词 c 也很可能出现在文档中，反之亦然，那么 w 和 c 彼此相似。在这种情况下，词对 ("keira", "knightley") 可能比 ("data", "science") 获得更高的分数。目标函数变为：

And we can also model the probability for generating negative pairs. For example, Pr(w) can be the uniform distribution where all words have the same probability of being selected, disregarding how often they appear.

我们也可以对生成负样本对的概率进行建模。例如，Pr(w) 可以是均匀分布，所有词被选中的概率相同，而不考虑它们出现的频率。

Sampling from a distribution

从分布中采样

If we could compute and store the similarity for all pairs (u, v), then sampling according to the similarity becomes trivial: just store the pairs with their similarity scores as weights and sample using an algorithm like numpy.random.choice. However, this might be computationally infeasible.

如果我们能够计算并存储所有词对 (u,v) 的相似度，那么根据相似度进行采样就变得非常简单：只需将词对及其相似度分数作为权重存储，并使用 numpy.random.choice 等算法进行采样。然而，这在计算上可能不可行。

There are different approaches to deal with the problem with a larger number of pairs. In general, we want to use as positive pairs only those that have a high similarity score.

有几种不同的方法可以处理大规模词对的问题。一般来说，我们希望只将那些具有高相似度分数的词对作为正样本对。

If your similarity measure is based mainly on counts, then a subsample of the data will preserve the most frequent pairs but many infrequent pairs will be filtered out. For example, we can consider only a subset of the documents in a corpus. Frequent word pairs such as (“data”, “science”) will likely survive. But this might not be the case with (“keira”, “knightley”).

如果你的相似度度量主要基于计数，那么对数据的一个子集进行采样将保留最频繁的词对，但许多不频繁的词对会被过滤掉。例如，我们可以只考虑语料库中的一部分文档。像 ("data", "science") 这样的频繁词对很可能保留下来，但 ("keira", "knightley") 可能就不一定了。

For each object, consider only the t nearest neighbors. For example, we might use the publicly available implementation from scikit-learn which uses algorithms like kd-trees to speed up similarity search. These algorithms work well for data that is not very high dimensional. Otherwise, one can consider approaches such as Locality-sensitive hashing that will generate similar words. This is especially true for measures like Jaccard similarity.

对于每个对象，只考虑 t 个最近邻。例如，我们可以使用 scikit-learn 的公开实现，它使用 kd-tree 等算法来加速相似度搜索。这些算法对于维数不高的数据效果很好。否则，可以考虑使用局部敏感哈希等方法生成相似词。对于 Jaccard 相似度等度量尤其如此。

A practical implementation

实际实现

For illustrative purposes, we implemented a simple solution for learning document embeddings from text corpora. The problem is orthogonal to the problem of training word embeddings: we train vector representations for documents based on the words they contain.

出于说明目的，我们实现了一个简单的解决方案，用于从文本语料库中学习文档嵌入。该问题与训练词嵌入的问题正交：我们基于文档所包含的单词来训练文档的向量表示。

We consider the IMDB sentiment analysis dataset. The dataset consists of movie reviews by users and each review is labeled with a positive or negative sentiment.

我们使用 IMDB 情感分析数据集。该数据集包含用户对电影的评论，每条评论都标注了正面或负面情感。

After preprocessing the text, we transformed the documents to vectors by using a tf-idf encoding such that each document

对文本进行预处理后，我们使用 tf-idf 编码将文档转换为向量，使得每个文档：

The parameter min_df says we consider only words that appear in at least 0.1% of the documents. Essentially, this prevents us from using very specific words that might appear just in a couple of documents.

参数 min_df 表示我们只考虑出现在至少 0.1% 的文档中的词。本质上，这可以防止我们使用那些可能只出现在少数文档中的非常特定的词。

For each input vector, find its t nearest neighbors. This can be achieved using an off-the-shelf package such as scikit-learn’s K-NearestNeighbor which returns nearest neighbors for:

对于每个输入向量，找到其 t 个最近邻。这可以使用现成的包来实现，例如 scikit-learn 的 K-NearestNeighbor，它返回以下项的最近邻：

Compute the similarities for the generated n*t positive pairs, sort them in an array, and sample according to their weight using numpy.random.choice():

计算生成的 n*t 个正样本对的相似度，将它们排序到一个数组中，并使用 numpy.random.choice() 根据其权重进行采样：

Use a Keras generator to generate positive and negative pairs:

使用 Keras 生成器生成正样本对和负样本对：

Feed the generated pairs into a shallow neural network with an embeddings layer, a Dot layer computing the inner product, and an output layer with a sigmoid activation function:

将生成的词对输入到一个浅层神经网络中，该网络包含一个嵌入层、一个计算内积的 Dot 层以及一个带有 sigmoid 激活函数的输出层：

The above approach will train embeddings:

上述方法将训练嵌入：

Then we can extract the embedding layer for each word and cluster the documents (similarly to what is shown in the gif in Figure 1). We observe that the sentiment distribution in the two clusters is very different:

然后我们可以提取每个词的嵌入层并对文档进行聚类（类似于图1中动图所示）。我们观察到两个聚类中的情感分布非常不同：

Code

代码

The Python implementation for the above is publicly available at: https://github.com/konstantinkutzkov/sim2vec

上述内容的 Python 实现可公开获取于：https://github.com/konstantinkutzkov/sim2vec

[1] Omer Levy, Yoav Goldberg. Neural Word Embedding as Implicit Matrix Factorization. NIPS 2014: 2177-2185

[1] Omer Levy, Yoav Goldberg. Neural Word Embedding as Implicit Matrix Factorization. NIPS 2014: 2177-2185

[2] Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22–29, 1990.

[2] Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22-29, 1990.