Get Expert Help From The Gensim Authors
• Consulting in Machine Learning & NLP
• Commercial document similarity engine: ScaleText.ai
• Corporate trainings in Python Data Science and Deep Learning
models.word2vec – Word2vec embeddings¶This module implements the word2vec family of algorithms, using highly optimized C routines, data streaming and Pythonic interfaces.
The word2vec algorithms include skip-gram and CBOW models, using either hierarchical softmax or negative sampling: Tomas Mikolov et al: Efficient Estimation of Word Representations in Vector Space, Tomas Mikolov et al: Distributed Representations of Words and Phrases and their Compositionality.
There are more ways to train word vectors in Gensim than just Word2Vec.
See also Doc2Vec, FastText and
wrappers for VarEmbed and WordRank.
The training algorithms were originally ported from the C package https://code.google.com/p/word2vec/ and extended with additional functionality and optimizations over the years.
For a tutorial on Gensim word2vec, with an interactive web app trained on GoogleNews, visit https://rare-technologies.com/word2vec-tutorial/.
Make sure you have a C compiler before installing Gensim, to use the optimized word2vec routines (70x speedup compared to plain NumPy implementation, https://rare-technologies.com/parallelizing-word2vec-in-python/).
Initialize a model with e.g.:
>>> from gensim.test.utils import common_texts, get_tmpfile
>>> from gensim.models import Word2Vec
>>>
>>> path = get_tmpfile("word2vec.model")
>>>
>>> model = Word2Vec(common_texts, size=100, window=5, min_count=1, workers=4)
>>> model.save("word2vec.model")
The training is streamed, meaning sentences can be a generator, reading input data from disk on-the-fly, without loading the entire corpus into RAM.
It also means you can continue training the model later:
>>> model = Word2Vec.load("word2vec.model")
>>> model.train([["hello", "world"]], total_examples=1, epochs=1)
(0, 2)
The trained word vectors are stored in a KeyedVectors instance in model.wv:
>>> vector = model.wv['computer'] # numpy vector of a word
The reason for separating the trained vectors into KeyedVectors is that if you don’t need the full model state any more (don’t need to continue training), the state can discarded, resulting in a much smaller and faster object that can be mmapped for lightning fast loading and sharing the vectors in RAM between processes:
>>> from gensim.models import KeyedVectors
>>>
>>> path = get_tmpfile("wordvectors.kv")
>>>
>>> model.wv.save(path)
>>> wv = KeyedVectors.load("model.wv", mmap='r')
>>> vector = wv['computer'] # numpy vector of a word
Gensim can also load word vectors in the “word2vec C format”, as a
KeyedVectors instance:
>>> from gensim.test.utils import datapath
>>>
>>> wv_from_text = KeyedVectors.load_word2vec_format(datapath('word2vec_pre_kv_c'), binary=False) # C text format
>>> wv_from_bin = KeyedVectors.load_word2vec_format(datapath("euclidean_vectors.bin"), binary=True) # C bin format
It is impossible to continue training the vectors loaded from the C format because the hidden weights,
vocabulary frequencies and the binary tree are missing. To continue training, you’ll need the
full Word2Vec object state, as stored by save(),
not just the KeyedVectors.
You can perform various NLP word tasks with a trained model. Some of them
are already built-in - you can see it in gensim.models.keyedvectors.
If you’re finished training a model (i.e. no more updates, only querying),
you can switch to the KeyedVectors instance:
>>> word_vectors = model.wv
>>> del model
to trim unneeded model state = use much less RAM and allow fast loading and memory sharing (mmap).
Note that there is a gensim.models.phrases module which lets you automatically
detect phrases longer than one word. Using phrases, you can learn a word2vec model
where “words” are actually multiword expressions, such as new_york_times or financial_crisis:
>>> from gensim.test.utils import common_texts
>>> from gensim.models import Phrases
>>>
>>> bigram_transformer = Phrases(common_texts)
>>> model = Word2Vec(bigram_transformer[common_texts], min_count=1)
gensim.models.word2vec.BrownCorpus(dirname)¶Bases: object
Iterate over sentences from the Brown corpus (part of NLTK data).
gensim.models.word2vec.LineSentence(source, max_sentence_length=10000, limit=None)¶Bases: object
Iterate over a file that contains sentences: one line = one sentence. Words must be already preprocessed and separated by whitespace.
| Parameters: |
|
|---|
Examples
>>> from gensim.test.utils import datapath
>>> sentences = LineSentence(datapath('lee_background.cor'))
>>> for sentence in sentences:
... pass
gensim.models.word2vec.PathLineSentences(source, max_sentence_length=10000, limit=None)¶Bases: object
Like LineSentence, but process all files in a directory
in alphabetical order by filename.
The directory must only contain files that can be read by gensim.models.word2vec.LineSentence:
.bz2, .gz, and text files. Any file not ending with .bz2 or .gz is assumed to be a text file.
The format of files (either text, or compressed text files) in the path is one sentence = one line, with words already preprocessed and separated by whitespace.
Warning
Does not recurse into subdirectories.
| Parameters: |
|
|---|
gensim.models.word2vec.Text8Corpus(fname, max_sentence_length=10000)¶Bases: object
Iterate over sentences from the “text8” corpus, unzipped from http://mattmahoney.net/dc/text8.zip.
gensim.models.word2vec.Word2Vec(sentences=None, corpus_file=None, size=100, alpha=0.025, window=5, min_count=5, max_vocab_size=None, sample=0.001, seed=1, workers=3, min_alpha=0.0001, sg=0, hs=0, negative=5, ns_exponent=0.75, cbow_mean=1, hashfxn=<built-in function hash>, iter=5, null_word=0, trim_rule=None, sorted_vocab=1, batch_words=10000, compute_loss=False, callbacks=(), max_final_vocab=None)¶Bases: gensim.models.base_any2vec.BaseWordEmbeddingsModel
Train, use and evaluate neural networks described in https://code.google.com/p/word2vec/.
Once you’re finished training a model (=no more updates, only querying)
store and use only the KeyedVectors instance in self.wv to reduce memory.
The model can be stored/loaded via its save() and
load() methods.
The trained word vectors can also be stored/loaded from a format compatible with the
original word2vec implementation via self.wv.save_word2vec_format
and gensim.models.keyedvectors.KeyedVectors.load_word2vec_format().
Some important attributes are the following:
wv¶This object essentially contains the mapping between words and embeddings. After training, it can be used directly to query those embeddings in various ways. See the module level docstring for examples.
| Type: | Word2VecKeyedVectors |
|---|
vocabulary¶This object represents the vocabulary (sometimes called Dictionary in gensim) of the model. Besides keeping track of all unique words, this object provides extra functionality, such as constructing a huffman tree (frequent words are closer to the root), or discarding extremely rare words.
| Type: | Word2VecVocab |
|---|
trainables¶This object represents the inner shallow neural network used to train the embeddings. The semantics of the network differ slightly in the two available training modes (CBOW or SG) but you can think of it as a NN with a single projection and hidden layer which we train on the corpus. The weights are then used as our embeddings (which means that the size of the hidden layer is equal to the number of features self.size).
| Type: | Word2VecTrainables |
|---|
| Parameters: |
|
|---|
Examples
Initialize and train a Word2Vec model
>>> from gensim.models import Word2Vec
>>> sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]]
>>> model = Word2Vec(sentences, min_count=1)
accuracy(**kwargs)¶Deprecated. Use self.wv.accuracy instead.
See accuracy().
build_vocab(sentences=None, corpus_file=None, update=False, progress_per=10000, keep_raw_vocab=False, trim_rule=None, **kwargs)¶Build vocabulary from a sequence of sentences (can be a once-only generator stream).
| Parameters: |
|
|---|
build_vocab_from_freq(word_freq, keep_raw_vocab=False, corpus_count=None, trim_rule=None, update=False)¶Build vocabulary from a dictionary of word frequencies.
| Parameters: |
|
|---|
clear_sims()¶Remove all L2-normalized word vectors from the model, to free up memory.
You can recompute them later again using the init_sims() method.
cum_table¶delete_temporary_training_data(replace_word_vectors_with_normalized=False)¶Discard parameters that are used in training and scoring, to save memory.
Warning
Use only if you’re sure you’re done training a model.
| Parameters: | replace_word_vectors_with_normalized (bool, optional) – If True, forget the original (not normalized) word vectors and only keep the L2-normalized word vectors, to save even more memory. |
|---|
doesnt_match(**kwargs)¶Deprecated, use self.wv.doesnt_match() instead.
Refer to the documentation for doesnt_match().
estimate_memory(vocab_size=None, report=None)¶Estimate required memory for a model using current settings and provided vocabulary size.
| Parameters: |
|
|---|---|
| Returns: | A dictionary from string representations of the model’s memory consuming members to their size in bytes. |
| Return type: | dict of (str, int) |
evaluate_word_pairs(**kwargs)¶Deprecated, use self.wv.evaluate_word_pairs() instead.
Refer to the documentation for
evaluate_word_pairs().
get_latest_training_loss()¶Get current value of the training loss.
| Returns: | Current training loss. |
|---|---|
| Return type: | float |
hashfxn¶init_sims(replace=False)¶Deprecated. Use self.wv.init_sims instead.
See init_sims().
intersect_word2vec_format(fname, lockf=0.0, binary=False, encoding='utf8', unicode_errors='strict')¶Merge in an input-hidden weight matrix loaded from the original C word2vec-tool format, where it intersects with the current vocabulary.
No words are added to the existing vocabulary, but intersecting words adopt the file’s weights, and non-intersecting words are left alone.
| Parameters: |
|
|---|
iter¶layer1_size¶load(*args, **kwargs)¶Load a previously saved Word2Vec model.
See also
save()| Parameters: | fname (str) – Path to the saved file. |
|---|---|
| Returns: | Loaded model. |
| Return type: | Word2Vec |
load_word2vec_format(fname, fvocab=None, binary=False, encoding='utf8', unicode_errors='strict', limit=None, datatype=<type 'numpy.float32'>)¶Deprecated. Use gensim.models.KeyedVectors.load_word2vec_format() instead.
log_accuracy(section)¶Deprecated. Use self.wv.log_accuracy instead.
See log_accuracy().
min_count¶most_similar(**kwargs)¶Deprecated, use self.wv.most_similar() instead.
Refer to the documentation for most_similar().
most_similar_cosmul(**kwargs)¶Deprecated, use self.wv.most_similar_cosmul() instead.
Refer to the documentation for
most_similar_cosmul().
n_similarity(**kwargs)¶Deprecated, use self.wv.n_similarity() instead.
Refer to the documentation for n_similarity().
predict_output_word(context_words_list, topn=10)¶Get the probability distribution of the center word given context words.
| Parameters: |
|
|---|---|
| Returns: | topn length list of tuples of (word, probability). |
| Return type: | list of (str, float) |
reset_from(other_model)¶Borrow shareable pre-built structures from other_model and reset hidden layer weights.
Useful when testing multiple models on the same corpus in parallel.
| Parameters: | other_model (Word2Vec) – Another model to copy the internal structures from. |
|---|
sample¶save(*args, **kwargs)¶Save the model.
This saved model can be loaded again using load(), which supports
online training and getting vectors for vocabulary words.
| Parameters: | fname (str) – Path to the file. |
|---|
save_word2vec_format(fname, fvocab=None, binary=False)¶Deprecated. Use model.wv.save_word2vec_format instead.
See gensim.models.KeyedVectors.save_word2vec_format().
score(sentences, total_sentences=1000000, chunksize=100, queue_factor=2, report_delay=1)¶Score the log probability for a sequence of sentences.
This does not change the fitted model in any way (see train() for that).
Gensim has currently only implemented score for the hierarchical softmax scheme, so you need to have run word2vec with hs=1 and negative=0 for this to work.
Note that you should specify total_sentences; you’ll run into problems if you ask to score more than this number of sentences but it is inefficient to set the value too high.
See the article by Matt Taddy: “Document Classification by Inversion of Distributed Language Representations” and the gensim demo for examples of how to use such scores in document classification.
| Parameters: |
|
|---|
similar_by_vector(**kwargs)¶Deprecated, use self.wv.similar_by_vector() instead.
Refer to the documentation for similar_by_vector().
similar_by_word(**kwargs)¶Deprecated, use self.wv.similar_by_word() instead.
Refer to the documentation for similar_by_word().
similarity(**kwargs)¶Deprecated, use self.wv.similarity() instead.
Refer to the documentation for similarity().
syn0_lockf¶syn1¶syn1neg¶train(sentences=None, corpus_file=None, total_examples=None, total_words=None, epochs=None, start_alpha=None, end_alpha=None, word_count=0, queue_factor=2, report_delay=1.0, compute_loss=False, callbacks=())¶Update the model’s neural weights from a sequence of sentences.
Notes
To support linear learning-rate decay from (initial) alpha to min_alpha, and accurate
progress-percentage logging, either total_examples (count of sentences) or total_words (count of
raw words in sentences) MUST be provided. If sentences is the same corpus
that was provided to build_vocab() earlier,
you can simply use total_examples=self.corpus_count.
Warning
To avoid common mistakes around the model’s ability to do multiple training passes itself, an
explicit epochs argument MUST be provided. In the common and recommended case
where train() is only called once, you can set epochs=self.iter.
| Parameters: |
|
|---|
Examples
>>> from gensim.models import Word2Vec
>>> sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]]
>>>
>>> model = Word2Vec(min_count=1)
>>> model.build_vocab(sentences) # prepare the model vocabulary
>>> model.train(sentences, total_examples=model.corpus_count, epochs=model.iter) # train word vectors
(1, 30)
wmdistance(**kwargs)¶Deprecated, use self.wv.wmdistance() instead.
Refer to the documentation for wmdistance().
gensim.models.word2vec.Word2VecTrainables(vector_size=100, seed=1, hashfxn=<built-in function hash>)¶Bases: gensim.utils.SaveLoad
Represents the inner shallow neural network used to train Word2Vec.
load(fname, mmap=None)¶Load an object previously saved using save() from a file.
| Parameters: |
|
|---|
See also
save()| Returns: | Object loaded from fname. |
|---|---|
| Return type: | object |
| Raises: | AttributeError – When called on an object instance instead of class (this is a class method). |
prepare_weights(hs, negative, wv, update=False, vocabulary=None)¶Build tables and model weights based on final vocabulary settings.
reset_weights(hs, negative, wv)¶Reset all projection weights to an initial (untrained) state, but keep the existing vocabulary.
save(fname_or_handle, separately=None, sep_limit=10485760, ignore=frozenset([]), pickle_protocol=2)¶Save the object to a file.
| Parameters: |
|
|---|
See also
load()seeded_vector(seed_string, vector_size)¶Get a random vector (but deterministic by seed_string).
update_weights(hs, negative, wv)¶Copy all the existing weights, and reset the weights for the newly added vocabulary.
gensim.models.word2vec.Word2VecVocab(max_vocab_size=None, min_count=5, sample=0.001, sorted_vocab=True, null_word=0, max_final_vocab=None, ns_exponent=0.75)¶Bases: gensim.utils.SaveLoad
Vocabulary used by Word2Vec.
add_null_word(wv)¶create_binary_tree(wv)¶Create a binary Huffman tree using stored vocabulary
word counts. Frequent words will have shorter binary codes.
Called internally from build_vocab().
load(fname, mmap=None)¶Load an object previously saved using save() from a file.
| Parameters: |
|
|---|
See also
save()| Returns: | Object loaded from fname. |
|---|---|
| Return type: | object |
| Raises: | AttributeError – When called on an object instance instead of class (this is a class method). |
make_cum_table(wv, domain=2147483647)¶Create a cumulative-distribution table using stored vocabulary word counts for drawing random words in the negative-sampling training routines.
To draw a word index, choose a random integer up to the maximum value in the table (cum_table[-1]), then finding that integer’s sorted insertion point (as if by bisect_left or ndarray.searchsorted()). That insertion point is the drawn index, coming up in proportion equal to the increment at that slot.
Called internally from build_vocab().
prepare_vocab(hs, negative, wv, update=False, keep_raw_vocab=False, trim_rule=None, min_count=None, sample=None, dry_run=False)¶Apply vocabulary settings for min_count (discarding less-frequent words) and sample (controlling the downsampling of more-frequent words).
Calling with dry_run=True will only simulate the provided settings and report the size of the retained vocabulary, effective corpus length, and estimated memory requirements. Results are both printed via logging and returned as a dict.
Delete the raw vocabulary after the scaling is done to free up RAM, unless keep_raw_vocab is set.
save(fname_or_handle, separately=None, sep_limit=10485760, ignore=frozenset([]), pickle_protocol=2)¶Save the object to a file.
| Parameters: |
|
|---|
See also
load()scan_vocab(sentences=None, corpus_file=None, progress_per=10000, workers=None, trim_rule=None)¶sort_vocab(wv)¶Sort the vocabulary so the most frequent words have the lowest indexes.
gensim.models.word2vec.score_cbow_pair(model, word, l1)¶Score the trained CBOW model on a pair of words.
| Parameters: | |
|---|---|
| Returns: | Logarithm of the sum of exponentiations of input words. |
| Return type: | float |
gensim.models.word2vec.score_sg_pair(model, word, word2)¶Score the trained Skip-gram model on a pair of words.
| Parameters: | |
|---|---|
| Returns: | Logarithm of the sum of exponentiations of input words. |
| Return type: | float |
gensim.models.word2vec.train_cbow_pair(model, word, input_word_indices, l1, alpha, learn_vectors=True, learn_hidden=True, compute_loss=False, context_vectors=None, context_locks=None, is_ft=False)¶Train the passed model instance on a word and its context, using the CBOW algorithm.
| Parameters: |
|
|---|---|
| Returns: | Error vector to be back-propagated. |
| Return type: | numpy.ndarray |
gensim.models.word2vec.train_sg_pair(model, word, context_index, alpha, learn_vectors=True, learn_hidden=True, context_vectors=None, context_locks=None, compute_loss=False, is_ft=False)¶Train the passed model instance on a word and its context, using the Skip-gram algorithm.
| Parameters: |
|
|---|---|
| Returns: | Error vector to be back-propagated. |
| Return type: | numpy.ndarray |
