Introduction to Word2Vec (Skip-gram) | by Zolzaya Luvsandorj

NLP Fundamentals

A short introduction to phrase embeddings in Python

When working with textual content information, we have to remodel textual content into numbers. There are other ways to signify textual content in numerical information. Bag of phrases (a.ok.a. BOW) is a well-liked and easy method to signify textual content in numbers. Nonetheless, there isn’t a notion of phrase similarity in bag of phrases as a result of every phrase is represented independently. In consequence, embeddings of phrases like ‘nice’ and ‘superior’ are as comparable to one another as they’re to an embedding of the phrase ‘e book’.

Phrase embedding is one other good way of representing textual content with numbers. With this method, every phrase is represented by an embedding, a dense vector (i.e. an array of numbers). The method preserves relationship between phrases and is ready to seize phrase similarity. Phrases that seem in comparable contexts have nearer vectors within the vector area. In consequence, the phrase ‘nice’ is more likely to have extra comparable embedding to ‘superior’ than ‘e book’.

Photograph by Sebastian Svenson on Unsplash

On this submit, we are going to take a look at an outline of phrase embeddings, particularly a kind of embedding algorithm referred to as Word2Vec, and look below the hood to know how the algorithm operates on a toy instance in Python.

Picture by creator | Comparability of preprocessing an instance doc: “Hi there world!” with two approaches. Assumed vocabulary measurement of 5 for bag of phrases method and embedding measurement of three for phrase embedding.

When utilizing the bag of phrases method, we remodel textual content right into a doc time period matrix of m by n the place m is the variety of paperwork/textual content data and n is the variety of distinctive phrases throughout all paperwork. This normally ends in a giant sparse matrix. If you wish to familiarise with the method intimately, try this tutorial.

In phrase embeddings, every phrase is represented by a vector, normally with a measurement of 100 to 300. Word2Vec is a well-liked technique to create embeddings. The fundamental instinct behind Word2Vec is that this: We are able to get helpful details about a phrase by observing its context/neighbours. In Word2Vec, there are two architectures or studying algorithms we are able to use to acquire vector illustration (simply one other phrase for embedding) of phrases: Continuous Bag of Phrases (a.ok.a. CBOW) and Skip-gram.
◼️ CBOW: Predict focus phrase given surrounding context phrases
◼️ Skip-gram: Predict context phrases given focus phrase (the main target of this submit)

At this stage, this will not make a lot sense. We are going to quickly take a look at an instance of this and it’ll turn into clearer.

When coaching embeddings utilizing the Skip-gram algorithm, we undergo the next three steps at a high-level:
◼️ Get hold of textual content: We begin with unlabelled textual content corpus — so it’s an unsupervised studying drawback.
◼️ Rework information: Then, we preprocess the information and rearrange the preprocessed information into focus phrases as characteristic and context phrases as goal for a fictitious supervised studying drawback. So, it turns into a multiclassification drawback the place P(context phrase | focus phrase). Right here’s an instance of what this would possibly seem like on a single doc:

Picture by creator | We first preprocess textual content into tokens. Then, for every token as focus phrase, we discover the context phrases with a window measurement of two. This implies we contemplate 2 tokens earlier than and after the main target phrase as context phrases. We are able to see that not all tokens have 2 tokens earlier than and after in a small instance textual content like this. In these instances, we use the out there tokens. On this instance, we use the time period phrase and token loosely and interchangeably.

Having a number of targets for a similar characteristic may be complicated to think about. Right here’s one other manner to consider the way to put together the information:

Primarily, we put together characteristic and goal pairs.
◼️ Construct a easy neural community: Then, we prepare a easy neural community with a single hidden layer for the supervised studying drawback utilizing the newly constructed dataset. The principle purpose we’re coaching a neural community is to get the educated weights from the hidden layer which turns into the phrase embeddings. The embedding for phrases that happen in comparable contexts are usually comparable to one another.

Having lined the overview, it’s time to implement it in Python to consolidate what we now have realized.

For the reason that focus of this submit is to develop higher instinct of how the algorithm works, we are going to give attention to constructing it ourselves moderately than utilizing pretrained Word2Vec embeddings to deepen our understanding.

🔗 Disclaimer: Whereas growing the code for this submit, I’ve closely used the next repositories:
◼️ word-embedding-creation by Eligijus112 (His Medium web page: Eligijus Bujokas)
◼️ word2vec_numpy by DerekChia

I wish to thank these superior authors for making their helpful work out there for others. Their repositories are nice further studying sources if you wish to deepen your understanding of word2vec.

🔨 Word2vec with Gensim

We are going to use this pattern toy dataset from Eligijus112’s repository together with his permission. Let’s import libraries and the dataset.

import numpy as np
import pandas as pd
from nltk.tokenize.regexp import RegexpTokenizer
from nltk.corpus import stopwords
from gensim.fashions import Word2Vec, KeyedVectors
from scipy.spatial.distance import cosineimport tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Enter, Denseimport matplotlib.pyplot as plt
import seaborn as sns
sns.set(fashion='darkgrid', context='speak')textual content = ["The prince is the future king.",
"Daughter is the princess.",
"Son is the prince.",
"Only a man can be a king.",
"Only a woman can be a queen.",
"The princess will be a queen.",
"Queen and king rule the realm.", 
"The prince is a strong man.",
"The princess is a beautiful woman.",
"The royal family is the king and queen and their children.",
"Prince is only a boy now.",
"A boy will be a man."]

We are going to now preprocess the textual content very calmly. Let’s create a operate that lowercases textual content, tokenises the paperwork into alphabetic tokens and removes stopwords.

The extent of preprocessing can fluctuate from implementation to implementation. In some implementations, one could select to do little or no preprocessing protecting the textual content nearly as it’s. On the opposite spectrum, one may also select to do a extra thorough preprocessing than this instance.

def preprocess_text(doc):
tokeniser = RegexpTokenizer(r"[A-Za-z]{2,}")
tokens = tokeniser.tokenize(doc.decrease())
key_tokens = [token for token in tokens 
if token not in stopwords.words('english')]
return key_tokenscorpus = []
for doc in textual content:
corpus.append(preprocess_text(doc))
corpus

Now every doc consists of tokens. We are going to construct Word2Vec utilizing Gensim on our customized corpus:

dimension = 2
window = 2word2vec0 = Word2Vec(corpus, min_count=1, vector_size=dimension, 
window=window, sg=1)
word2vec0.wv.get_vector('king')

Picture by creator

We select window measurement of two for the contexts. This implies we are going to look 2 tokens earlier than and after the main target token. dimension can also be set to 2. This refers back to the measurement of the vector. We selected 2 as a result of we are able to simply visualise it in two-dimensional chart and we’re working with a really small textual content corpus. These two hyperparameters might be tuned with totally different values to enhance the usefulness of the phrase embeddings for a use case. Whereas making ready Word2Vec, we ensured to make use of Skip-gram algorithm by specifying sg=1. As soon as the embedding is prepared, we are able to see the embedding for a token 'king'.

Let’s see how intuitive the embeddings are. We are going to decide a pattern phrase: 'king' and see if the phrases which are most much like it within the vector area is sensible. Let’s discover essentially the most comparable 3 phrases to 'king':

n=3
word2vec0.wv.most_similar(constructive=['king'], topn=n)

Picture by creator

This record of tuples present essentially the most comparable phrases and their cosine similarity to 'king'. This consequence is just not unhealthy given we’re working with a really small information.

Let’s put together a DataFrame of the embeddings for the vocabulary, the gathering of distinctive tokens:

embedding0 = pd.DataFrame(columns=['d0', 'd1'])
for token in word2vec0.wv.index_to_key:
embedding0.loc[token] = word2vec0.wv.get_vector(token)
embedding0

Now, we are going to visualise the tokens in two-dimensional vector area:

sns.lmplot(information=embedding0, x='d0', y='d1', fit_reg=False, facet=2)
for token, vector in embedding0.iterrows():
plt.gca().textual content(vector['d0']+.02, vector['d1']+.03, str(token), 
measurement=14)
plt.tight_layout()

🔗 If you wish to be taught extra about Word2Vec in Gensim, right here’s a tutorial by Radim Rehurek, the creator of Gensim.

Alright, this was a pleasant warm-up. Within the subsequent part, we are going to create a Word2Vec embedding ourselves.

🔨 Guide Word2Vec — Strategy 1

We are going to begin by discovering the vocabulary from the corpus. We are going to assign a price to every token within the vocabulary:

vocabulary = sorted([*set([token for document in corpus for token in 
document])])
n_vocabulary = len(vocabulary)
token_index ={token: i for i, token in enumerate(vocabulary)}
token_index

Now, we are going to make token pairs as a preparation for the neural community.

token_pairs = []for doc in corpus:
for i, token in enumerate(doc):
for j in vary(i-window, i+window+1):
if (j>=0) and (j!=i) and (j<len(doc)):
token_pairs.append([token] + [document[j]])n_token_pairs = len(token_pairs)
print(f"{n_token_pairs} token pairs")token_pairs[:5]

Token pairs are prepared however they’re nonetheless in textual content type. Now we have to one-hot-encode them in order that they’re appropriate for the neural community.

X = np.zeros((n_token_pairs, n_vocabulary))
Y = np.zeros((n_token_pairs, n_vocabulary))for i, (focus_token, context_token) in enumerate(token_pairs):    
X[i, token_index[focus_token]] = 1
Y[i, token_index[context_token]] = 1
print(X[:5])

Now that the enter information is prepared, we are able to construct a neural community with a single hidden layer:

tf.random.set_seed(42)
word2vec1 = Sequential([
Dense(units=dimension, input_shape=(n_vocabulary,), 
use_bias=False, name='embedding'),
Dense(units=n_vocabulary, activation='softmax', name='output')
])
word2vec1.compile(loss='categorical_crossentropy', optimizer='adam')
word2vec1.match(x=X, y=Y, epochs=100)

We specified hidden layer to haven’t any bias phrases. Since we wish the hidden layer to have linear activation, we didn’t must specify. The variety of items within the layer displays the scale of the vector: dimension.

Let’s extract the weights, our embeddings, from the hidden layer.

embedding1 = pd.DataFrame(columns=['d0', 'd1'])
for token in token_index.keys():
ind = token_index[token]
embedding1.loc[token] = word2vec1.get_weights()[0][ind]
embedding1

Utilizing our new embeddings, let’s see essentially the most comparable 3 phrases to 'king':

vector1 = embedding1.loc['king']
similarities = {}for token, vector in embedding1.iterrows():
theta_sum = np.dot(vector1, vector)
theta_den = np.linalg.norm(vector1) * np.linalg.norm(vector)
similarities[token] = theta_sum / theta_densimilar_tokens = sorted(similarities.objects(), key=lambda x: x[1], 
reverse=True)
similar_tokens[1:n+1]

Picture by creator

Nice, this is sensible. We are able to save the embeddings and cargo them utilizing Gensim. As soon as loaded into Gensim, we are able to examine our similarity calculation.

with open('embedding1.txt' ,'w') as text_file:
text_file.write(f'{n_vocabulary} {dimension}n')
for token, vector in embedding1.iterrows():
text_file.write(f"{token} {' '.be a part of(map(str, vector))}n")
text_file.shut()embedding1_loaded = KeyedVectors.load_word2vec_format('embedding1.txt', binary=False)
embedding1_loaded.most_similar(constructive=['king'], topn=n)

The similarities calculated by Gensim matches our handbook calculations.

We are going to now visualise the embeddings within the vector area:

sns.lmplot(information=embedding1, x='d0', y='d1', fit_reg=False, facet=2)
for token, vector in embedding1.iterrows():
plt.gca().textual content(vector['d0']+.02, vector['d1']+.03, str(token), 
measurement=14)
plt.tight_layout()