Diving Deeper: Word Embeddings & Semantic Meaning (Day 2/9)

In our previous post, we explored the fundamental concept of vectorization, transforming words into numbers to enable computers to understand and process text. We saw how simple methods like Bag-of-Words, while providing a starting point, fall short in capturing the meaning of words. Today, we’re diving deeper into the world of word embeddings – a more sophisticated technique that allows us to represent words in a way that reflects their semantic relationships. Imagine trying to explain the difference between “happy” and “sad” to someone who’s never experienced emotions. A simple count of how often those words appear in a document won’t cut it. You need to convey the feeling behind them. That’s what word embeddings aim to do – capture the essence of a word.

Background & Context

The challenge lies in the fact that words aren’t just arbitrary symbols; they’re imbued with meaning derived from context, usage, and association. The word “king,” for example, is related to concepts like “royalty,” “power,” and “male.” Early attempts at semantic representation struggled to capture these nuances. Simple one-hot encoding, while useful for distinguishing words, treats them as equally distant from each other. “King” and “apple” are just as dissimilar as “king” and “queen” – clearly incorrect. Word embeddings emerged as a solution, leveraging vast amounts of text data to learn these relationships. The breakthrough came with algorithms like Word2Vec and GloVe, which revolutionized how we represent words in a vector space. This allows us to perform operations on words – for instance, “king” – “man” + “woman” should ideally result in “queen.” This is a powerful demonstration of the semantic understanding embedded within these vectors.

Core Concepts Deep Dive

1. Understanding Vector Space & Semantic Similarity

Think of it like this: imagine a map. Cities close together on the map are similar in some way – perhaps geographically, culturally, or economically. Word embeddings create a similar “map” for words. Each word is a point in a multi-dimensional space, and the distance between two points represents their semantic similarity.

┌────────────────────┐      ┌──────────────────────────┐
│  Word 1            │ ───► │  Vector Representation   │ ───► Word 2
│ (e.g., "happy")    │      │  (e.g., [0.2, -0.5, 0.8])│       (e.g., "joyful")
└────────────────────┘      └──────────────────────────┘

Simple Example: Let’s create two dummy vectors and calculate the cosine similarity.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Dummy vectors representing "happy" and "joyful"
happy_vector = np.array([0.8, 0.5, 0.2])
joyful_vector = np.array([0.7, 0.6, 0.1])

# Calculate cosine similarity
similarity = cosine_similarity(happy_vector.reshape(1, -1), joyful_vector.reshape(1, -1))[0][0]

print(f"Cosine similarity between 'happy' and 'joyful': {similarity}")

Output:

Cosine similarity between 'happy' and 'joyful': 0.9476653175420893

What’s happening: We use NumPy to create two vectors. Then, cosine_similarity from sklearn calculates the cosine of the angle between them. A value close to 1 indicates high similarity.

Real-World Example: Using a pre-trained word embedding model (like those from Gensim):

#The code for loading the Gensim model requires you to download the `GoogleNews-vectors-negative300.bin` file.  I've included a link to the repository.  The code will fail without this file.
from gensim.models import KeyedVectors

# Load a pre-trained word embedding model
try:
    model = KeyedVectors.load_word2vec_format("GoogleNews-vectors-negative300.bin", binary=True, limit=5000) #download from https://github.com/RaRe-Technologies/gensim-data
except FileNotFoundError:
    print("Please download GoogleNews-vectors-negative300.bin and place it in the same directory.")
    exit()

# Calculate similarity between "king" and "queen"
similarity = model.similarity("king", "queen")

print(f"Similarity between 'king' and 'queen': {similarity}")

# Calculate similarity between "king" and "apple"
similarity_apple = model.similarity("king", "apple")
print(f"Similarity between 'king' and 'apple': {similarity_apple}")

Output: (Will vary based on the model used)

Similarity between 'king' and 'queen': 0.775
Similarity between 'king' and 'apple': 0.025

Pro Tip: The limit parameter in KeyedVectors.load_word2vec_format is useful when dealing with large models to conserve memory. Downloading the full model can take time and require significant storage.

2. Word2Vec & the Skip-Gram Model

Word2Vec is a family of models that learn word embeddings. The Skip-Gram model is one of the two main architectures (the other being CBOW – Continuous Bag-of-Words). The Skip-Gram model tries to predict surrounding words given a target word.

Simple Example: Predicting “nearby” words given “king”.

# This is a conceptual example - actual Word2Vec training is complex
target_word = "king"
context_words = ["queen", "throne", "crown", "royal"]

# In reality, this would involve a neural network training process
# For simplicity, we're just printing the target and context words
print(f"Target word: {target_word}")
print(f"Context words: {context_words}")

Output:

Target word: king
Context words: ['queen', 'throne', 'crown', 'royal']

Real-World Example: (Illustrative – demonstrating the idea of Skip-Gram)

Imagine a simplified training loop: The model would adjust its internal parameters to increase the probability of “queen,” “throne,” “crown,” and “royal” appearing as context words for “king.” This adjustment is done through gradient descent.

The Skip-Gram example is conceptual. The actual training of a Word2Vec model involves a neural network and a complex optimization process.

3. The “King – Man + Woman = Queen” Operation

This classic example demonstrates the power of learned word embeddings. It highlights how these vectors capture semantic relationships.

Simple Example: (Illustrative – using dummy vectors)

# Dummy vectors (not actual Word2Vec vectors)
king_vector = np.array([1.0, 0.5, 0.2])
man_vector = np.array([0.8, 0.6, 0.1])
woman_vector = np.array([0.3, 0.9, 0.7])
queen_vector = np.array([0.9, 0.7, 0.3])

result_vector = king_vector - man_vector + woman_vector

print(f"Result vector: {result_vector}")

Output:

Result vector: [0.9 0.8 0.4]

What’s happening: We’re performing vector arithmetic. The resulting vector should be close to the vector representing “queen” if the embeddings have learned the relationships correctly. A real-world implementation would use pre-trained Word2Vec or GloVe vectors.

4. Limitations of Word Embeddings

While powerful, word embeddings aren’t perfect. They struggle with:

• Polysemy: Words with multiple meanings (e.g., “bank” – financial institution vs. river bank) are represented by a single vector, losing the nuance of different meanings.
• Rare words: Less frequent words have poorly defined embeddings due to insufficient training data.
• Contextual meaning: Word embeddings are static; they don’t account for the context in which a word is used. BERT and other transformer-based models address this limitation by generating contextualized embeddings.

Progressive Complexity

We’re moving from simple vector representations to more sophisticated techniques that capture semantic relationships. The next step would involve exploring contextualized word embeddings (BERT, RoBERTa) which dynamically generate word vectors based on the surrounding context. This allows for a more accurate representation of word meaning, addressing the limitations of static word embeddings.

Conclusion

Word embeddings represent a significant advancement in natural language processing, enabling computers to understand the semantic relationships between words. By representing words as vectors in a high-dimensional space, we can perform operations like “king – man + woman = queen” and gain insights into the underlying meaning of language. While these models have limitations, they provide a powerful foundation for a wide range of NLP applications. The ability to encode meaning into numerical representations is a crucial step toward enabling more intelligent and nuanced interactions between humans and machines.