How Did Computers Learn to Understand the Meaning of Words?

How did today's AI systems — ChatGPT, Claude, Gemini, Grok — learn to process human language and respond to our questions?

One piece of research marked a pivotal turning point on the way to that answer: word2vec, a method that packed the meaning of words efficiently into vectors.

Why Traditional NLP Could Not Understand Words

Traditional NLP systems treated words simply as indices in a vocabulary — atomic units with no inherent relationship to one another.

For example: apple = 137, banana = 921.

A computer has no way of knowing how similar 137 and 921 are. From the machine's perspective, the relationship between "apple" and "banana" is no different from the relationship between "apple" and "car."

The Core Idea Behind word2vec

The idea of distributed representations had existed before word2vec. What word2vec changed was making it possible to learn those representations quickly and practically at scale on large datasets.

At the same time, it removed the non-linear hidden layer that had accounted for the bulk of computation in earlier neural network models. Rather than building a more sophisticated model, the authors chose to train more efficiently on a larger dataset.

These two ideas — representing words as coordinates, and keeping computation lightweight — were realized through two models: CBOW and Skip-gram.

How Are Vectors Built?

In the old approach: apple = 137, banana = 921. A computer cannot tell how similar 137 and 921 are.

word2vec creates coordinates instead: apple = (0.2, …), banana = (0.19, …). This makes it possible to compute distances between words.

These coordinates are built through three steps.

The first is 1-of-V encoding (One-hot Encoding). Given a vocabulary of size $V$, each word is represented as a vector where only the position corresponding to that word is 1, and every other position is 0. At this stage, there is no information about similarity between words — it is no different from the old 137 and 921.

This encoded word then passes through a Projection Layer. In simple terms, the Projection Layer is a giant coordinate dictionary. The one-hot vector is just an index telling the model which page to open in that dictionary; the values on that page are the word's actual vector.

More precisely, every word passes through a shared Projection Matrix, where each row holds the vector for one word. For example, if one million words are mapped to 300-dimensional vectors, this matrix has dimensions $1{,}000{,}000 \times 300$. When a word's index is multiplied by this matrix, the corresponding dense vector — the coordinates written on that page — is extracted.

At the start, every page in this dictionary is filled with random numbers. But as the model performs the CBOW and Skip-gram tasks, those values become more precise. The model repeatedly asks, "How likely is this word to appear in this context?" and when the prediction is wrong, it nudges the relevant word vector in the Projection Matrix slightly. This process is called backpropagation. After hundreds of millions of such updates, words that frequently appear in similar contexts end up close together in the vector space.

In plain terms: turning words into vectors is like scattering words at random coordinates, then reading billions of sentences and continuously adjusting those coordinates so that words with similar meanings cluster together.

Why Does King − Man + Woman = Queen?

Once training is complete, each word has its own unique vector, and those values encode not just numbers but linguistic regularities.

Because these vectors preserve linear regularity, computing $vector(\text{King}) - vector(\text{Man}) + vector(\text{Woman})$ yields a result closest to $vector(\text{Queen})$.

The same holds for relationships like France:Paris = Germany:Berlin. Through simple algebraic operations on word vectors — known as word offsets — the model can capture complex relationships between words.

How Do CBOW and Skip-gram Differ?

word2vec introduced two new model architectures.

Continuous Bag-of-Words (CBOW)

CBOW predicts the word at the current position from the surrounding context words.

The input layer takes the words surrounding the target word — for example, the four words before and the four words after it.

These input words share the same projection matrix, and their vectors are averaged into a single position. This happens in the projection layer.

Because word order does not affect the projection in this process, the model is called a "Bag-of-Words" model. It is similar to earlier feed-forward neural networks (NNLM), but without the non-linear hidden layer, making it much faster to compute.

CBOW's strength lies in syntactic tasks. Because it learns by averaging surrounding words, it can also be faster to train than Skip-gram.

Think of it like a fill-in-the-blank question on a standardized reading test. There is a gap in the middle of a sentence, and the model fills it in using the words around it as clues. This is why CBOW is strong at capturing grammatical patterns.

Continuous Skip-gram

Skip-gram works in the opposite direction from CBOW: given the current word, it predicts the surrounding words in the same sentence.

The current word is fed into a log-linear classifier, which predicts the words within a window of size $C$ on either side.

Expanding this window $C$ improves the quality of word vectors but increases computational cost. The model also applies distance weighting: words farther from the target are less likely to be closely related, so they are sampled less frequently and given lower weight.

For example, when $C = 10$, a random number $R$ between 1 and 10 is chosen at each step, and $R$ words on each side — a total of $R \times 2$ words — are used as classification labels.

Skip-gram shows dramatically superior performance on semantic tasks compared to CBOW and other neural network models — particularly in capturing precise relationships such as France:Paris = Germany:Berlin, or King − Man + Woman = Queen.

When to Use Each

Using both models simultaneously would be computationally expensive, so in practice one is chosen depending on the objective. In early experiments, CBOW tended to perform better on grammatical similarity and syntactic tasks, while Skip-gram tended to excel on semantic relationships. In practice, however, performance varies with the data and training conditions.

The original paper used Hierarchical Softmax to reduce the output computation cost. Subsequent research also introduced the simpler Negative Sampling. A Huffman tree structure based on word frequency is used to process large vocabularies efficiently.

Why Both Mattered

The two models were also used in combination with other architectures to improve performance. For example, combining the Continuous Skip-gram model with an RNN-based language model on the Microsoft Sentence Completion Challenge yielded higher accuracy than either model alone, setting a new state-of-the-art at the time.

Because the two models have complementary strengths, their outputs can also be combined through a weighted combination to produce more refined results. The flexibility to choose one or combine them depending on the objective was one of the reasons word2vec remained in practical use for so long.

In Formula

CBOW:

$Q = N \times D + D \times \log_{2}{V}$

Skip-gram:

$Q = C \times (D + D \times \log_2{V})$

Where $N$ is the number of context words, $D$ is the vector dimension, $V$ is the vocabulary size, and $C$ is the maximum window distance.

How Did word2vec Spread?

It was not just the paper that was released. Google made the trained vectors and the implementation publicly available as well.

Google released approximately three million word and phrase embeddings trained on roughly 100 billion words of news data at 300 dimensions, along with the code to use them. They also made it possible for anyone to train the CBOW and Skip-gram architectures on their own data.

This is why word2vec spread rapidly beyond the lab and into the broader NLP industry. But what truly mattered was not the library itself — it was bringing the idea of representing words in a meaningful vector space to a practical, usable scale.

The paper also emphasized that a two-stage training approach — learning word vectors first, then stacking other models on top — was highly efficient.

Why Is It Called word2vec?

Let's step back and look at everything covered so far: distributed representations, Projection Layer, CBOW, Skip-gram, Hierarchical Softmax. The names are complex, but all of it was doing one thing.

Converting words into vectors.

The name says it all: "word to vector." Every mechanism in the paper — whether designed to reduce computation or to improve accuracy — was ultimately an answer to a single question: how do we produce good coordinates more efficiently and more precisely?

The Limitations of word2vec

Up to this point, word2vec might seem perfect. But these vectors have one critical limitation.

Each word gets exactly one vector.

Consider the word "bank." It can mean the side of a river, or a financial institution. word2vec cannot distinguish between these two meanings. Regardless of the sentence, "bank" always occupies the same coordinates.

This is because word2vec produces static embeddings. Once training is complete, the vectors are fixed. There is no mechanism to handle words whose meaning shifts depending on context.

Why Does This Lead All the Way to Today's LLMs?

What word2vec created was not just a set of vectors — it was a way of working. The approach of pre-training word vectors first and then building other tasks on top of them became practical and widespread. The practice of reusing pre-trained word vectors across NLP tasks was made practical and widely adopted here.

But word2vec also left an unresolved problem: the vectors are fixed.

ELMo, which was covered in an earlier post, solved that problem. ELMo made it possible for the same word to have a different vector depending on context — so "bank" would have one vector when used to mean a riverbank and another when used to mean a financial institution. That line of development led to BERT, which reads context bidirectionally and captures the meaning of a word in context with even greater precision than ELMo.

Closing

When people learn language, they start with words. The same was true when computers began learning language. It started with converting human words into computer words — numbers, or more precisely, coordinates.

word2vec was not the first model to teach a computer the meaning of words. What it did was scale up the method of encoding relationships between words into a vector space — making it fast and practical at a size that mattered.

Today's LLMs learn contextual token representations through architectures very different from word2vec. But the underlying perspective — that linguistic relationships can be expressed as numbers and positions in space — has carried forward.

The most important thing word2vec left behind is not a library or a famous vector operation.

It showed, at scale, that the relationships in language can be transformed into a structure that can be computed.