The Journey of a Token: What Actually Occurs Inside a Transformer

On this article, you’ll find out how a transformer converts enter tokens into context-aware representations and, finally, next-token possibilities.

Matters we are going to cowl embody:

• How tokenization, embeddings, and positional info put together inputs
• What multi-headed consideration and feed-forward networks contribute inside every layer
• How the ultimate projection and softmax produce next-token possibilities

Let’s get our journey underway.

The Journey of a Token: What Actually Occurs Inside a Transformer (click to enlarge)
Picture by Editor

The Journey Begins

Giant language fashions (LLMs) are based mostly on the transformer structure, a fancy deep neural community whose enter is a sequence of token embeddings. After a deep course of — that appears like a parade of quite a few stacked consideration and feed-forward transformations — it outputs a likelihood distribution that signifies which token needs to be generated subsequent as a part of the mannequin’s response. However how can this journey from inputs to outputs be defined for a single token within the enter sequence?

On this article, you’ll be taught what occurs inside a transformer mannequin — the structure behind LLMs — on the token degree. In different phrases, we are going to see how enter tokens or elements of an enter textual content sequence flip into generated textual content outputs, and the rationale behind the adjustments and transformations that happen contained in the transformer.

The outline of this journey by way of a transformer mannequin can be guided by the above diagram that reveals a generic transformer structure and the way info flows and evolves by way of it.

Getting into the Transformer: From Uncooked Enter Textual content to Enter Embedding

Earlier than getting into the depths of the transformer mannequin, a number of transformations already occur to the textual content enter, primarily so it’s represented in a kind that’s absolutely comprehensible by the inner layers of the transformer.

Tokenization

The tokenizer is an algorithmic part sometimes working in symbiosis with the LLM’s transformer mannequin. It takes the uncooked textual content sequence, e.g. the consumer immediate, and splits it into discrete tokens (typically subword items or bytes, typically complete phrases), with every token within the supply language being mapped to an identifier i.

Token Embeddings

There’s a realized embedding desk E with form |V| × d (vocabulary measurement by embedding dimension). Wanting up the identifiers for a sequence of size n yields an embedding matrix X with form n × d. That’s, every token identifier is mapped to a d-dimensional embedding vector that types one row of X. Two embedding vectors can be comparable to one another if they’re related to tokens which have comparable meanings, e.g. king and emperor, or vice versa. Importantly, at this stage, every token embedding carries semantic and lexical info for that single token, with out incorporating details about the remainder of the sequence (not less than not but).

Positional Encoding

Earlier than absolutely getting into the core elements of the transformer, it’s essential to inject inside every token embedding vector — i.e. inside every row of the embedding matrix X — details about the place of that token within the sequence. That is additionally known as injecting positional info, and it’s sometimes achieved with trigonometric capabilities like sine and cosine, though there are strategies based mostly on realized positional embeddings as effectively. An almost-residual part is summed to the earlier embedding vector e_t related to a token, as follows:

[
x_t^{(0)} = e_t + p_{text{pos}}(t)
]

with p_pos(t) sometimes being a trigonometric-based operate of the token place t within the sequence. In consequence, an embedding vector that previously encoded “what a token is” solely now encodes “what the token is and the place within the sequence it sits”. That is equal to the “enter embedding” block within the above diagram.

Now, time to enter the depths of the transformer and see what occurs inside!

Deep Contained in the Transformer: From Enter Embedding to Output Chances

Let’s clarify what occurs to every “enriched” single-token embedding vector because it goes by way of one transformer layer, after which zoom out to explain what occurs throughout all the stack of layers.

The method

[
h_t^{(0)} = x_t^{(0)}
]

is used to indicate a token’s illustration at layer 0 (the primary layer), whereas extra generically we are going to use ht(l) to indicate the token’s embedding illustration at layer l.

Multi-headed Consideration

The primary main part inside every replicated layer of the transformer is the multi-headed consideration. That is arguably essentially the most influential part in all the structure in the case of figuring out and incorporating into every token’s illustration a variety of significant details about its function in all the sequence and its relationships with different tokens within the textual content, be it syntactic, semantic, or every other kind of linguistic relationship. A number of heads on this so-called consideration mechanism are every specialised in capturing completely different linguistic features and patterns within the token and all the sequence it belongs to concurrently.

The results of having a token illustration ht(l) (with positional info injected a priori, don’t overlook!) touring by way of this multi-headed consideration inside a layer is a context-enriched or context-aware token illustration. Through the use of residual connections and layer normalizations throughout the transformer layer, newly generated vectors change into stabilized blends of their very own earlier representations and the multi-headed consideration output. This helps enhance coherence all through all the course of, which is utilized repeatedly throughout layers.

Feed-forward Neural Community

Subsequent comes one thing comparatively much less advanced: a number of feed-forward neural community (FFN) layers. For example, these will be per-token multilayer perceptrons (MLPs) whose objective is to additional rework and refine the token options which might be regularly being realized.

The primary distinction between the eye stage and this one is that spotlight mixes and incorporates, in every token illustration, contextual info from throughout all tokens, however the FFN step is utilized independently on every token, refining the contextual patterns already built-in to yield helpful “data” from them. These layers are additionally supplemented with residual connections and layer normalizations, and because of this course of, now we have on the finish of a transformer layer an up to date illustration ht(l+1) that can change into the enter to the subsequent transformer layer, thereby getting into one other multi-headed consideration block.

The entire course of is repeated as many occasions because the variety of stacked layers outlined in our structure, thus progressively enriching the token embedding with an increasing number of higher-level, summary, and long-range linguistic info behind these seemingly indecipherable numbers.

Ultimate Vacation spot

So, what occurs on the very finish? On the prime of the stack, after going by way of the final replicated transformer layer, we receive a ultimate token illustration ht*(L) (the place t* denotes the present prediction place) that’s projected by way of a linear output layer adopted by a softmax.

The linear layer produces unnormalized scores known as logits, and the softmax converts these logits into next-token possibilities.

Logits computation:

[
text{logits}_j = W_{text{vocab}, j} cdot h_{t^*}^{(L)} + b_j
]

Making use of softmax to calculate normalized possibilities:

[
text{softmax}(text{logits})_j = frac{exp(text{logits}_j)}{sum_{k} exp(text{logits}_k)}
]

Utilizing softmax outputs as next-token possibilities:

[
P(text{token} = j) = text{softmax}(text{logits})_j
]

These possibilities are calculated for all potential tokens within the vocabulary. The following token to be generated by the LLM is then chosen — typically the one with the very best likelihood, although sampling-based decoding methods are additionally frequent.

Journey’s Finish

This text took a journey, with a delicate degree of technical element, by way of the transformer structure to offer a common understanding of what occurs to the textual content that’s offered to an LLM — essentially the most outstanding mannequin based mostly on a transformer structure — and the way this textual content is processed and remodeled contained in the mannequin on the token degree to lastly flip right into a mannequin’s output: the subsequent phrase to generate.

We hope you’ve gotten loved our travels collectively, and we stay up for the chance to embark upon one other within the close to future.