Self-Attention Explained with PyTorch

Today, I will explain the core idea of a transformer using a small PyTorch code example.

This explanation focuses only on self-attention, which is the heart of transformers.

First, we import PyTorch. PyTorch allows us to work with tensors and matrix operations.

Transformers are built almost entirely using matrix math. Next, we create an input

tensor. This tensor represents one sentence with four words. Each word is represented

as a vector of six numbers. These vectors are called embeddings, and they're how

words are represented inside a model. Now we create three new representations from each word:

query, key, and value. Query means what the word is looking