The mathematics of modern AI
Move from vectors and probability to optimization, backpropagation, and the attention equation.
Steps
9 steps-
Vectors
Begin with the objects used to represent features and states.
Vectors represent ordered numerical components and serve as the basic carrier for embeddings and hidden states. Vector operations become model behavior only after training assigns meaning to dimensions and geometry.
-
Matrices
See how collections of vectors become transformations.
Matrices represent linear maps and batches of vectors throughout neural-network computation. Matrix multiplication composes learned projections, but nonlinearities are required for deep networks to express nonlinear functions.
-
Dot product
Connect vector comparison to weighted combinations.
The dot product combines coordinate-wise products into a scalar and is used as a similarity score in attention. Its magnitude depends on vector norms, which is why scaled dot-product attention divides by the square root of key dimension.
-
Conditional probability
Establish how evidence changes a probability.
Conditional probability represents uncertainty about one event given information about another. Autoregressive language models factor sequence likelihood into conditional next-token probabilities.
-
Entropy
Introduce uncertainty and the quantity optimized in many objectives.
Entropy measures the expected negative log probability under a distribution. Higher entropy indicates greater uncertainty within the chosen outcome space, not disorder in every informal sense.
-
Gradient descent
Follow how parameters move toward a lower loss.
Gradient descent updates parameters in the direction opposite the local loss gradient. The learning rate controls step size; convergence behavior also depends on curvature, noise, scaling, and optimizer state.
-
Chain rule
Prepare the calculus used to connect layers during training.
The chain rule differentiates a composition by multiplying local derivatives along dependency paths. Reverse-mode automatic differentiation reuses these local derivatives efficiently when a scalar loss depends on many parameters.
-
Backpropagation
Trace how gradients move through a neural network.
Backpropagation efficiently applies the chain rule through a composed computational graph. It computes gradients; an optimizer such as stochastic gradient descent or Adam decides how to use them for updates.
-
Scaled dot-product attention
Bring the pieces together in a central Transformer operation.
Scaled dot-product attention forms weights from query-key products and uses them to combine value vectors. The scale factor counters growth in dot-product variance as key dimension increases.