Mathematics
21 entries
| Entry | Summary | As of |
|---|---|---|
| Adamoptimization method | Adam maintains moving estimates of first and second gradient moments to adapt parameter-wise updates. Bias correction compensates for moment estimates initialized at zero during early steps. | 2026-07-13 |
| Attention complexitycomplexity analysis | Dense self-attention constructs interactions for every pair of positions in a sequence. Its score matrix therefore grows quadratically with sequence length before implementation-specific optimizations. | 2026-07-13 |
| Backpropagationdifferentiation algorithm | Backpropagation efficiently applies the chain rule through a composed computational graph. It computes gradients; an optimizer such as stochastic gradient descent or Adam decides how to use them for updates. | 2026-07-13 |
| Bayes' ruleprobability identity | Bayes' rule reverses a conditional probability using a prior and evidence likelihood. The denominator normalizes the posterior and is often the computational challenge in probabilistic models. | 2026-07-13 |
| Chain rulecalculus rule | The chain rule differentiates a composition by multiplying local derivatives along dependency paths. Reverse-mode automatic differentiation reuses these local derivatives efficiently when a scalar loss depends on many parameters. | 2026-07-13 |
| Conditional probabilityprobability concept | Conditional probability represents uncertainty about one event given information about another. Autoregressive language models factor sequence likelihood into conditional next-token probabilities. | 2026-07-13 |
| Cosine similaritysimilarity measure | Cosine similarity normalizes a dot product by both vector norms. It compares direction rather than raw magnitude, but its relevance still depends on how the embedding space was trained. | 2026-07-13 |
| Cross-entropyloss function | Cross-entropy scores a predicted probability distribution against observed targets. For next-token training, minimizing cross-entropy is equivalent to maximizing the likelihood of the observed token sequence. | 2026-07-13 |
| Dot productoperation | The dot product combines coordinate-wise products into a scalar and is used as a similarity score in attention. Its magnitude depends on vector norms, which is why scaled dot-product attention divides by the square root of key dimension. | 2026-07-13 |
| Entropyinformation measure | Entropy measures the expected negative log probability under a distribution. Higher entropy indicates greater uncertainty within the chosen outcome space, not disorder in every informal sense. | 2026-07-13 |
| Gaussian Error Linear Unitactivation function | GELU gates an input smoothly according to the Gaussian cumulative distribution. It differs from ReLU's hard zero threshold and is widely used in Transformer feed-forward blocks. | 2026-07-13 |
| Gradient descentoptimization method | Gradient descent updates parameters in the direction opposite the local loss gradient. The learning rate controls step size; convergence behavior also depends on curvature, noise, scaling, and optimizer state. | 2026-07-13 |
| KL divergenceinformation measure | Kullback-Leibler divergence measures an expected log-density ratio from one distribution to another. It is asymmetric and does not satisfy all requirements of a mathematical distance. | 2026-07-13 |
| Layer normalizationnormalization method | Layer normalization normalizes activations across features within an example and applies learned scale and shift. Unlike batch normalization, its statistics do not require other examples in the mini-batch. | 2026-07-13 |
| Matricesmathematical object | Matrices represent linear maps and batches of vectors throughout neural-network computation. Matrix multiplication composes learned projections, but nonlinearities are required for deep networks to express nonlinear functions. | 2026-07-13 |
| Neural scaling lawsempirical relationship | Scaling-law research models loss as power-law relationships with compute, data, and parameter count over studied regimes. An empirical scaling trend supports planning within its evidence range but is not a physical law guaranteeing every architecture or dataset. | 2026-07-13 |
| Scaled dot-product attentionneural-network operation | Scaled dot-product attention forms weights from query-key products and uses them to combine value vectors. The scale factor counters growth in dot-product variance as key dimension increases. | 2026-07-13 |
| Singular value decompositionmatrix factorization | Singular value decomposition factors a matrix into orthogonal directions and nonnegative singular values. Truncated SVD supplies a best low-rank approximation under common matrix norms and informs compression techniques. | 2026-07-13 |
| Softmaxfunction | Softmax maps a vector of real scores to positive components that sum to one. Subtracting the maximum score before exponentiation preserves the result while improving numerical stability. | 2026-07-13 |
| Tensorsmathematical object | Tensors generalize scalars, vectors, and matrices to arrays with more axes. Axis meaning—batch, token, head, channel, or feature—is part of the computation and must be tracked explicitly. | 2026-07-13 |
| Vectorsmathematical object | Vectors represent ordered numerical components and serve as the basic carrier for embeddings and hidden states. Vector operations become model behavior only after training assigns meaning to dimensions and geometry. | 2026-07-13 |