Gradient descent
Gradient descent updates parameters in the direction opposite the local loss gradient. The learning rate controls step size; convergence behavior also depends on curvature, noise, scaling, and optimizer state.
Overview. Gradient descent updates parameters in the direction opposite the local loss gradient. The learning rate controls step size; convergence behavior also depends on curvature, noise, scaling, and optimizer state.
Mechanism and context. The gradient gives the local direction of increasing loss, so subtracting a learning-rate-scaled gradient proposes a lower-loss parameter update. In a neural network this operation is repeated over batches, while the optimizer determines how present and past gradient information shape each step.
Practical use. Use the formula as a debugging map: inspect the loss, gradient scale, learning rate, and parameter change separately. A flat or unstable training curve may reflect the objective, data scale, numerical behavior, or update rule rather than a vague failure of the model.
Related reading. Read Backpropagation for how the gradient is computed through layers, Cross-entropy for a common training objective, and Adam for an adaptive update rule that maintains moment estimates.
Claims and evidence
Claim ledger, 2Gradient descent updates parameters in the direction opposite the local loss gradient.
The learning rate controls step size; convergence behavior also depends on curvature, noise, scaling, and optimizer state.