Adam
Adam maintains moving estimates of first and second gradient moments to adapt parameter-wise updates. Bias correction compensates for moment estimates initialized at zero during early steps.
\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}
Overview. Adam maintains moving estimates of first and second gradient moments to adapt parameter-wise updates.
Context. Bias correction compensates for moment estimates initialized at zero during early steps.
Claims and evidence
Claim ledger, 2Adam maintains moving estimates of first and second gradient moments to adapt parameter-wise updates.
Bias correction compensates for moment estimates initialized at zero during early steps.