Cross-entropy
Cross-entropy scores a predicted probability distribution against observed targets. For next-token training, minimizing cross-entropy is equivalent to maximizing the likelihood of the observed token sequence.
H(p,q)=-\sum_i p_i\log q_i
Overview. Cross-entropy scores a predicted probability distribution against observed targets.
Context. For next-token training, minimizing cross-entropy is equivalent to maximizing the likelihood of the observed token sequence.
Claims and evidence
Claim ledger, 2Cross-entropy scores a predicted probability distribution against observed targets.
For next-token training, minimizing cross-entropy is equivalent to maximizing the likelihood of the observed token sequence.