Mixture-of-experts models
A sparsely gated mixture-of-experts layer activates only selected expert subnetworks for each input. Conditional computation can increase parameter capacity without using every parameter on every token.
Overview. A sparsely gated mixture-of-experts layer activates only selected expert subnetworks for each input.
Context. Conditional computation can increase parameter capacity without using every parameter on every token.
Claims and evidence
Claim ledger, 2A sparsely gated mixture-of-experts layer activates only selected expert subnetworks for each input.
Conditional computation can increase parameter capacity without using every parameter on every token.