Mixture-of-Experts (MoE) — Expert Model Collaboration

Mixture-of-Experts (MoE) is a model architecture that improves computational efficiency in large language models while keeping parameter counts high but active computation cost low. It works by activating only specific "expert" sub-networks for each input.