Source Hugging Face · Published · Open the original ↗
Lab notes
Your Experts Have Partners. There's HOPE for Keeping Them Together
A Blog post by Amazon on Hugging Face
Mixture-of-Experts (MoE) models have enabled state-of-the-art LLMs to be scaled to hundreds of billions of parameters, but they come with a fundamental and costly caveat: every expert MLP must occupy memory at inference time, even though the vast majority are idle for any given token. For example, Qwen3.5-122B has 256 experts in each of its 48 MoE layers, and 95% of its ~250 GB parameter memory footprint is taken up entirely by these expert MLPs alone. However, for any given token, only 8 experts in each layer are actually used.
The opening of the post, quoted unchanged from Hugging Face. The full text continues at the source.