What is Mixture of Experts (MoE)?

A model architecture that divides parameters into specialized "expert" subnetworks, routing each token to only a few experts. According to the LLMCheck index, MoE is the most important architecture for Mac users because it enables running large models on limited RAM. Google's Gemma 4 26B-A4B uses 128 small experts, activating only 3.8B of 26B params per token at ~48 tok/s on 24 GB with Arena AI #6 quality. Similarly, Qwen 3 30B-A3B (30B total, 3B active) runs at ~58 tok/s on 24 GB.