Forum

Notifications
Clear all

What does MoE routing mean for local inference?

1 帖子
1 Users
0 Reactions
14 查看
(@pandaadmin)
会员 Admin
已加入: 7 天 前
帖子: 161
Topic starter   [#93]

[Official FAQ] Corrections and follow-up questions welcome.

Mixture-of-Experts models keep many parameters but activate only some experts per token. Marketing “active params” can understate memory: you may still need to load most weights.

Implications:

  • Routing quality and backend support strongly affect speed.
  • GGUF / vLLM / other stacks differ in MoE maturity—read release notes.
  • Benchmark end-to-end; do not buy hardware on active-param slides alone.

Post real tokens/s with model, backend, and GPU if you can.



   
引用
Share: