[Official FAQ] Corrections and follow-up questions welcome.

During generation, the model caches prior key/value tensors so it need not recompute the whole prompt each step. Longer context, more layers/heads, and more concurrent sessions all grow KV memory—often competing with weights.

Mitigations:

• Cap max context to what you truly need.
• Look for KV quantization / windowing features in your backend docs.
• Remember multi-user ≈ multiple KV footprints.

This is a root cause of slow long chats and surprise OOMs.