[Official FAQ] Share your experience or follow-up questions below.

First find out where the time goes: a long wait before the first token (prompt processing) or slow token-by-token output (generation).

Measure first. In Ollama, ollama run <model> --verbose prints timing stats. llama.cpp includes llama-bench.

Common causes and fixes

• Model not fully on GPU: partial CPU offload is a major slowdown. Run ollama ps to see the CPU/GPU split, or check -ngl in llama.cpp. Fix by using a smaller quant, a smaller model, or a shorter context.
• Model too large: generation speed is mostly bound by memory bandwidth, so larger models are slower. Dropping one size class is the most direct fix.
• Quantization level: lower-bit files are smaller and often faster, with some quality trade-off.
• Very long context: long histories slow prompt processing and use more memory. Start a new chat or summarize.
• Flash attention: available in several backends and may help on supported hardware. Check the flag for your version.
• Speculative decoding: a small draft model speeds up a larger one. Supported in llama.cpp and vLLM, with gains that depend on the task.
• System issues: close other GPU-heavy apps, plug in a laptop and use a high-performance power mode, and watch for thermal throttling.
• Reasoning models produce long thinking text, which can feel slow even if the token rate is normal.

For many simultaneous users, a throughput-oriented server such as vLLM is usually a better choice.

Post your GPU, model, quant and measured speed, and others can tell you whether it looks normal.