[Official FAQ] Share your experience or follow-up questions below.

GGUF is the model file format used by llama.cpp and tools built on it (Ollama, LM Studio and others). The suffix describes the quantization, meaning how many bits are used to store each weight.

• Q8_0: about 8 bits per weight. Very close to the original quality, but the largest file.
• Q6_K, Q5_K_M, Q5_K_S: high quality, medium size. S means small and M means medium, which is a slightly different mix of precision across layers.
• Q4_K_M: around 4–5 bits per weight. A very popular default because it is small and the quality loss is usually acceptable.
• Q3_K_*, Q2_K: much smaller, but quality drops more noticeably. Consider these only if memory is really tight.
• IQ variants: newer "importance-aware" quants that try to keep quality at very low bit counts. Support and speed can vary by backend.
• F16 / BF16: unquantized, largest.

What should I pick?

• Start with Q4_K_M or Q5_K_M.
• If you have spare memory, go up. If you are short, go down one step before switching to a smaller model.
• Small models (a few billion parameters) tend to suffer more from aggressive quantization.
• With the same memory budget, a bigger model at 4-bit often beats a smaller model at 8-bit, but this is a rule of thumb, not a law. Test it on your own tasks.

Also note that GGUF quants are different from GPTQ, AWQ and EXL2, which are other formats used by other tools.

Which quant do you use day to day, and have you noticed quality differences?