Local LLM Model Size Chooser: 7B, 14B, 32B or 70B for Your VRAM and Task
Local LLM model size chooser: pick 7B, 14B, 32B or 70B from your VRAM and task with a five-step method, memory formulas and a table of starting sizes.
-
-
What Is GGUF? Getting Started with llama.cpp: Download, Run and Quantize
What is GGUF? Learn the GGUF format, how to install and run llama.cpp, how Q4_K_M and other quantizations change file size, and how it relates to Ollama.
-
LLM Quantization Explained: What Q4_K_M Means and How to Choose a GGUF Quant
LLM quantization explained: what Q4_K_M means, GGUF quant sizes and speeds side by side, and how to pick Q4, Q5 or Q8 for your VRAM, with Ollama commands.
-
How Much VRAM Does an LLM Need? Parameters, Quantization and Context
How much VRAM does an LLM need? Use weights + KV cache + overhead to estimate from parameters, quantization and context, with quick tables for 8GB to 48GB.
