[Official FAQ] Share your experience or follow-up questions below.

If generation is unexpectedly slow and your GPU sits idle, the model is probably running on CPU. Here is a checklist.

• Confirm the driver: run nvidia-smi (NVIDIA). If this fails, fix the driver first.
• Check what the tool reports: in Ollama, ollama ps shows how much of the model is on GPU versus CPU. The server log also says which GPU it detected or why none was used.
• Model too big: if it does not fit in VRAM, the tool may place much or all of it on CPU. Try a smaller quant, a smaller model, or lower context.
• Wrong build (llama.cpp): make sure you installed a build with GPU support (CUDA, Metal, Vulkan or ROCm), and that you pass -ngl with a positive number so layers are offloaded.
• Python packages: a CPU-only PyTorch or llama-cpp-python build will not use the GPU. Reinstall the CUDA-enabled version.
• Containers and WSL: Docker needs the NVIDIA Container Toolkit and --gpus all. In WSL2, use the Windows host driver.
• Multiple GPUs or hybrid laptops: the tool may pick the integrated GPU or the wrong device. Check the device selection settings of your tool, for example CUDA_VISIBLE_DEVICES for NVIDIA.
• AMD or Intel GPUs: support depends on card model, OS and backend. Check your tool's supported hardware list.
• Another process: something else may be holding the VRAM. Look at nvidia-smi for other processes.

When asking for help, include your OS, GPU model, driver version, tool and version, and the startup log lines about GPU detection.