How Large a Model Can a 4090 Run? The 24GB Ceiling for 8B, 32B, 70B and MoE, Worked Out
How large a model can a 4090 run? 24GB limits at BF16, Q8, Q4 and with RAM offload, sizes and speed ceilings for 8B, 32B, 70B and MoE, plus dual-GPU options.
How large a model can a 4090 run? 24GB limits at BF16, Q8, Q4 and with RAM offload, sizes and speed ceilings for 8B, 32B, 70B and MoE, plus dual-GPU options.