A r/LocalLLaMA post that briefly hit #1 hot rejects tok/s-only bragging. The author ran 9 private C++/CUDA regressions from their own inference engine on 2× RTX 5070 Ti: local Qwen3.8-Flash-Next (NVFP4 / 3-bit) and Qwen3.8-27B vs Claude Opus/Sonnet/Haiku 5.5. Agents had to diagnose and patch from code only (no build/CUDA in the sandbox). Flash-Next landed near Haiku 5.5 (~0.50 vs 0.47); Opus 5.5 still led (~0.74). 3-bit often kept score but made the 27B think ~30–40% longer—so higher tok/s did not mean cheaper per bug fixed. Gemma 4 26B barely worked as an agent here. Limits: 9 tasks, one domain, wide CIs, different harnesses.Practical takeaway: for home labs, measure “bugs fixed per hour” and tokens-per-task, not only decode speed. Aggressive quants can look fast while burning wall-clock on longer reasoning.Do you evaluate quants on your own regressions, or only on perplexity/bench screenshots?Source: https://www.reddit.com/r/LocalLLaMA/search/?q=Not%20another%20tok%2Fs%20post%3A%20local%20quants%20vs%20Claude%20on%209%20real%20bugs&restrict_sr=1&sort=new