Train a Local LLM: LoRA Basics, VRAM Math, and Pitfalls
“How do I train a local LLM?” is one of the most common private-deployment questions we hear. Many people assume training means pretraining from scratch and immediately shop for a thousand-GPU cluster. For individuals and most company teams, the realistic path is different: pick an open-weight base model, fine-tune with LoRA/QLoRA on high-quality licensed data, then load the adapter in your local inference stack. This article stays in educational, lawful fine-tuning—tone, format, domain phrasing—and does not provide jailbreak, uncensored, or safety-bypass training recipes.
Verification notes. VRAM figures are estimates from public papers and common rules of thumb, not our lab measurements. Data-size bands are heuristics. Pair with AI training PC config, local LoRA VRAM, and private LLM training for enterprises. Enterprise data requires authorization and redaction review first.
Key takeaways (updated 2026-11-05)
- Separate four levels: prompts → RAG → LoRA/QLoRA → pretraining; most “local training” stops at level three.
Same 7B: Q4 inference is a few GB; QLoRA often fits one consumer GPU; full fine-tuning state can land near ~16 bytes/parameter before activations.
Data quality beats dumping 100k noisy rows; freeze an eval set before you train.
- Training hardware: see A87; inference builds: three budget PCs.
Train a local LLM: choose the right level first
| Level | What you change | Typical data | Compute | When it is enough |
|---|---|---|---|---|
| Prompts & rules | Tone, format, denylist | Dozens of examples | Inference only | Consistent voice |
| RAG | Retrievable knowledge | Document corpus | Inference + embedding | Policy/knowledge Q&A (local RAG) |
| LoRA/QLoRA | Behavior & style | Thousands of curated rows | One to a few GPUs | Strict schemas, domain tone |
| Pretraining | The base itself | Massive token corpora | Clusters | Foundation-model vendors |
If the pain is “the model does not know this year’s internal policy,” try RAG before fine-tuning. Fine-tuning is better at how to speak than at memorizing facts that change weekly.
VRAM math: inference vs QLoRA vs full fine-tuning
“What hardware do I need to train?” depends on the level. These public approximations set the order of magnitude (not full activation accounting):
| Size | Q4 inference (≈) | QLoRA (order of magnitude) | Full FT state (~16 bytes/param, excl. activations) |
|---|---|---|---|
| 7B | ~4.3GB | Often tryable on 12–24GB | ~112GB |
| 14B | ~8.5GB | More commonly 24GB+ | ~224GB |
| 32B | ~19.5GB | Often 48GB-class or smarter parallelism | ~512GB |
Practical advice: learn LoRA on a 7B/8B base + single 24GB GPU. Treating “full fine-tune 70B on a gaming PC” as a weekend project is how budgets die. More hardware detail: training PC config. Occasional jobs: rent vs buy.
Data is the real bottleneck
Training without an evaluation set is flying without instruments. Minimum viable loop:
- Freeze an eval set (tens to hundreds of items) covering formats, terminology, and lawful refusal boundaries—not jailbreaks.
- Train set roughly 500–20,000 rows depending on goal: style transfer wants clean and smaller; domain dialogs need coverage; structured output needs strict schemas.
- Deduplicate, redact, authorize: contracts and personal data do not enter training without approval.
- Human spot-check ~50 random rows as if you would send them to a colleague.
LoRA in one paragraph
Full fine-tuning edits all weights. LoRA inserts low-rank adapters; you can merge later or load adapters at runtime. QLoRA quantizes the base while training LoRA, which is why it dominates personal local training. Engineering essentials: pick a permitted base, do not blindly copy learning rates, watch the eval set, and archive seeds plus config for reproducibility.
Common stacks: PyTorch with a mainstream fine-tuning library; inference via Ollama, llama.cpp, vLLM, or Strata. Windows users can prototype inference with no-CLI install; serious training is still smoother on Linux or WSL2.
Hyperparameters: change less, evaluate more
Sweeping every knob at once hides what worked. Fix a known-good starter config, change data first, then move one major knob per run on the same eval set. Log steps, wall time, peak VRAM, schema success rate, and human pass rate. If metrics stall, inspect contradictory labels before you buy a bigger GPU.
Save checkpoints on a schedule and keep the “best eval so far” artifact, not only the final step. You should be able to resume after interruption; archive the container image or pip freeze so you can reproduce results months later.
Designing an eval set that actually helps
Cover four buckets: your standard happy-path prompts; messy edge cases; lawful compliance refusals your policy requires; and regression items the base already handled that you must not break. Keep train and eval disjoint. Start with automated checks (valid JSON, required keywords, required refusals) plus a small human Likert sample. Enterprises should add role-based differences: the same question may have different allowed answers per role.
Serving the adapter is half the job
After training, load the adapter in the real stack—API server, web UI, or IDE plugin. Tokenizer and chat template must match the base or online quality collapses. Merged weights are simple; runtime adapters are flexible when you juggle many specialties. Blind-test with production-like prompts. For departments, keep a rollback switch to the pure base or previous adapter, aligned with enterprise private deployment versioning.
Pitfalls we see repeatedly
- Fine-tuning problems RAG would solve cheaper.
- Watching train loss without a held-out set.
- Interpreting “more helpful” as “strip safety”—out of scope and risky; see uncensored compliance risks.
- Inflating context and batch until VRAM explodes.
- Skipping regression after merge (chat improves, tools break).
- Checkpoint sprawl filling disks (storage guide).
A weekend experiment plan
- Choose a 7B/8B open-weight base whose license fits your use.
- Prepare ~800 clean instruction rows + 100 eval items.
- Run QLoRA on one 24GB GPU; record hyperparameters.
- Compare base vs adapter on format success, terminology, and policy refusals.
- Scale data or move to 14B only after metrics improve.
Connecting to inference builds and enterprise process
Individuals can attach adapters to local inference; departments need versioning, rollback, access control, and audits (content safety checklist, enterprise custom private deployment). Whether appliances include training features: all-in-one vs DIY.
What we verified—and what we did not
We do not ship reproducible “de-alignment” datasets, do not guarantee any framework’s flags forever, and do not give legal advice. VRAM tables are theoretical magnitudes; drivers, context length, and framework versions change outcomes.
A three-day validation instead of a three-month fantasy
Long guides still leave people unsure what to do on Monday morning. Replace fantasy roadmaps with a three-day validation: day one ends with a successful install and one real conversation; day two runs ten authentic tasks and logs failure modes; day three decides whether to add retrieval, change model size, or simply improve prompts. Hardware purchases and fine-tuning plans should follow evidence, not anxiety.
If the same failure repeats—tables break, citations point at the wrong paragraph—fix the process for that failure class before you buy a more expensive GPU. Table issues may need stricter output constraints or post-processing; citation issues usually need chunking and retrieval work. A tiny spreadsheet of failure classes is already a cheap evaluation set.
Responsibility boundaries (worth repeating)
Local deployment reduces the intuitive risk of data leaving your network; it does not finish compliance for you. Pasting customer lists into prompts, fine-tuning on unauthorized repositories, or exposing an intranet API to the public internet are process failures. Write down which data classes may never enter prompts or train sets, who can merge adapters, and how to halt and audit incidents. Tutorials answer “how to run”; charters answer “whether we may run” and “who owns the blast radius.”
For hobbyists, the bar can be light: do not spread others’ private data, do not pursue illegal uses, and do not treat mystery weights as trusted software. For companies, boundaries must be auditable. In both cases, “open” never means “no obligations.”
How to read neighboring articles on this site
If the bottleneck is buying too large or too small a machine, start with budget builds and VRAM math. If knowledge goes stale, read RAG first. If tone and format are unstable, then read LoRA training. If the pain is organizational access control, read enterprise private deployment and the safety checklist. If the pain is licensing, read the open-weight and free-local boundary pieces. Reading by bottleneck beats reading by publish date. When you hit an internal link, open the relevant section instead of only bookmarking the title.
Maintenance after the exciting first weekend
Weights accumulate, logs grow, drivers age. Schedule a monthly hour: delete unused weights, check disk health, confirm backups still restore, and skim upstream security notes. If you treat a local model like a forever-forgotten portable app, the common six-month ending is a full disk and a broken driver after an OS update—followed by the false conclusion that “local AI does not work.” Maintenance is boring; it is also what keeps the shopping and training advice valuable.
Composite anecdotes (anonymized patterns)
We repeatedly see a student laptop with 8GB VRAM chasing a 70B download “because the chart looked cool,” a startup exposing Ollama on 0.0.0.0 “just for the hackathon,” and a department fine-tuning on raw mail exports that still contain personal data. None of these failures required exotic research. They required slowing down at license text, network bind addresses, and data classification. If your project currently matches one of those patterns, pause new downloads until the basics are written down and reviewed by a second person.
A three-day/6 validation instead of a three-month fantasy
Long guides still leave people unsure what to do on Monday morning. Replace fantasy roadmaps with a three-day validation: day one ends with a successful install and one real conversation; day two runs ten authentic tasks and logs failure modes; day three decides whether to add retrieval, change model size, or simply improve prompts. Hardware purchases and fine-tuning plans should follow evidence, not anxiety.
If the same failure repeats—tables break, citations point at the wrong paragraph—fix the process for that failure class before you buy a more expensive GPU. Table issues may need stricter output constraints or post-processing; citation issues usually need chunking and retrieval work. A tiny spreadsheet of failure classes is already a cheap evaluation set.
Responsibility boundaries (worth repeating)
Local deployment reduces the intuitive risk of data leaving your network; it does not finish compliance for you. Pasting customer lists into prompts, fine-tuning on unauthorized repositories, or exposing an intranet API to the public internet are process failures. Write down which data classes may never enter prompts or train sets, who can merge adapters, and how to halt and audit incidents. Tutorials answer “how to run”; charters answer “whether we may run” and “who owns the blast radius.”
For hobbyists, the bar can be light: do not spread others’ private data, do not pursue illegal uses, and do not treat mystery weights as trusted software. For companies, boundaries must be auditable. In both cases, “open” never means “no obligations.”
How to read neighboring articles on this site
If the bottleneck is buying too large or too small a machine, start with budget builds and VRAM math. If knowledge goes stale, read RAG first. If tone and format are unstable, then read LoRA training. If the pain is organizational access control, read enterprise private deployment and the safety checklist. If the pain is licensing, read the open-weight and free-local boundary pieces. Reading by bottleneck beats reading by publish date. When you hit an internal link, open the relevant section instead of only bookmarking the title.
Extra verification and reading order
Read this article beside the VRAM, budget-build, and private-deployment guides: confirm workload and VRAM first, then tools and models, and only then training or enterprise governance. Keep key prompts, weight filenames, and eval notes in one place so a two-week retrospective beats a folder of unread bookmarks. If your environment has special compliance rules, follow legal and security counsel; this page is technical education.
The lasting value of local deployment is rarely a single successful demo. It is repeatability, rollback, and explainability. Whether you choose free tools or an enterprise path, keep a change log: which weight changed on which day, who approved an adapter merge, which eval failed. That log makes the next incident an order of magnitude cheaper. If you still do not know where to start, write three real tasks first, then return to the VRAM and build tables as a multiple-choice problem instead of buying hardware on impulse.
FAQ
What is the cost-effective way to train a local LLM?
Confirm RAG is insufficient, then LoRA/QLoRA; prove metrics on 7B/8B single-GPU before scaling.
What configuration do I need?
For learning LoRA, a 24GB-class GPU is comfortable; full fine-tuning or pretraining is not a consumer goal. See A87.
LoRA vs full fine-tuning?
LoRA trains fewer parameters, uses less VRAM, and manages multiple adapters better; full FT is for teams with clusters.
How much data do I need?
Style goals may work with hundreds to a few thousand clean rows; complex domains need more—but dirty 100k rows lose to clean 5k.
Can I train on Windows?
Yes, with more driver friction; many teams train on Linux/cloud and infer locally.
Can fine-tuning make a model unsafe?
Any fine-tune can shift refusals and bias. Regress safety and business policy on an eval set and follow the law; malicious uses are out of scope here.
Want feedback on your training plan?
Post in the community or email admin@microaione.com with base model, GPU, goal tasks, and whether the eval schema can be shared. Our Shenzhen team can help you decide RAG vs LoRA and label estimates clearly.
Sources: QLoRA paper on arXiv, Hugging Face PEFT docs, our local LoRA VRAM guide, AI training PC config.
