Private LLM Deployment for Business: Training, Steps, Costs and Sizing (With Compute Worked Examples)
“We want to train our own model in-house, and the data cannot leave the building.” It is one of the most common requests we get, and the answer is often a surprise: for private LLM training and deployment, most companies do not need to train from scratch. They need to pick an open-weight model, connect it to their own knowledge base, run a light fine-tune if necessary, and deploy it safely inside the network. This article splits “training” into four levels, uses formulas to show how much memory and compute each level takes, then gives deployment steps, budget tiers and a list of mistakes, so you can decide which level you belong to before spending money.
How we checked this: every memory and compute figure is a theoretical value derived from public formulas and paper conclusions, not a measurement of ours. Hardware reference prices come from our 2026-10-30 median table (real quotes govern). GPU rental prices come from RunPod’s pricing page (updated 2026-10-28). The process section draws on CAICT’s all-in-one machine report. The effective compute and token counts in the training examples are assumptions and are labelled as such. We have not delivered training projects for companies and have no customer cases to publish.
Key points (updated 2026-11-04)
- Customisation has four levels: prompts and rules, retrieval-augmented generation (RAG), fine-tuning (LoRA/QLoRA), and pretraining from scratch. Validate at the lowest level first.
For one 7B model: Q4 inference is about 4.3 GB; full training at 16 bytes per parameter is about 112 GB before activations; the QLoRA paper reports fine-tuning a 65B model on a single 48 GB GPU.
Pretraining a 7B model from scratch, using the common 6 x parameters x tokens approximation, 20 tokens per parameter and an assumed 160 TFLOPS effective compute, needs about 10,000 GPU-hours, roughly 20,000 USD in rental alone at RunPod’s H100 PCIe community price of 1.99 USD/hour, before data, reruns and staff.
- On the deployment side, permissions, audit and evaluation matter more than the model; see our enterprise private deployment guide.
Private LLM training and deployment: separate the four levels first
Many discussions treat “training” as one thing. In practice it is four very different investments:
| Level | What you do | Data needed | Compute needed | Good for |
|---|---|---|---|---|
| 1. Prompts and rules | Write system prompts, output formats, banned terms | A few dozen examples | Inference only | Consistent tone and format |
| 2. Retrieval-augmented generation (RAG) | Chunk and embed documents, retrieve at question time | Your documents, hundreds to hundreds of thousands of pages | Inference plus embedding | Knowledge-base Q&A and policy lookup (see local knowledge base) |
| 3. Fine-tuning (LoRA/QLoRA) | Adjust model behaviour with thousands to tens of thousands of curated Q&A pairs | Carefully prepared instruction data | One to a few GPUs | Fixed style, industry phrasing, structured output |
| 4. Pretraining from scratch | Train a base model from random initialisation | Trillions of tokens | Hundreds to thousands of GPUs | The business of base-model vendors |
In practice, most “private knowledge” problems are solved at level 2: when knowledge changes you re-index rather than retrain, and fine-tuning is better suited to changing how the model speaks than to making it memorise new facts. One finding in the QLoRA paper is worth remembering: data quality matters far more than quantity. A 9,000-example OASST1 subset beat a 450,000-example subsampled FLAN v2 on chatbot performance.
Memory math: inference, fine-tuning and full training
Start with the basic memory requirements. For inference the weights dominate: Q4_K_M is about 0.6 GB per billion parameters and FP16 is 2 GB. For full training with mixed-precision Adam, the ZeRO paper gives 16 bytes per parameter for model states (2 for parameters, 2 for gradients, 12 for optimizer states), before counting activations.
| Model size | Q4 inference (about 0.61 GB per billion) | FP16 inference | Full training (16 bytes/parameter, no activations) |
|---|---|---|---|
| 7B | about 4.3 GB | 14 GB | about 112 GB |
| 14B | about 8.5 GB | 28 GB | about 224 GB |
| 32B | about 19.5 GB | 64 GB | about 512 GB |
This is why full training on one workstation is rarely realistic, and why QLoRA changed the picture. The QLoRA paper says it cuts the memory for fine-tuning a 65B model from more than 780 GB with regular 16-bit fine-tuning to under 48 GB, so it fits on a single 48 GB GPU. For small models (7B to 14B), QLoRA fine-tuning on a 24 GB card is a realistic direction; see local LoRA fine-tuning VRAM and training PC configurations for specifics. Note that the paper’s conclusions come from academic benchmarks; you must evaluate results on your own business data.
How much compute does training from scratch need: an order-of-magnitude example
If someone offers to “train a dedicated model for you from scratch,” two common approximations let you estimate the cost:
- Training compute is roughly 6 x parameters x training tokens (a common approximation).
- A frequently cited rule of thumb from the Chinchilla study is about 20 training tokens per parameter, used here as an order-of-magnitude assumption.
Assume effective compute of 160 TFLOPS per GPU (an assumption) and rental at RunPod’s H100 PCIe community-cloud price of 1.99 USD/hour (pricing page updated 2026-10-28):
| Model size | Training tokens (assumed) | Compute (FLOPs) | GPU-hours | Rental only (USD) |
|---|---|---|---|---|
| 1B | 20 billion | about 1.2 x 10^20 | about 208 | about 415 |
| 7B | 140 billion | about 5.9 x 10^21 | about 10,208 | about 20,315 |
| 70B | 1.4 trillion | about 5.9 x 10^23 | about 1,020,833 | about 2.03 million |
The point of the table is not a quote but the order of magnitude: from 1B to 70B the cost differs by thousands of times. And it covers one ideal run only, without data cleaning, failed experiments, evaluation, storage or staff. Open-weight models of similar sizes are already available for free, so unless you have a distinctive data advantage and a long-term budget, pretraining from scratch is rarely the right choice for a business.
Six steps to roll out a private model
- Define the problem and success criteria. Write down three concrete business questions and acceptance rules (for example, contract-clause answers are sampled and scored by the business owner), not “use an LLM.”
- Pick a base model and size. Choose among open-weight models by licence, language ability and size; work backwards from the model size chooser and the VRAM guide.
- Run a RAG pilot first. One workstation, one department, four to six weeks. Judge whether answers cite sources and whether they leak across permissions.
- Decide whether fine-tuning is needed. Only when RAG cannot fix style, format or industry phrasing. Prepare high-quality instruction data and try QLoRA on a small model first.
- Security and compliance review. Permission filtering, audit logs, prompt-injection defences (see the enterprise content-safety checklist and prompt injection vs jailbreak); compliance questions belong to your legal team.
- Scale and operate. Concurrency and expansion, version control, backup and evaluation regression. CAICT’s selection flow also stresses leaving headroom for peak concurrency and running a PoC on your own data; see the six selection steps.
A practical example: a 50-person company
Imagine a 50-person engineering firm that wants an assistant for internal procedures and past project reports. A sensible path is: index the procedure manuals and reports with RAG on one 128 GB unified-memory machine, let ten volunteers use it for a month, and log every answer that was wrong or lacked a source. Only if the log shows a pattern that retrieval cannot fix, such as reports that must follow a fixed template, would the next step be a small QLoRA fine-tune on a few thousand reviewed examples. This example is hypothetical and meant to show the order of decisions, not an outcome we have observed.
Budget tiers: what each level looks like
| Tier | Typical setup | Reference price scale (real quotes govern) | Who maintains it |
|---|---|---|---|
| RAG pilot | 128 GB unified-memory mini PC or a 24 to 32 GB VRAM workstation | Hardware roughly 25,000 to 66,000 CNY (our reference prices: AI Max+ 395 mini PC about 24,566 CNY, RTX 5090 workstation about 65,737 CNY) | Part-time operations is enough |
| Fine-tuning experiments | One 24 to 48 GB GPU, or rent GPUs by the hour | Rental example: RTX 4090 community cloud about 0.34 USD/hour (RunPod, 2026-10-28) | Someone who understands training |
| Department-level service | Multi-GPU server plus gateway and permissions | Quoted per design; no single number | Dedicated or outsourced operations |
| Pretraining from scratch | Clusters of hundreds to thousands of accelerators | See the table above, millions of dollars and up | A professional training team |
For a detailed rent-or-buy calculation, see rent a GPU or buy one.
Common mistakes and hidden costs
- Treating “training” as “making the model remember our documents.” Memory of facts comes from retrieval, not fine-tuning. Fine-tuning can teach phrasing while still inventing facts.
- Underestimating data preparation. Curating good instruction data often takes longer than the training run, and the QLoRA paper stresses quality over quantity.
- Starting without an evaluation set. Without business-scored samples you cannot tell whether things got better or worse.
- Ignoring licences. Open weights do not mean free commercial use; read the licence on the model card and have legal confirm compliance.
- Counting training but not serving. Permissions, logs, concurrency, monitoring and updates are the long-term costs.
- Forgetting iteration cost. When the base model changes generation, fine-tuning may need redoing, while RAG usually only needs a model swap.
What we checked and what we did not do
The QLoRA memory claim comes from the paper’s abstract and text. The 16 bytes per parameter comes from the ZeRO paper’s analysis of mixed-precision Adam. The compute formula and tokens-per-parameter figure are common approximations used as order-of-magnitude assumptions. GPU rental prices come from RunPod’s pricing page, and hardware reference prices from our 2026-10-30 table. The GPU-hours are formula results, not training logs of ours, and 160 TFLOPS effective compute is an assumption. We have not run enterprise training projects and publish no customer cases.
FAQ
Does a company have to train its own LLM to deploy privately?
No. Most scenarios are solved by an open-weight model plus retrieval. Consider fine-tuning only when style or format needs customising.
How much VRAM does local fine-tuning need?
At 16 bytes per parameter, full training of a 7B model is about 112 GB. QLoRA quantises the base model to 4 bits, and the paper reports fine-tuning a 65B model on a 48 GB GPU. Your sequence length and batch size change the result.
How much does it cost to train a 7B model from scratch?
Under this article’s assumptions, about 10,000 GPU-hours, roughly 20,000 USD of rental, excluding data and staff. It is an order-of-magnitude estimate, not a quote.
Fine-tuning or RAG for a company knowledge base?
RAG first, because updates are easy and answers can cite sources. Fine-tuning suits changes in style and format.
What security measures does a private deployment need?
At minimum permission filtering, audit logs, rate limits and prompt-injection defences, with your legal team judging compliance requirements.
Can I use cloud GPUs if data must not leave the network?
Putting data in the cloud means it has left your network. If compliance is strict, train in your own or a dedicated environment, and have your compliance team decide.
Not sure which level you belong to?
Join the community or email admin@microaione.com with your business questions, data volume, sensitivity and budget. Our Shenzhen team can help judge whether you need fine-tuning and how to configure pilot hardware. We pilot first and scale later, and make no performance promises we have not verified.
References: QLoRA: Efficient Finetuning of Quantized LLMs (arXiv), ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv), Chinchilla: Training Compute-Optimal Large Language Models (arXiv), RunPod pricing, CAICT: LLM all-in-one machine report (2025, Chinese).
