Llama Guard Local Deployment Tutorial: Add Guardrails to a Local LLM with Llama Guard, Qwen3Guard and ShieldGemma

Running a local model is easy, but the moment you open it to other people you need a guardrail that is independent of the model: moderate the input before it reaches the model, moderate the output before it goes back, block violations and keep a record. The small models built for this are called guard models. This is a hands-on Llama Guard local deployment tutorial: it compares Llama Guard, ShieldGemma, Qwen3Guard and gpt-oss-safeguard, runs a guard with Ollama, shows a code structure for input and output moderation, works out the VRAM budget and explains how to measure false positives. One point up front: a guard is a defensive tool, nothing here is about bypassing it, and its verdict is not legal “compliance”, only one technical layer. For the overall design see jailbreak defenses explained.

Key points (updated 2026-11-04)

• Easiest to start: Ollama’s llama-guard3:1b (about 1.6GB) and llama-guard3:8b (about 4.9GB) with 128K context; ShieldGemma comes as 2B (1.7GB), 9B (5.8GB) and 27B (17GB) with 8K context.

• Qwen3Guard (released September 2025) has 0.6B, 4B and 8B sizes in Gen and Stream variants, three severity labels (safe, controversial, unsafe) and 119 languages.

• gpt-oss-safeguard (released October 2025) comes in 20b and 120b, is Apache 2.0, and takes your own policy at inference time.

• On 24GB, a 27B main model plus a 1B guard totals about 20GB of weights and fits; a 32B main model plus an 8B guard is about 25GB and does not, so move the guard to CPU or use a smaller pair.

• A guard is not magic: measure false blocks and misses on your own data and re-test regularly.

Why a local model needs a separate guardrail

A local model has only the alignment training in its weights and none of the outer filtering a hosted product has. For personal use that is fine; but once you turn it into a team assistant, a public chatbot or part of a support flow you need a separate moderation layer, for three reasons:

1. Alignment training can fail, especially against new phrasings (see LLM jailbreak explained).
2. Policy should be under your control: what your business blocks and allows should be defined by you, not left entirely to the model.
3. You need a record: blocked requests should be logged for review and improvement.

Guard models are small and fast and can sit beside the main model. The wider “do you need to jailbreak it” discussion is in Does Qwen need jailbreaking for local deployment?

Choosing among four families of guard models

A few notes. First, we have not tested the accuracy of these models; the table comes from official release pages and Ollama pages, and some sizes are estimated at 0.61 GB per billion parameters. Use the model cards and your own evaluation when choosing. Second, Qwen3Guard’s Stream variant needs custom code allowed at load time (trust_remote_code), which means trusting the repository author and pinning versions and checksums; we covered that kind of risk in how to spot suspicious model repositories. Third, the benefit of a three-level label is that “controversial” content can go to human review instead of a blunt allow or block.

Llama Guard local deployment with Ollama in practice

Install Ollama following the Ollama install guide, then pull a guard:

ollama pull llama-guard3:1b     # about 1.6GB, fine for CPU or small VRAM
ollama pull llama-guard3:8b     # about 4.9GB, finer judgments

You hand Llama Guard 3 a conversation and it replies with safe or unsafe on the first line and, if unsafe, category codes on the second. To moderate only the user input, put just the user message in the conversation; to moderate the model’s answer, put the user message plus the answer. Here is a minimal example calling the local Ollama API (moderation only, no attack content):

import requests

OLLAMA = "http://127.0.0.1:11434/api/chat"

def guard(model, messages):
    r = requests.post(OLLAMA, json={
        "model": model, "messages": messages,
        "stream": False, "options": {"temperature": 0}}, timeout=60)
    r.raise_for_status()
    return r.json()["message"]["content"].strip()

def parse_llama_guard(text):
    lines = [l.strip() for l in text.splitlines() if l.strip()]
    safe = bool(lines) and lines[0].lower() == "safe"
    cats = lines[1].split(",") if (not safe and len(lines) > 1) else []
    return safe, [c.strip() for c in cats]

Chain them into a sandwich flow (Figure 1):

def chat_with_guard(user_text, main_model, guard_model="llama-guard3:1b"):
    # 1. input moderation
    safe, cats = parse_llama_guard(
        guard(guard_model, [{"role": "user", "content": user_text}]))
    if not safe:
        log_blocked("input", user_text, cats)      # keep a record, mind privacy
        return "This request cannot be processed."
    # 2. main model
    answer = guard(main_model, [{"role": "user", "content": user_text}])
    # 3. output moderation
    safe, cats = parse_llama_guard(guard(guard_model, [
        {"role": "user", "content": user_text},
        {"role": "assistant", "content": answer}]))
    if not safe:
        log_blocked("output", answer, cats)
        return "The answer was blocked by the safety policy."
    return answer

Three engineering details:

• Fail closed: if the guard times out or returns something unparseable, block rather than allow.
• Temperature 0: moderation should be as deterministic as possible.
• Redact logs and limit retention: logging blocked content carries privacy risk; see local LLMs and privacy.

If you expose the service through an API, put this logic in the gateway; the local API server guide is a good starting point.

Handling Qwen3Guard and ShieldGemma output

Qwen3Guard’s Gen variant returns a few lines of text, roughly “Safety: level” and “Categories: category”, and for answers possibly a “Refusal” field; the exact format is on the model card. Parse the level from the first line:

import re
def parse_qwen3guard(text):
    m = re.search(r"Safety:\s*(Safe|Controversial|Unsafe)", text, re.I)
    level = m.group(1).lower() if m else "unknown"
    return level      # treat unknown as blocked

Send controversial to a human or a degraded path rather than allowing it outright. ShieldGemma needs your policy text in the prompt and asks whether the content violates it, in the format of the official examples. gpt-oss-safeguard works similarly with the policy as input, suits complex or fast-changing policies, but is larger and slower.

VRAM and latency budget

A guard usually sits on the same machine as the main model, so add their weights together and leave room for cache and runtime overhead.

Worked example: a 27B Q4 at about 18.5GB plus Llama Guard 3 1B at 1.6GB gives 20.1GB of weights, under the 21GB budget (see what 24GB of VRAM can run for where the budget comes from); but that excludes KV cache, so context length has to stay modest. If VRAM is short, run the guard on CPU: guards of 1B to 4B are usually workable there at the cost of added latency.

On latency, a guard is mostly prefill: read the input, emit a few tokens. Prefill compute is about 2 × parameters × input tokens, so an 8B guard on 500 tokens is about 2 × 8×10^9 × 500 = 8×10^12 floating-point operations; if we assume about 30 TFLOPS effective throughput that is roughly 0.27 seconds (an assumption, measure it). Moderating input and output costs that twice. A 1B guard cuts it to about an eighth.

Evaluate false blocks and misses

After deployment you must evaluate; do not trust the accuracy a model claims about itself:

1. Prepare two sets: requests your business normally sees (to measure false blocks) and public benchmark samples of the categories your policy blocks (to measure misses, using public benchmarks such as HarmBench in an isolated environment);
2. Record false-block and miss rates, split by category;
3. Adjust thresholds or change size and compare;
4. Re-run after every guard or main-model update.

Note that guards can behave very differently on Chinese, on industry jargon and on long text compared with English benchmarks; a guard that looks great in English tests may over-block in your business. For red-teaming methods see our local model red-teaming primer.

Choosing by scenario

Three rules help. First, start small: go live with 1B to 4B and decide about upgrading after you have false-block data. Second, evaluate input and output separately, because their error patterns differ. Third, keep a human channel: requests the guard gets wrong need somewhere to be appealed and reviewed, or false blocks hit the user experience directly. After deployment, record versions and evaluation results every time you change the guard or main model, so you can trace problems later.

The limits of a guard

• It does not replace permission control: for apps with tools a guard cannot stop all injection; combine it with least privilege.
• It does not guarantee compliance: a classification is a technical judgment, not a legal conclusion; consult professionals on compliance.
• It is not free: it adds latency, uses VRAM and can block normal requests.
• It can be attacked too: a guard is also a model and can be bypassed, which is why you layer defenses and keep testing; see jailbreak defenses explained.

FAQ

How much VRAM does Llama Guard local deployment need at minimum?

llama-guard3:1b is about 1.6GB and can run on small VRAM or even CPU; the 8B version is about 4.9GB.

Is Qwen3Guard or Llama Guard better for Chinese?

Qwen3Guard is officially described as covering 119 languages with Chinese a focus; we have not compared them. Evaluate both on your own Chinese data.

Can a guard stop every jailbreak?

No. It is one layer, to be combined with alignment training, permission control, monitoring and red-teaming.

Will a guard slow things down?

Yes. Input and output are each moderated once, and the cost depends on guard size and text length; 1B to 4B guards usually suit online services better than 8B and up.

What do I do with a “controversial” label?

Qwen3Guard’s three levels let you send “controversial” to human review or a degraded path instead of allowing or blocking outright.

Can I customize the policy?

ShieldGemma and gpt-oss-safeguard accept policy text in the input; Llama Guard 3 mainly uses the official categories, so check the model card on customization.

Talk to us

If you want to add a guardrail to your own local model, send your deployment, budget and language scenario to the community or email admin@microaione.com and we can help pick a size and estimate VRAM. More related articles are in the LLM jailbreak and safety hub.

References: Llama Guard paper (arXiv 2312.06674), Ollama Llama Guard 3 page, Ollama ShieldGemma page, Qwen3Guard official repository, OpenAI gpt-oss-safeguard announcement. Data read on 2026-11-04, Beijing time.