What Hardware Does a 14T-Parameter MoE Need? numbat’s GDN+MLA, NVFP4 and MTP Explained, Plus How to Spot Fake ‘Leaked’ Models and Their Legal Risks
In our previous piece we fact-checked the burner22/numbat repository on Hugging Face (Anthropic Mythos leak fact-check) and concluded that several pieces of evidence point to a forgery. But the repo has real teaching value: its configuration describes a complete 14T-parameter MoE, which lays out what a trillion-scale model looks like and what hardware it needs. This article is not about whether it is real. It uses numbat to answer three questions: what the parts of a 14T-parameter MoE are (hybrid GDN+MLA attention, 1,224-expert routing, MTP, NVFP4); how much hardware it takes just to hold it, and why no personal computer can run it; and how to spot a fake, avoid malicious code and understand the legal exposure the next time a “leaked model” appears. We did not download any weights or run any code. The numbers come from the repo’s config.json, Hugging Face metadata, vendors’ public specs and our own arithmetic, each labelled as such.
Key takeaways (updated 2026-11-02)
- By numbat’s config the model has about 13.7T total parameters and roughly 0.6T active per token (our rough estimate), about 4.4% of the total, in the same sparsity range as DeepSeek-V3 (37B of 671B) and Kimi K2 (32B of about 1.04T).
At about 4.5 bits per weight in NVFP4, 7.72 TB is only the cost of holding the weights: 55 H200s, 42 GPUs’ worth of GB200 NVL72, or 322 RTX 4090s.
Even streaming experts from NVMe, our lower-bound estimate is about 337 GB read per token, roughly 24 seconds per token on a single 14 GB/s drive. A smaller model such as DeepSeek V4.1 Flash can reach a few tokens per second because it activates far fewer experts.
To spot a fake leaked model, check provenance, whether sizes are consistent (necessary but not sufficient), weight statistics, self-contradictions, whether it forces trust_remote_code, and whether files are safetensors or pickle.
Risk comes in two layers: technically, pickle and remote code can execute arbitrary code on load; legally, a real leak can involve trade-secret and copyright issues. This is general information, not legal advice.
How Big Is a 14T-Parameter MoE? A Comparison With Known Models
A mixture-of-experts (MoE) model splits its feed-forward layers into many “experts” and routes each token to only a few, so total parameters are huge but per-token compute resembles a much smaller model. If the basics are new to you, start with running MoE models locally. Below, numbat’s config numbers sit next to public models (the numbat row is the repo’s claim, not proof it is a real model):
| Model | Total parameters | Active per token | Routed experts | Experts used per token | Active share |
|---|---|---|---|---|---|
| DeepSeek-V3 | 671B | 37B | 256 + 1 shared | 8 | about 5.5% |
| Kimi K2 | about 1.04T | 32B | 384 + 1 shared | 8 | about 3.1% |
| Qwen3-Next-80B-A3B | 80B | about 3B | 512 + 1 shared | 10 | about 3.75% |
| DeepSeek V4.1 Flash (our earlier articles) | about 763B | not verified by us | 384 per layer | 6 | not listed |
| numbat (repo claim) | about 13.7T | about 0.6T (our rough estimate from the config) | 1,224 + 1 shared | 48 | about 4.4% |
The DeepSeek-V3 and Kimi K2 figures come from their technical reports (DeepSeek-V3, Kimi K2); Qwen3-Next comes from an NVIDIA technical blog. numbat’s sparsity is not unusual; its active share is within the known range. What is staggering is the absolute scale: its per-token active parameters (about 0.6T) are close to DeepSeek-V3’s entire parameter count (671B). Compare our math in DeepSeek V4.1 Flash local deployment: that model’s BF16 weights of about 1.5 TB already need a multi-GPU server, and numbat has about 18 times as many parameters.
Architecture Breakdown: GDN+MLA, MoE Routing, MTP and NVFP4
The table maps config.json fields to public precedents. “Public precedent” means each technique has appeared in a published paper or open model, so a plausible-looking config cannot prove where the model came from.
| Component | numbat’s configured value | Public precedent | What it does |
|---|---|---|---|
| Hybrid attention | 86 of 114 layers GDN, 28 MLA (layers 3, 7, 11 and so on, one in four) | Gated DeltaNet paper; Qwen3-Next uses a 3:1 mix of GDN and full attention | Linear attention cuts long-context cost; a few full-attention layers protect accuracy |
| MLA | 96 heads, KV compression rank 512, RoPE dimension 64 | Multi-head latent attention from DeepSeek-V2 | Compresses the KV cache into a low-dimensional latent |
| MoE routing | 1,224 routed experts + 1 shared, pick 48; 72 groups, 8 groups per token; sigmoid scoring; correction bias | DeepSeek-V3 group-limited routing and auxiliary-loss-free balancing | Limits communication and balances expert load |
| MTP | Depth 1, training loss weight 0.1, used as a self-speculative drafter at inference | DeepSeek-V3 multi-token prediction | Drafts one extra token to speed decoding |
| Quantization | NVFP4: 4-bit E2M1 values, one FP8 scale per 16, plus an FP32 global scale | NVIDIA Blackwell’s NVFP4 format | Shrinks weights to about 4.5 bits |
GDN and MLA: Why Long Context Gets Cheaper
Gated DeltaNet (paper) is a gated form of linear attention: instead of a cache that grows with context, it keeps a fixed-size state. numbat’s GDN layers are configured with 48 heads, key dimension 128, value dimension 256 and an FP32 state. By our arithmetic each layer’s state is about 6.3 MB, 86 layers about 541 MB, and it does not grow with context. The MLA layers compress each token’s keys and values to 512 + 64 numbers; across 28 layers that is 16,128 numbers, about 32 KB per token in BF16, or about 4.2 GB for a 131,072-token context. That is per sequence, so long context is far cheaper than with classic attention. These figures are our arithmetic from the config under the assumptions of a BF16 cache and FP32 state, and they leave out convolution state and the MTP layer. For how context and caches consume memory, see context length and KV cache VRAM.
MoE Routing, MTP and NVFP4
The routing details are almost a scaled-up DeepSeek-V3 recipe: group-limited routing (1,224 experts in 72 groups of 17, choosing 48 from only 8 groups), sigmoid scoring, a correction bias instead of an auxiliary loss to balance load, and a config entry that freezes the correction bias at 98% of training tokens. In DeepSeek-V3, MTP is both a training objective and a way to do speculative decoding; numbat’s config has a single MTP layer of depth 1. NVFP4 is NVIDIA’s 4-bit floating-point format for Blackwell (NVIDIA’s introduction): 4 bits per weight with one 8-bit scale shared by 16 weights, so about 4.5 bits in practice. The quantization config also excludes embeddings, the output head, router gates, normalization layers, the first 2 layers, the last 5 dense layers and the MTP layer from NVFP4 (they stay in higher precision), which is common practice.
The Hardware Math: How Much Equipment Does 7.72 TB Take?
Weight size is the first wall. 14T parameters take about 28 TB in BF16, about 14 TB in FP8 and, in NVFP4, the repo’s actual 7.72 TB. This is the number of devices needed merely to hold the weights, excluding KV cache, activations and runtime overhead:
| Device | Memory per device | Count to hold 7.72 TB | Notes |
|---|---|---|---|
| RTX 4090 | 24 GB | 322 | Consumer card, no fast interconnect |
| RTX 5090 | 32 GB | 242 | Same |
| H100 | 80 GB | 97 | About 12 eight-GPU nodes |
| AWS Trainium2 | 96 GiB | 75 | AWS documentation |
| H200 | 141 GB | 55 | About 7 eight-GPU nodes |
| GB200 NVL72 | 186 GB per GPU | 42 | A rack has 72 GPUs and about 13.4 TB of HBM3e |
The data comes from NVIDIA’s public GB200 NVL72 specification and the AWS Neuron documentation; the counts are our arithmetic. The realistic unit for actually running this model is a whole GB200 NVL72 rack: about 13.4 TB of HBM3e holds 7.72 TB of weights and leaves roughly 5.7 TB for caches and activations. That is not a scale individuals or small teams can touch.
Could you stream experts from an SSD as with small models? In our analysis of running DeepSeek V4.1 Flash on 12 GB of VRAM with SSD streaming we explained that this works because that model activates only 6 experts per token. numbat activates 48, and we estimate about 0.6T x 0.5625 bytes, roughly 337 GB, of weights per token. Under a lower-bound assumption of batch size 1, reading all active weights once per token with no caching or prefetch overlap:
| Storage tier | Assumed bandwidth | Read time per token (lower bound) |
|---|---|---|
| NVMe Gen4 SSD | about 7 GB/s | about 48 seconds |
| NVMe Gen5 SSD | about 14 GB/s | about 24 seconds |
| Dual-channel DDR5 (needs 7.7 TB of RAM) | about 90 GB/s | about 3.7 seconds |
| GB200 NVL72 rack aggregate HBM bandwidth | 576 TB/s (NVIDIA spec) | about 0.6 ms (theoretical) |
The first three bandwidths are typical values we chose, not measurements, and only illustrate orders of magnitude; the last is NVIDIA’s published aggregate, and real throughput depends on interconnect and batch size. The conclusion is direct: on any personal device, generating a sentence would take tens of minutes or more. For the trade-offs between RAM and VRAM, see RAM vs VRAM and CPU offload and the GPU VRAM buying guide.
Why It Cannot Run Locally
Besides size there are two more reasons. First, the runtime: numbat is not a built-in architecture in mainstream inference frameworks, and the repo asks you to load its own implementation with trust_remote_code=True. The author of discussion #11 says the reference implementation has no KV cache, loops over tokens and experts, dequantizes all weights on every call and assumes a single device, so it cannot serve real inference (that is the commenter’s statement; we did not run it). Second, the format: NVFP4’s fast path relies on native Blackwell support, and older consumer cards cannot reach the same speed. Together, even someone who really held the 7.72 TB would have no ready-made route to run it on personal hardware.
Spotting a Fake “Leaked” Model: Eight Checks
The next time you see “weights of a big lab leaked”, run through this list. The numbat column comes from our previous fact-check.
| Check | What to look at | Rule of thumb |
|---|---|---|
| Provenance | Who uploaded it, account history, any official statement | A brand-new account with “don’t ask where it came from” is a warning sign |
| Size and numbers | Whether parameters x bits match total size | Matching is necessary, not sufficient |
| Weight statistics | Whether normalization layers look identical at every depth, any outliers | A trained network does not show the same bell curve at every layer |
| Reproducibility | Whether tensors can be regenerated from names or simple rules | If they can, they were generated, not trained |
| Self-contradiction | Whether metadata and text disagree | “Mid-pretraining” at 93% of steps is a classic example |
| Code | Whether trust_remote_code is mandatory | If so, read the code first and work in isolation |
| File format | safetensors versus pickle, .bin or .pt | Pickle can execute code on load |
| Popularity | Downloads and likes | Not evidence; counts can be inflated or driven by curiosity |
Security Risks: Malicious Pickle, Remote Code and More
The file format decides what happens when you load a model.
| Mechanism | Risk | Mitigation |
|---|---|---|
| Pickle-family formats (.bin, .pt, .pkl) | Python’s pickle can execute arbitrary code on load; Hugging Face’s security documentation warns about exactly this | Use safetensors only; if you must load a .pt, use weights_only=True in recent PyTorch and do it in isolation |
| trust_remote_code=True | The repo’s Python files are imported and executed, equivalent to running a stranger’s script; see the Transformers documentation | Read the code in a browser first; pin a commit hash; use a VM with no network or credentials |
| “Install scripts” in a README | Asks you to run curl piped to bash or install unknown packages | Do not run them; read the source first |
| Fake “driver” or “accelerator” download links | Trojans, browser credential theft | Never download executables from anywhere but the vendor |
| Letting an AI agent fetch and run it | The agent executes the loader with your privileges | Never give an agent an unreviewed repo |
The safetensors format holds only data and a text header and cannot execute code, which is why it became the default. To be clear: we read numbat’s two Python files and saw no malicious calls, but “did not see” is not “safe”, we did not audit every file, and the repo can change at any time. A sensible workflow: step one, read the README and config in a browser without downloading anything; step two, use HTTP range requests to read only the safetensors header and a few tensors; step three, load only if necessary and only in a disconnected, credential-free, disposable VM; step four, never hand such a repo to automation or an AI agent. For privacy and data safety in local deployments, see local LLMs and privacy.
Legal and Copyright Risks
If a repo really were leaked weights from a closed model, downloading, keeping and redistributing could carry legal exposure. The table is a general overview, not legal advice:
| Region | Rules that may apply | Notes |
|---|---|---|
| United States | Defend Trade Secrets Act (18 U.S.C. § 1836 provides a civil action) | Acquiring or using a trade secret you know was improperly obtained can lead to liability |
| China | Anti-Unfair Competition Law Article 9; Criminal Law Article 219 | Trade-secret infringement can carry civil, administrative or even criminal liability |
| European Union | Trade Secrets Directive (2016/943) | Member states implement it in national law |
| Copyright | Whether weights are copyrightable is unsettled across jurisdictions | Accompanying code and documentation are clearly protected |
On precedent, BleepingComputer reported that Anthropic issued DMCA notices against mirrors of the leaked Claude Code source, which shows vendors do enforce. Even if a repo later proves fake, spreading “leak” links can still expose you to risk and can steer others to repos carrying malicious code. We do not provide download or magnet links, and we suggest you do not share any.
What We Did and Did Not Do
We read the repo’s config.json and custom code, estimated size, device counts and read times from public specifications and our own arithmetic, and consulted papers and vendor documentation. We did not download weights, run code or test any hardware. Figures marked “assumed” are order-of-magnitude estimates, not measurements.
FAQ
Does a 14T-parameter MoE really exist?
The largest open MoE we can cite is Kimi K2 at about 1T; numbat’s 14T has no reliable confirmation. We cannot say that scale does not exist inside a closed lab, only that we have no evidence.
What is NVFP4 and why is it so small?
It is NVIDIA’s 4-bit floating-point format for Blackwell: 4 bits per weight with one 8-bit scale per 16 weights, about 4.5 bits overall, roughly 28% of BF16’s size. The cost is the need for matching hardware and software support.
What does mixing GDN and MLA buy you?
GDN handles most layers with a fixed-size state, and MLA compresses keys and values into a low-dimensional latent; together they sharply cut long-context cache costs. Both techniques have public papers.
Could I run it on a PC with an SSD and RAM?
By our lower-bound estimate each token needs about 337 GB read, about 24 seconds per token on a 14 GB/s SSD before any other overhead, and you would first need to own 7.72 TB of data. In practice it is pointless.
What is the biggest risk of loading an unknown model?
Code execution: both pickle formats and trust_remote_code=True run untrusted code on load. Use safetensors only and work in an isolated environment.
Is downloading a “leaked” model illegal?
It depends on whether it is really improperly obtained trade-secret material and on where you live. This is general information, not legal advice; consult a lawyer if it matters. We provide no download links.
Talk With Us
Want to know what size of model your GPU and RAM can run, or want help deciding whether a new viral model is worth the trouble? Join the community or email admin@microaione.com with your setup and use case.
References: DeepSeek-V3 technical report, Kimi K2 technical report, Gated DeltaNet, NVIDIA NVFP4 introduction, GB200 NVL72, Hugging Face: pickle security. Data read on 2026-11-02, Beijing time.
Further reading: Fact-checking leaked and cracked model claims
