How to Fact-Check Leaked Model Claims and “Cracked” Releases: A 7-Step Method (numbat and GSQ-RCO Examples)

Every few weeks the community sees a message like this: “the weights of a closed model have leaked,” or “a model was cracked, refusals are gone and it needs only 12 GB of VRAM.” Such messages spread fast and are hard to judge at a glance. This article turns the method we used on two such stories, numbat and GSQ-RCO, into a reusable 7-step method for checking leaked model claims: source, metadata, arithmetic consistency, contradictions in the claim itself, number definitions, independent replication and safe handling. It covers method and evidence only. It gives no download links and does not teach how to bypass a model’s safety limits. The case data come from our two earlier fact-check articles, read on 2026-11-01 and 2026-11-02; this article was updated on 2026-11-04.

Key points (updated 2026-11-04)

• The 7 steps: find the primary source, check metadata, test arithmetic consistency, look for internal contradictions, pin down definitions and terms, look for independent replication and official statements, then handle safely and grade the conclusion.

• “The numbers add up” is not proof: numbat’s 14T parameters and 7.72 TB are fully self-consistent, which only shows its author did careful arithmetic.

• “Refusals cleared” depends on sample size: 0 refusals observed on 81 prompts still leaves a 95% upper bound of about 3.7% on the true refusal rate (rule of three, our conversion).

• Check terminology on “uncensoring” claims too: GSQ-RCO is the name of a quantization method, not an uncensoring method.

• Grade conclusions: confirmed, leaning false, or cannot tell; and state what we did and did not do.

The 7 steps of fact-checking leaked model claims

First the framework: each step answers one question and yields a basis that others can check.

Step 1: find the primary source, do not stop at reposts

For any “leak” or “crack” message, ask two questions first: who published it, and what has the vendor said. In the numbat case the repository called itself a checkpoint of Anthropic’s next model, but our earlier check found no source tying the repository to Anthropic and no Anthropic statement about it; the only known official positions were that the earlier Claude Code source leak did not involve model weights and that Mythos Preview is not planned for general availability. The README’s “why you should believe this” list, such as the codename appearing in leaked source and an architecture that fits expectations, is not provenance evidence: a codename is public information and anyone who read the news could use it to name a repository. Details are in the Mythos weight leak fact-check.

Step 2: check metadata and let the platform tell you the facts

Many platforms’ public interfaces give creation time, commit history and storage size without downloading any large file. Hugging Face repository pages and its API, for instance, show the file list, total size and commits. Look at: whether the creation time fits when the story appeared; whether uploads were batched or in one go; whether storage size matches the claimed size; whether the uploader’s account is new or has a history. Metadata cannot prove truth, but it can rule out the case where the claim and the files are not the same thing at all.

Step 3: test arithmetic consistency, but never treat consistency as evidence

Worked example: numbat’s size. The repository claimed about 14T parameters and 7.72 TB. From our earlier check: NVFP4 packs two 4-bit values into one byte, the Hugging Face page counts bytes and shows 6.85T, and converted that is about 13.67T 4-bit weights; NVFP4 shares one 8-bit scale across every 16 weights, so each weight takes about 4.5 bits, or 0.5625 bytes. Size ≈ 13.67×10^12 × 0.5625 + the BF16 part (about 16.85 billion values × 2 bytes ≈ 34 GB) ≈ 7.69 TB + 0.03 TB ≈ 7.72 TB, matching the 7.72064 TB Hugging Face records.

That shows two things. First, “Hugging Face shows 6.85T, which does not match 14T, so it is fake” is a misreading. Second, all the numbers adding up does not show the model is real, because a forger can make the numbers consistent too. The value of arithmetic checks is to rule out wrong criticism and sloppy fakes, not to prove authenticity.

Step 4: look for contradictions inside the claim

Contradictions visible without any replication are the most persuasive. numbat’s README says the checkpoint came from “mid-training,” yet its metadata puts it at step 1.83 million of a run of about 1.96 million, roughly 93%, with the learning rate in a decay phase, which looks more like late training; it claims NVFP4 evaluation on Trainium2, while in the AWS Neuron documentation we checked, Trainium2’s listed data types do not include FP4. One contradiction may be wording; several stacked together lower credibility. When reading a README, set the config, logs and evaluation notes beside it instead of reading only the promotional top.

Step 5: pin down definitions and terms

The most common problem with “uncensored” claims is not forgery but unclear definitions. Take GSQ-RCO DeepSeek V4.1 Flash (details in the GSQ-RCO fact-check):

• Term: GSQ-RCO is a quantization method from ISTA-DASLab in Austria, not an uncensoring method; “abliteration 2.0” in the reposts was also a wording error.
• Baseline: a KL of 0.02 is the shift of the edited model relative to the unedited quantized build, not to the original model, and it is a top-64 lower bound.
• Sample size: the author’s own English harmful-prompt test set has only 81 prompts (per our earlier check, separate from test sets such as XSTest), and “refusals cleared” means a drop from 85.2%–88.9% to 0.0%.

Worked example: sample size. With 0 refusals observed on 81 prompts, how low can we say the true refusal rate is? Statistics’ “rule of three” gives an approximation: with 0 events in n trials, the 95% upper confidence bound is about 3 ÷ n. For n = 81 that is about 3.7%; for n = 300 about 1.0%; for n = 1000 about 0.3%. So “refusals cleared” can only be read as “no refusal was observed in these 81 prompts,” not “it never refuses.” This is our conversion, not the original authors’ conclusion. For more on evaluation see how refusal rates are measured and how to read LLM benchmarks.

Step 6: look for independent replication and official statements

The strongest evidence is that others can reproduce it independently. In the numbat case, community members reported in the discussions that 43 full tensors could be regenerated bit for bit from “a random generator seeded by the tensor name,” with predictions recorded before reading; another member said the large NVFP4 matrices were the same. Our practice is to write these as “so-and-so reports,” because we did not reproduce them ourselves. Readers can follow the published method and read a few tiny byte ranges over HTTP range requests rather than download the whole repository. Also search for an official reply; not finding one does not mean none exists, so write only “we did not find one.”

Step 7: handle it safely and grade the conclusion

The aim of checking is not to “download it and verify by hand.” For an unknown model repository, do not download the large files, and above all do not load it with trust_remote_code=True: that executes Python code provided by the repository when it loads, which means trusting a stranger. We read numbat’s two Python files and saw only references to torch, transformers and the standard library, with no network, subprocess or file-writing calls, but that was a spot check, not an audit. Separately, if “leaked” weights are real, they may raise trade-secret and copyright issues; see legal and compliance risks of uncensored models.

Finally, grade conclusions in three tiers: confirmed (primary source or independent replication), leaning false (several pieces of evidence point the same way but nothing is decisive) and cannot tell (insufficient evidence). We judged numbat as “leaning toward forgery” and stated three things we still cannot rule out: we did not read all the data, “forgery” is only a judgment based on public evidence, and even a forgery does not mean a real leak could never happen.

The two cases side by side

A template you can use right away

For the next “leak” or “crack” message, take notes in this format:

1. The message text, source and read date (with time zone).
2. What is the primary source? Any official statement?
3. Platform metadata: creation, commits, size, uploader.
4. Hand calculation: parameters, bit width, size, VRAM.
5. List of internal contradictions.
6. Are the metric’s baseline, sample size and method name accurate?
7. Independent replication and official reply; mark “we reproduced” or “so-and-so reports.”
8. Safe handling; conclusion grade; “what we did and did not do.”

How we checked this, and what we did not do

The method comes from our earlier checks of numbat and GSQ-RCO; the case data are taken from those two articles and attributed as in the originals. The rule of three is a general statistical approximation that we used for the conversion. This time we did not re-download any weights, not run any repository code and did not contact any official body; the case conclusions remain the judgments made then, and we will update them if new evidence appears.

FAQ

What should I look at first when fact-checking leaked model claims?

The primary source: who published it and whether the vendor said anything. Many reposts are not a source.

Does “the numbers add up” show it is real?

No. Consistency only shows careful arithmetic by the maker, and a fake can achieve it too. Its use is to rule out wrong criticism and sloppy fakes.

Can I trust “refusals cleared”?

It depends on sample size and definitions. With 0 refusals observed on 81 prompts, the 95% upper bound on the true refusal rate is about 3.7%, so read it only as “not observed in this set.”

Can I download an unknown model to try?

We advise against it, and never use trust_remote_code=True, which executes the repository’s code. If metadata and small reads can settle the question, do not download.

I am not technical; how do I judge a message?

Look for an official source, independent replication and self-contradictions. Without all three, treat it as unconfirmed.

How should I word the conclusion?

Use three tiers (confirmed, leaning false, cannot tell), state what we did and did not do, and give the read date.

Want us to check a claim?

Join the community or email admin@microaione.com with the link and your question, and we will check it with this method.

More in this series: A short history of LLM jailbreaks; Jailbreak, “po jia” and cracking compared. All articles are collected in the LLM Jailbreak and Safety hub.

References: Hugging Face Hub security docs, Transformers: custom code and trust_remote_code, Rule of three (Wikipedia), Anthropic: Project Glasswing, and our earlier numbat and GSQ-RCO fact-check articles. Read on 2026-11-04.