A public compendium · updated daily
AI model distillation: how frontier intelligence is compressed, priced, and contested.
Distillation moves capability from a large teacher model into a small, cheap student. It is the quiet engine behind most models people actually pay for, the subject of an open dispute between the largest labs, and a live regulatory question in three jurisdictions. This compendium tracks all of it.
Key figures · one per perspective
Citations: Hinton, Vinyals & Dean (2015)
25,899 citations
Semantic Scholar citation count for arXiv:1503.02531, retrieved 2026-09-04
Nvidia single-day market-cap loss
589 USD billions
27 Jan 2025, after DeepSeek-R1 and its distilled students shipped. Largest one-day loss in US market history.
Claude exchanges in largest disclosed campaign
28,800,000 exchanges
Anthropic's June 10, 2026 letter to Senate Banking alleges Alibaba/Qwen-affiliated operators ran 28.8M+ exchanges between Apr 22 and Jun 5, 2026
Fraudulent accounts in Anthropic's Feb 2026 report
24,000 accounts
DeepSeek, Moonshot AI and MiniMax combined; MiniMax alone drove 13M+ exchanges
DeepSeek-R1-Distill family, all-time HF downloads
97,824,225 downloads
Sum of the six official R1-Distill repos (1.5B, 7B, 8B, 14B, 32B, 70B) via the Hugging Face API on 2026-09-03
Enterprise inference price index
1.17 USD / M tokens
Silicon Data index cited by Jefferies; hit a 2026 low of $1.16–$1.18 on 6–8 Aug 2026, down from $2.04 on 31 May and $1.45 in late July.
Methods catalogued here
27 methods
Counted in extras.methods; families extend the response / feature / relation trichotomy of Gou et al. (arXiv 2006.05525) with six additional families specific to generative and compression-composed KD; eleven families in total.
Dated events in this timeline
130 events
Every event carries its own source URL; spans 2006-08-20 to 2026-09-03.
The perspectives · 8 sections
Academic
Knowledge distillation research: from dark knowledge to on-policy reasoning transfer
Knowledge distillation began as a model-compression trick — Buciluă, Caruana and Niculescu-Mizil compressed an ensemble into a single net in 2006, and Hinton, Vinyals and Dean gave it its modern soft-target formulation in 2015 (now ~25.9k citations).
8 figures · 6 tables · 6 charts · 48 sources
Financial
What it costs to distill an AI model: prices, training costs and the market shocks
Distillation is, at bottom, an arbitrage: the capability embedded in a $40M-$500M frontier training run can be harvested through an API for a four- or five-figure query bill and re-trained into a small model for hundreds of dollars.
8 figures · 8 tables · 7 charts · 63 sources
Political
Is AI distillation legal? Policy, export controls and the geopolitical fight
Between January 2025 and September 2026, model distillation went from an obscure machine-learning technique to a named object of US national security policy.
8 figures · 6 tables · 5 charts · 100 sources
Company
Which AI labs use distillation, which sell it, and which ban it
Every major AI lab now uses knowledge distillation to build its small and mid-tier models: Google states in the Gemini 2.5 report that all models 'Flash size and below' are distilled, Meta co-distilled Llama 4 Maverick from the 2-trillion-parameter Behemoth,…
8 figures · 7 tables · 7 charts · 75 sources
Developer
How to distill an LLM: tooling, platforms and recipes (2026)
By September 2026 a developer can distill a model three ways: with open libraries (Hugging Face TRL now ships four distillation trainers, plus Arcee DistillKit, torchtune, NVIDIA Model Optimizer/NeMo and an Axolotl KD plugin), with managed cloud pipelines…
8 figures · 7 tables · 8 charts · 58 sources
Customer
Distilled vs frontier AI models: which small model to actually deploy
For an enterprise buyer in September 2026, the distillation question is no longer "is the small model good enough" but "which small model, and what does the licence let me do with it".
8 figures · 5 tables · 6 charts · 78 sources
Library
The distillation method library: every method, how it works and where it fails
Knowledge distillation is not one technique but a family of at least two dozen distinct methods, separated by what signal crosses from teacher to student (logits, hidden features, pairwise relations, sampled text, preferences, or denoising trajectories) and by…
8 figures · 4 tables · 5 charts · 66 sources
Timeline
History of AI distillation, 2006 to 2026
Knowledge distillation began as an academic model-compression trick (Bucila 2006, Hinton 2015) and spent a decade as a research topic before becoming the default way to build small language models (DistilBERT 2019, Gemma, Llama 3.2, Qwen3).
6 figures · 1 table · 3 charts · 111 sources
Live signals · refreshed 19 October 2026
arXiv preprints matching “knowledge distillation”, per year
12 rows| Year | Preprints papers |
|---|---|
| 2015 | 2 |
| 2016 | 6 |
| 2017 | 20 |
| 2018 | 54 |
| 2019 | 163 |
| 2020 | 342 |
| 2021 | 467 |
| 2022 | 641 |
| 2023 | 807 |
| 2024 | 1,016 |
| 2025 | 1,120 |
| 2026 | 866 |
All time: 5,505 preprints; last 30 days: 102. A paper count measures attention, not quality.
Sources: info.arxiv.org · global-distillation.com
Tracked distilled models on Hugging Face
10 rows| Model | Downloads downloads, 30 days | Likes likes |
|---|---|---|
| deepseek-ai/DeepSeek-R1-Distill-Qwen-32B | 368,398 | 1,657 huggingface.co |
| deepseek-ai/DeepSeek-R1-Distill-Qwen-7B | 385,097 | 934 huggingface.co |
| deepseek-ai/DeepSeek-R1-Distill-Llama-8B | 134,030 | 911 huggingface.co |
| deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B | 1,296,471 | 1,638 huggingface.co |
| distilbert/distilbert-base-uncased | 7,241,353 | 1,489 huggingface.co |
| Qwen/Qwen3-8B | 9,122,623 | 2,125 huggingface.co |
| google/gemma-3-4b-it | 1,080,492 | 1,583 huggingface.co |
| meta-llama/Llama-3.2-3B-Instruct | 1,201,977 | 2,814 huggingface.co |
| microsoft/Phi-4-mini-instruct | 370,764 | 861 huggingface.co |
| HuggingFaceTB/SmolLM3-3B | 751,206 | 1,066 huggingface.co |
16,823 models on the Hub match “distill”.
Sources: huggingface.co · global-distillation.com
Distillation tooling on GitHub
15 rows| Repository | Stars stars | Forks forks |
|---|---|---|
| huggingface/trl | 19,548 | 3,079 github.com |
| arcee-ai/DistillKit | 1,084 | 143 github.com |
| pytorch/torchtune | 5,818 | 759 github.com |
| NVIDIA/NeMo | 18,602 | 3,650 github.com |
| axolotl-ai-cloud/axolotl | 12,591 | 1,598 github.com |
| unslothai/unsloth | 78,236 | 7,238 github.com |
| hiyouga/LLaMA-Factory | 75,576 | 9,263 github.com |
| vllm-project/vllm | 94,099 | 23,568 github.com |
| sgl-project/sglang | 37,312 | 9,678 github.com |
| huggingface/open-r1 | 26,475 | 2,452 github.com |
| open-thoughts/open-thoughts | 2,346 | 197 github.com |
| EleutherAI/lm-evaluation-harness | 14,264 | 3,693 github.com |
| SafeAILab/EAGLE | 2,555 | 297 github.com |
| NovaSky-AI/SkyThought | 3,401 | 345 github.com |
| simplescaling/s1 | 6,672 | 756 github.com |
Stars measure attention, not adoption.
Sources: docs.github.com · global-distillation.com
Recent items
8 items- The AI Model Distillation Paradox
- Jensen Huang says AI distillation is 'competition.'
- Feds accuse China of 'systematic' distillation of U.S. AI models
- Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it
- The tech world is suddenly obsessed with one concept in AI: Distillation
- Top White House official escalating the fight over Moonshot AI's Kimi K3 model
- Updating IP Regulations for AI Distillation
- Responding to AI Distillation Without Panic
Items are listed as published, never summarised into a claim.
Latest events · most recent 8
OpenAI launches GPT-6 Astra at $10/$50 per million tokens
OpenAI calls Astra a generational leap and the first model to trigger its highest internal cyber safeguards; president Greg Brockman frames it as the start of AGI. At 40x the input price of GPT-5.4 mini it resets the gap that the next distilled tier will have to close.
Source: cnbc.comAnthropic releases Claude Fable 5.1 and Claude Mythos 5.1
Same weights with different safeguard levels: Fable 5.1 is generally available with cheaper cache reads and fewer false-positive refusals, while Mythos 5.1 stays limited to vetted US organisations under trusted-access programs.
Source: techcrunch.comAnthropic publishes how Claude's text watermark works
Anthropic uses a SynthID-Text-style scheme that biases the randomness used to choose among equally good word options with a cryptographic key, leaving no hidden characters and no token cost. Anthropic frames it as provenance for EU transparency compliance rather than an anti-distillation control, since it only marks words Claude actually chooses.
Source: anthropic.comDeepSeek V4-Pro (1.6T parameters) reaches general availability
The 0813 build ends a preview that began on 24 April and completes the V4 rollout across app, web and API. V4's post-training merges RL-trained domain specialists into the unified model by on-policy distillation.
Source: api-docs.deepseek.comAnthropic announces it will watermark text generated by Claude
The commitment follows Anthropic's July 2026 signature on the EU Code of Practice on Transparency of AI-Generated Content and applies at the model level to every Claude surface for models released after 2 August.
Source: techcrunch.comMeta releases Muse Glimmer 30B, distilled from the closed Muse Spark
A 29.6B dense vision-language agentic model with a ViT-G/14 encoder and 128K context, Apache 2.0, quantised to run under 20GB of RAM. It is the clearest 2026 case of a US lab shipping open weights that are an admitted distillation of its own unreleased flagship.
Source: neowin.netByteDance founder Zhang Yiming bans distillation inside the Seed AI team
Zhang tells an internal all-hands that ByteDance will not distil closed or open-weight models, including Kimi K3, even at the cost of falling behind domestic rivals, and Seed issues a policy with API-level detection. It is the first Chinese lab to renounce the practice outright.
Source: technode.comSenators introduce the BLADE Act (S. 5252) to block large-scale adversarial distillation
Hagerty, Scott, Kim and Cortez Masto's bipartisan bill directs the executive branch to publicly list foreign distillation actors, work with industry on detection, and authorises Commerce export controls and Treasury sanctions against them.
Source: hagerty.senate.gov