A public compendium · updated daily
How frontier intelligence is compressed, priced, and contested.
Distillation moves capability from a large teacher model into a small, cheap student. It is the quiet engine behind most models people actually pay for, the subject of an open dispute between the largest labs, and a live regulatory question in three jurisdictions. This compendium tracks all of it.
Key figures · one per perspective
Citations: Hinton, Vinyals & Dean (2015)
25,899 citations
Semantic Scholar citation count for arXiv:1503.02531, retrieved 2026-09-04
Nvidia single-day market-cap loss
589 USD billions
27 Jan 2025, after DeepSeek-R1 and its distilled students shipped. Largest one-day loss in US market history.
Claude exchanges in largest disclosed campaign
28,800,000 exchanges
Anthropic's June 10, 2026 letter to Senate Banking alleges Alibaba/Qwen-affiliated operators ran 28.8M+ exchanges between Apr 22 and Jun 5, 2026
Fraudulent accounts in Anthropic's Feb 2026 report
24,000 accounts
DeepSeek, Moonshot AI and MiniMax combined; MiniMax alone drove 13M+ exchanges
DeepSeek-R1-Distill family, all-time HF downloads
97,824,225 downloads
Sum of the six official R1-Distill repos (1.5B, 7B, 8B, 14B, 32B, 70B) via the Hugging Face API on 2026-09-03
Enterprise inference price index
1.17 USD / M tokens
Silicon Data index cited by Jefferies; hit a 2026 low of $1.16–$1.18 on 6–8 Aug 2026, down from $2.04 on 31 May and $1.45 in late July.
Methods catalogued here
27 methods
Counted in extras.methods; families extend the response / feature / relation trichotomy of Gou et al. (arXiv 2006.05525) with six additional families specific to generative and compression-composed KD; eleven families in total.
Dated events in this timeline
130 events
Every event carries its own source URL; spans 2006-08-20 to 2026-09-03.
The perspectives · 8 sections
Academic
Academic view of AI distillation: from dark knowledge to on-policy reasoning transfer
Knowledge distillation began as a model-compression trick — Buciluă, Caruana and Niculescu-Mizil compressed an ensemble into a single net in 2006, and Hinton, Vinyals and Dean gave it its modern soft-target formulation in 2015 (now ~25.9k citations).
8 figures · 6 tables · 6 charts · 48 sources
Financial
The economics of AI distillation: prices, training costs, and the market shocks
Distillation is, at bottom, an arbitrage: the capability embedded in a $40M-$500M frontier training run can be harvested through an API for a four- or five-figure query bill and re-trained into a small model for hundreds of dollars.
8 figures · 8 tables · 7 charts · 63 sources
Political
Political and geopolitical view of AI distillation
Between January 2025 and September 2026, model distillation went from an obscure machine-learning technique to a named object of US national security policy.
8 figures · 6 tables · 5 charts · 100 sources
Company
Company view of AI distillation: who uses it, who sells it, who polices it
Every major AI lab now uses knowledge distillation to build its small and mid-tier models: Google states in the Gemini 2.5 report that all models 'Flash size and below' are distilled, Meta co-distilled Llama 4 Maverick from the 2-trillion-parameter Behemoth,…
8 figures · 7 tables · 7 charts · 75 sources
Developer
Developer view of AI distillation: tooling, platforms and recipes (2026)
By September 2026 a developer can distill a model three ways: with open libraries (Hugging Face TRL now ships four distillation trainers, plus Arcee DistillKit, torchtune, NVIDIA Model Optimizer/NeMo and an Axolotl KD plugin), with managed cloud pipelines…
8 figures · 7 tables · 8 charts · 58 sources
Customer
The buyer’s view of AI distillation: which small model to actually deploy
For an enterprise buyer in September 2026, the distillation question is no longer "is the small model good enough" but "which small model, and what does the licence let me do with it".
8 figures · 5 tables · 6 charts · 78 sources
Library
The Distillation Method Library
Knowledge distillation is not one technique but a family of at least two dozen distinct methods, separated by what signal crosses from teacher to student (logits, hidden features, pairwise relations, sampled text, preferences, or denoising trajectories) and by…
8 figures · 4 tables · 5 charts · 66 sources
Timeline
Cross-cutting timeline of AI distillation, 2006 to 2026
Knowledge distillation began as an academic model-compression trick (Bucila 2006, Hinton 2015) and spent a decade as a research topic before becoming the default way to build small language models (DistilBERT 2019, Gemma, Llama 3.2, Qwen3).
6 figures · 1 table · 3 charts · 111 sources
Live signals · refreshed 4 September 2026
arXiv preprints matching “knowledge distillation”, per year
12 rows| Year | Preprints papers |
|---|---|
| 2015 | 2 |
| 2016 | 6 |
| 2017 | 20 |
| 2018 | 54 |
| 2019 | 163 |
| 2020 | 342 |
| 2021 | 467 |
| 2022 | 641 |
| 2023 | 807 |
| 2024 | 1,016 |
| 2025 | 1,122 |
| 2026 | 731 |
All time: 5,372 preprints; last 30 days: 78. A paper count measures attention, not quality.
Sources: info.arxiv.org · global-distillation.com
Tracked distilled models on Hugging Face
10 rows| Model | Downloads downloads, 30 days | Likes likes |
|---|---|---|
| deepseek-ai/DeepSeek-R1-Distill-Qwen-32B | 562,818 | 1,608 huggingface.co |
| deepseek-ai/DeepSeek-R1-Distill-Qwen-7B | 367,771 | 881 huggingface.co |
| deepseek-ai/DeepSeek-R1-Distill-Llama-8B | 379,197 | 875 huggingface.co |
| deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B | 461,220 | 1,571 huggingface.co |
| distilbert/distilbert-base-uncased | 7,067,963 | 1,127 huggingface.co |
| Qwen/Qwen3-8B | 13,232,997 | 1,343 huggingface.co |
| google/gemma-3-4b-it | 1,626,709 | 1,470 huggingface.co |
| meta-llama/Llama-3.2-3B-Instruct | 1,419,885 | 2,512 huggingface.co |
| microsoft/Phi-4-mini-instruct | 456,948 | 829 huggingface.co |
| HuggingFaceTB/SmolLM3-3B | 581,340 | 1,022 huggingface.co |
16,329 models on the Hub match “distill”.
Sources: huggingface.co · global-distillation.com
Distillation tooling on GitHub
15 rows| Repository | Stars stars | Forks forks |
|---|---|---|
| huggingface/trl | 19,223 | 2,958 github.com |
| arcee-ai/DistillKit | 1,052 | 141 github.com |
| pytorch/torchtune | 5,804 | 752 github.com |
| NVIDIA/NeMo | 18,388 | 3,603 github.com |
| axolotl-ai-cloud/axolotl | 12,440 | 1,421 github.com |
| unslothai/unsloth | 75,624 | 6,872 github.com |
| hiyouga/LLaMA-Factory | 74,577 | 9,141 github.com |
| vllm-project/vllm | 90,979 | 21,712 github.com |
| sgl-project/sglang | 35,476 | 8,557 github.com |
| huggingface/open-r1 | 26,452 | 2,448 github.com |
| open-thoughts/open-thoughts | 2,331 | 194 github.com |
| EleutherAI/lm-evaluation-harness | 13,890 | 3,538 github.com |
| SafeAILab/EAGLE | 2,524 | 295 github.com |
| NovaSky-AI/SkyThought | 3,399 | 344 github.com |
| simplescaling/s1 | 6,667 | 756 github.com |
Stars measure attention, not adoption.
Sources: docs.github.com · global-distillation.com
Recent items
8 items- Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it
- The tech world is suddenly obsessed with one concept in AI: Distillation
- Top White House official escalating the fight over Moonshot AI's Kimi K3 model
- Updating IP Regulations for AI Distillation
- Responding to AI Distillation Without Panic
- A brief history of distillation in AI
- American A.I. Companies Say Chinese Copycats Are Quickly Catching Up
- Anthropic Accuses Alibaba of Largest AI Distillation Attack: 28.8M Fraudulent
Items are listed as published, never summarised into a claim.
Latest events · most recent 8
OpenAI launches GPT-6 Astra at $10/$50 per million tokens
OpenAI calls Astra a generational leap and the first model to trigger its highest internal cyber safeguards; president Greg Brockman frames it as the start of AGI. At 40x the input price of GPT-5.4 mini it resets the gap that the next distilled tier will have to close.
Source: cnbc.comAnthropic releases Claude Fable 5.1 and Claude Mythos 5.1
Same weights with different safeguard levels: Fable 5.1 is generally available with cheaper cache reads and fewer false-positive refusals, while Mythos 5.1 stays limited to vetted US organisations under trusted-access programs.
Source: techcrunch.comAnthropic publishes how Claude's text watermark works
Anthropic uses a SynthID-Text-style scheme that biases the randomness used to choose among equally good word options with a cryptographic key, leaving no hidden characters and no token cost. Anthropic frames it as provenance for EU transparency compliance rather than an anti-distillation control, since it only marks words Claude actually chooses.
Source: anthropic.comDeepSeek V4-Pro (1.6T parameters) reaches general availability
The 0813 build ends a preview that began on 24 April and completes the V4 rollout across app, web and API. V4's post-training merges RL-trained domain specialists into the unified model by on-policy distillation.
Source: api-docs.deepseek.comAnthropic announces it will watermark text generated by Claude
The commitment follows Anthropic's July 2026 signature on the EU Code of Practice on Transparency of AI-Generated Content and applies at the model level to every Claude surface for models released after 2 August.
Source: techcrunch.comMeta releases Muse Glimmer 30B, distilled from the closed Muse Spark
A 29.6B dense vision-language agentic model with a ViT-G/14 encoder and 128K context, Apache 2.0, quantised to run under 20GB of RAM. It is the clearest 2026 case of a US lab shipping open weights that are an admitted distillation of its own unreleased flagship.
Source: neowin.netByteDance founder Zhang Yiming bans distillation inside the Seed AI team
Zhang tells an internal all-hands that ByteDance will not distil closed or open-weight models, including Kimi K3, even at the cost of falling behind domestic rivals, and Seed issues a policy with API-level detection. It is the first Chinese lab to renounce the practice outright.
Source: technode.comSenators introduce the BLADE Act (S. 5252) to block large-scale adversarial distillation
Hagerty, Scott, Kim and Cortez Masto's bipartisan bill directs the executive branch to publicly list foreign distillation actors, work with industry on detection, and authorises Commerce export controls and Treasury sanctions against them.
Source: hagerty.senate.gov