{ "perspective": "library", "title": "The Distillation Method Library", "updated": "2026-09-04", "summary": "Knowledge distillation is not one technique but a family of at least two dozen distinct methods, separated by what signal crosses from teacher to student (logits, hidden features, pairwise relations, sampled text, preferences, or denoising trajectories) and by whether you can open the teacher at all. This library catalogues 27 methods with their loss functions, data and access requirements, tooling, and honest trade-offs. The central split is white-box versus black-box: white-box methods (soft targets, feature hints, reverse-KL, on-policy GKD) need teacher logits or activations and give the densest supervision per token, while black-box methods (SFT on API outputs, chain-of-thought and reasoning-trace distillation) need only sampled text and are what most practitioners actually run — and what triggered the 2025-2026 legal fights. The frontier has moved from static logit matching to on-policy methods where the student generates and the teacher grades: the Qwen3 technical report (Table 21) gives 74.4% on AIME'24 for a Qwen3-8B student at 1,800 GPU-hours of on-policy distillation versus 67.6% for RL at 17,920 GPU-hours, and Thinking Machines reproduced the method on the Tinker API. Method choice is mostly determined by three constraints — teacher access, label availability, and whether the student's architecture or tokenizer matches the teacher's — and this library is organised so you can walk those three constraints to a shortlist.", "stats": [ { "label": "Methods catalogued here", "value": 27, "unit": "methods", "delta": "spanning 2014-2025", "note": "Counted in extras.methods; families extend the response / feature / relation trichotomy of Gou et al. (arXiv 2006.05525) with six additional families specific to generative and compression-composed KD; eleven families in total.", "source": "https://arxiv.org/abs/2006.05525" }, { "label": "DistilBERT GLUE retention", "value": 97, "unit": "% of BERT-base", "delta": "40% fewer params, 60% faster", "note": "The canonical compression datapoint for response+feature distillation on encoders.", "source": "https://arxiv.org/abs/1910.01108" }, { "label": "TinyBERT-4L GLUE retention", "value": 96.8, "unit": "% of BERT-base", "delta": "7.5x smaller, 9.4x faster", "note": "Adds attention-matrix and hidden-state losses on top of logit matching.", "source": "https://arxiv.org/abs/1909.10351" }, { "label": "DeepSeek-R1-Distill-Qwen-32B, AIME 2024", "value": 72.6, "unit": "% pass@1", "delta": "pure SFT on 800k R1 traces, no RL stage", "note": "Shows sequence-level reasoning-trace distillation alone can transplant frontier reasoning into a 32B open-weight student.", "source": "https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" }, { "label": "Alpaca teacher-data generation cost", "value": 500, "unit": "USD (upper bound)", "delta": "52k instructions; <$100 more to finetune", "note": "Stanford CRFM reports under $500 of OpenAI API calls plus 3 hours on 8x A100 80GB.", "source": "https://crfm.stanford.edu/2023/03/13/alpaca.html" }, { "label": "s1K reasoning-distillation dataset size", "value": 1000, "unit": "examples", "delta": "beats o1-preview on AIME24/MATH by up to 27%", "note": "1,000 curated questions with traces distilled from Gemini Thinking Experimental, plus budget forcing at inference.", "source": "https://arxiv.org/abs/2501.19393" }, { "label": "On-policy distillation vs RL, GPU-hours", "value": 1800, "unit": "GPU-hours", "delta": "vs 17,920 for RL at a lower score", "note": "Qwen3-8B student: 74.4% AIME'24 via on-policy distillation at 1,800 GPU-hours vs 67.6% via RL at 17,920.", "source": "https://thinkingmachines.ai/blog/on-policy-distillation/" }, { "label": "Latent Consistency Model training cost", "value": 32, "unit": "A100 GPU-hours", "delta": "50 steps -> 2-4 steps at 768x768", "note": "Diffusion distillation is the cheapest high-leverage distillation in this library.", "source": "https://arxiv.org/abs/2310.04378" } ], "keyFindings": [ { "title": "The first question is not 'which loss' but 'what does the teacher let you see'", "detail": "Every method in this library sits on one side of a hard line. White-box methods need per-token logits or internal activations and are only available if you host the teacher's weights. Black-box methods need only sampled text and work against any API. That single constraint eliminates roughly half the catalogue before you compare anything else. If you are distilling from a commercial API you are restricted to sequence-level, chain-of-thought, preference and dataset methods — no soft targets, no feature hints, no reverse-KL.", "audience": [ "developer", "customer" ], "sources": [ "https://arxiv.org/abs/2402.13116", "https://arxiv.org/abs/1606.07947" ] }, { "title": "Soft targets carry more information per example than labels — that is the whole original insight", "detail": "Hinton, Vinyals and Dean's 2015 argument is that a teacher's full probability vector over wrong classes (the 'dark knowledge': a 2 that looks slightly like a 7) has high entropy and therefore more bits per training case and lower gradient variance than a one-hot label. Raising the softmax temperature exposes those relative probabilities. Because gradients through the softened softmax scale as 1/T^2, the soft-target term must be multiplied by T^2 to keep the two objectives balanced when T is tuned.", "audience": [ "academic", "developer" ], "sources": [ "https://www.cs.toronto.edu/~hinton/absps/distillation.pdf", "https://arxiv.org/abs/1503.02531" ] }, { "title": "On-policy methods fixed the exposure-bias problem and are now the default frontier recipe", "detail": "Off-policy distillation trains the student on the teacher's own sequences, so at inference the student meets its own distribution for the first time and compounds errors. GKD (2023) and MiniLLM (2023) fix this by sampling from the student and having the teacher grade those tokens, with a divergence (reverse KL or generalized JSD) that tolerates a student too small to cover the teacher's modes. The Qwen3 technical report (Table 21) reports a Qwen3-8B student at 74.4 on AIME'24 for 1,800 GPU-hours of on-policy distillation versus 67.6 for RL at 17,920, up from 55.0 off-policy; Thinking Machines reproduced the method with the Tinker API and reached roughly 70% AIME'24 from a Qwen3-8B-Base SFT-400K checkpoint.", "audience": [ "developer", "academic" ], "sources": [ "https://arxiv.org/abs/2306.13649", "https://arxiv.org/abs/2306.08543", "https://arxiv.org/html/2505.09388v1", "https://thinkingmachines.ai/blog/on-policy-distillation/" ] }, { "title": "Rationales beat labels: distilling the reasoning is worth more than distilling the answer", "detail": "Distilling Step-by-Step extracts teacher rationales as a second supervised task alongside the label, and reports a 770M T5 outperforming a few-shot-prompted 540B PaLM using only 80% of the available data — roughly a 700x parameter reduction. DeepSeek pushed the same idea to its limit by fine-tuning Qwen and Llama students on 800k long chain-of-thought traces from R1, with no RL stage, reaching 72.6% pass@1 on AIME 2024 at 32B.", "audience": [ "developer", "customer", "academic" ], "sources": [ "https://arxiv.org/abs/2305.02301", "https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" ] }, { "title": "Distillation is not free: scaling laws say it only wins when the teacher cost is amortised", "detail": "Apple's 2025 distillation scaling law finds that when a teacher already exists, or when one teacher serves many students, distillation beats supervised learning up to a compute level that scales predictably with student size. But if you must train a teacher to produce exactly one student, plain supervised learning is generally preferable. This is the single most useful economic result in the field and it contradicts the folk assumption that distillation is always the cheaper path.", "audience": [ "financial", "developer" ], "sources": [ "https://arxiv.org/abs/2502.08606" ] }, { "title": "Prune first, then distill — compression now routinely stacks", "detail": "The strongest small-model recipes are compositions, not single methods. NVIDIA's Minitron prunes Llama 3.1 8B along depth or width and re-trains with distillation on under 1% of the original token budget (94B tokens vs 15T, ~160x fewer), yielding 1.8x-2.7x speedups. Sheared LLaMA combines targeted structured pruning with dynamic batch loading to beat same-size models trained from scratch. Quantization-aware distillation then recovers the accuracy that low-bit quantization costs.", "audience": [ "developer" ], "sources": [ "https://arxiv.org/abs/2408.11796", "https://arxiv.org/abs/2407.14679", "https://arxiv.org/abs/2310.06694", "https://arxiv.org/abs/2305.17888" ] }, { "title": "Tokenizer mismatch is the quiet blocker on white-box distillation", "detail": "Logit-matching requires teacher and student to share a vocabulary, which rules out most cross-family pairs (Llama teacher, Qwen student). Two 2024 lines of work attack this: Universal Logit Distillation uses an optimal-transport cost between sorted probability vectors so no token alignment is needed, and Dual-Space KD projects both models into a shared output space with a cross-model attention mechanism. Both are still less reliable than same-tokenizer KD, and torchtune's own roadmap lists cross-tokenizer support as future work.", "audience": [ "developer", "academic" ], "sources": [ "https://arxiv.org/abs/2402.12030", "https://arxiv.org/abs/2406.17328", "https://pytorch.org/blog/llama-into-torchtune/" ] }, { "title": "Distillation is now infrastructure, not research — and its abuse is now a public legal fight", "detail": "OpenAI shipped Model Distillation into its API in late 2024 (stored completions plus evals plus fine-tuning), and Amazon Bedrock Model Distillation reached general availability on 1 May 2025 with claims of up to 500% faster and up to 75% cheaper distilled models at under 2% accuracy loss for RAG. The flip side arrived on 23 February 2026, when Anthropic disclosed what it called industrial-scale distillation attacks: roughly 24,000 fraudulent accounts and over 16 million exchanges attributed to DeepSeek, Moonshot AI and MiniMax, targeting agentic reasoning, tool use and coding.", "audience": [ "political", "company", "customer" ], "sources": [ "https://openai.com/index/api-model-distillation/", "https://aws.amazon.com/blogs/machine-learning/amazon-bedrock-model-distillation-boost-function-calling-accuracy-while-reducing-cost-and-latency/", "https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks" ] }, { "title": "A fine-tuned teacher distills better than a raw one", "detail": "The PyTorch torchtune case study distilling Llama 3.1 8B into Llama 3.2 1B on Alpaca found that KD loss stayed nearly flat when the teacher was not first adapted to the transfer set, and that using a LoRA-finetuned teacher gave the best hellaswag and commonsense results. NVIDIA independently reports the same thing as 'teacher correction' — a light fine-tune of the teacher on the distillation corpus before pruning-and-distilling. The lesson: the teacher's distribution must match the transfer data or the distillation signal is nearly uninformative.", "audience": [ "developer" ], "sources": [ "https://pytorch.org/blog/llama-into-torchtune/", "https://arxiv.org/abs/2408.11796" ] }, { "title": "Distillation has escaped model compression entirely", "detail": "Four of the methods here do not shrink a model at all. Self-distillation (Born-Again Networks) trains a student identical to its teacher and still beats it. Dataset distillation compresses the training set rather than the network. Draft-model distillation (EAGLE, Medusa) trains a tiny head purely to make speculative decoding accept more tokens, leaving the target model's output distribution unchanged. Diffusion distillation compresses the number of sampling steps, not the parameter count — LCM reaches 2-4 step generation for 32 A100-hours.", "audience": [ "academic", "developer" ], "sources": [ "https://arxiv.org/abs/1805.04770", "https://arxiv.org/abs/1811.10959", "https://arxiv.org/abs/2401.15077", "https://arxiv.org/abs/2310.04378" ] } ], "tables": [ { "id": "method-families", "title": "The eleven families of distillation, and what actually crosses from teacher to student", "description": "Every method in this library belongs to one of these families. The family determines what you must be able to read out of the teacher, which is almost always the binding constraint.", "columns": [ { "key": "family", "label": "Family", "type": "text" }, { "key": "signal", "label": "Signal transferred", "type": "text" }, { "key": "origin", "label": "Origin paper", "type": "text" }, { "key": "year", "label": "Year", "type": "number" }, { "key": "access", "label": "Teacher access", "type": "text" }, { "key": "modality", "label": "Where it dominates", "type": "text" }, { "key": "methodsHere", "label": "Methods in this library", "type": "number" } ], "rows": [ { "family": "Response-based (logit)", "signal": "Softened output distribution over classes/tokens", "origin": "Hinton, Vinyals & Dean", "year": 2015, "access": "white-box", "modality": "Classifiers, encoders, same-tokenizer LLM pairs", "methodsHere": 6, "_source": "https://arxiv.org/abs/1503.02531" }, { "family": "Feature / intermediate", "signal": "Hidden activations at chosen layers, via a projector", "origin": "FitNets (Romero et al.)", "year": 2014, "access": "white-box", "modality": "CNNs, BERT-family encoders", "methodsHere": 2, "_source": "https://arxiv.org/abs/1412.6550" }, { "family": "Attention-based", "signal": "Spatial or head attention maps", "origin": "Zagoruyko & Komodakis", "year": 2016, "access": "white-box", "modality": "CNNs; transformer attention matrices", "methodsHere": 1, "_source": "https://arxiv.org/abs/1612.03928" }, { "family": "Relation-based", "signal": "Pairwise/triplet structure of the embedding space", "origin": "RKD (Park et al.)", "year": 2019, "access": "white-box", "modality": "Metric learning, retrieval, re-ID", "methodsHere": 3, "_source": "https://arxiv.org/abs/1904.05068" }, { "family": "Sequence-level / data", "signal": "Sampled teacher output sequences used as hard targets", "origin": "Kim & Rush", "year": 2016, "access": "black-box", "modality": "MT, instruction tuning, any API teacher", "methodsHere": 2, "_source": "https://arxiv.org/abs/1606.07947" }, { "family": "Rationale / reasoning", "signal": "Chain-of-thought traces as an extra supervised target", "origin": "Distilling Step-by-Step (Hsieh et al.)", "year": 2023, "access": "black-box", "modality": "Math, code, multi-step QA, agents", "methodsHere": 2, "_source": "https://arxiv.org/abs/2305.02301" }, { "family": "On-policy / divergence-choice", "signal": "Teacher scores the student's own samples", "origin": "GKD (Agarwal et al.) and MiniLLM (Gu et al.)", "year": 2023, "access": "white-box", "modality": "Generative LLMs where exposure bias dominates", "methodsHere": 3, "_source": "https://arxiv.org/abs/2306.13649" }, { "family": "Preference", "signal": "Teacher- or AI-generated preference pairs", "origin": "Zephyr dDPO (Tunstall et al.)", "year": 2023, "access": "black-box", "modality": "Chat alignment, style, helpfulness", "methodsHere": 2, "_source": "https://arxiv.org/abs/2310.16944" }, { "family": "Compression-composed", "signal": "KD used to repair a pruned or quantized network", "origin": "Minitron / LLM-QAT", "year": 2023, "access": "white-box", "modality": "Edge deployment, model-family shrinking", "methodsHere": 2, "_source": "https://arxiv.org/abs/2305.17888" }, { "family": "Trajectory / sampler", "signal": "Denoising trajectory or next-feature prediction", "origin": "Progressive Distillation (Salimans & Ho)", "year": 2022, "access": "white-box", "modality": "Diffusion samplers, speculative decoding drafts", "methodsHere": 2, "_source": "https://arxiv.org/abs/2202.00512" }, { "family": "Dataset / context", "signal": "The training set or the prompt itself, not the weights", "origin": "Dataset Distillation (Wang et al.); Askell et al.", "year": 2018, "access": "either", "modality": "Data-efficiency research; prompt internalisation", "methodsHere": 2, "_source": "https://arxiv.org/abs/1811.10959" } ], "notes": "'Methods in this library' counts entries in extras.methods; some methods legitimately span two families (e.g. TinyBERT is feature + attention + response) and are counted once under their dominant family.", "sources": [ "https://arxiv.org/abs/2503.12067", "https://arxiv.org/abs/2402.13116" ] }, { "id": "white-box-vs-black-box", "title": "White-box versus black-box distillation, dimension by dimension", "description": "The practical decision table. If you cannot host the teacher's weights, the right-hand column is your entire option space.", "columns": [ { "key": "dimension", "label": "Dimension", "type": "text" }, { "key": "whitebox", "label": "White-box (weights in hand)", "type": "text" }, { "key": "blackbox", "label": "Black-box (API only)", "type": "text" } ], "rows": [ { "dimension": "What you read from the teacher", "whitebox": "Full logit vector per token, hidden states, attention matrices, gradients", "blackbox": "Sampled text; sometimes top-k logprobs; nothing internal", "_source": "https://arxiv.org/abs/2402.13116" }, { "dimension": "Supervision density", "whitebox": "Dense: one target distribution per token position", "blackbox": "Sparse: one sampled sequence per prompt", "_source": "https://thinkingmachines.ai/blog/on-policy-distillation/" }, { "dimension": "Representative methods", "whitebox": "Soft targets, FitNets, attention transfer, RKD, CRD, MiniLLM, GKD, SKD, Minitron, QAD", "blackbox": "Seq-level KD, Alpaca/Vicuna-style SFT, Distilling Step-by-Step, R1-style trace distillation, dDPO, RLAIF", "_source": "https://arxiv.org/abs/1606.07947" }, { "dimension": "Tokenizer constraint", "whitebox": "Must match, unless you use ULD or DSKD", "blackbox": "None — text is tokenizer-agnostic", "_source": "https://arxiv.org/abs/2402.12030" }, { "dimension": "Architecture constraint", "whitebox": "Feature methods need layer-mapping and a projector; relation methods need comparable embedding spaces", "blackbox": "None", "_source": "https://arxiv.org/abs/1412.6550" }, { "dimension": "Marginal cost per training step", "whitebox": "One extra teacher forward pass (or cached logits)", "blackbox": "Zero after the dataset is generated; generation is a one-time API bill", "_source": "https://crfm.stanford.edu/2023/03/13/alpaca.html" }, { "dimension": "Typical wall-clock to first result", "whitebox": "Hours to days; needs a GPU large enough for teacher + student", "blackbox": "Hours; dataset generation can be parallelised across API keys", "_source": "https://pytorch.org/blog/llama-into-torchtune/" }, { "dimension": "Handles a very small student", "whitebox": "Better — reverse-KL and JSD variants let a low-capacity student pick modes instead of smearing", "blackbox": "Worse — the student must imitate full sequences it cannot represent", "_source": "https://arxiv.org/abs/2306.08543" }, { "dimension": "Legal exposure", "whitebox": "Governed by the teacher weights' licence (e.g. Llama, Apache-2.0, MIT)", "blackbox": "Governed by API terms; most frontier vendors forbid using outputs to build competing models", "_source": "https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks" }, { "dimension": "Detectability by the teacher's owner", "whitebox": "Not applicable", "blackbox": "High — Anthropic reports classifiers and behavioural fingerprinting that flag repeated narrow-capability prompt patterns across coordinated accounts", "_source": "https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks" }, { "dimension": "Managed-service support", "whitebox": "torchtune, TRL, DistillKit, NVIDIA Model Optimizer", "blackbox": "OpenAI Model Distillation, Amazon Bedrock Model Distillation, Vertex AI", "_source": "https://openai.com/index/api-model-distillation/" }, { "dimension": "Where it is state of the art", "whitebox": "Frontier-lab small models: Gemma 3 (all sizes distilled), Qwen3 strong-to-weak, Minitron", "blackbox": "Open-community reasoning models: DeepSeek-R1-Distill, s1, Orca, Zephyr", "_source": "https://arxiv.org/abs/2503.19786" } ], "notes": "'Grey-box' is a real third case: some APIs return top-k logprobs, which supports a truncated logit-matching objective but not full-vocabulary KL.", "sources": [ "https://arxiv.org/abs/2402.13116", "https://arxiv.org/abs/2306.08543", "https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks" ] }, { "id": "method-compute-data-needs", "title": "Every method: teacher access, data needed, compute profile, difficulty", "description": "The full 27-method matrix. 'Teacher compute' is the extra cost incurred by the teacher during student training (not the cost of training the teacher). Difficulty is a 1-5 implementation-effort rating, editorial, calibrated against the reference implementations linked in each method entry.", "columns": [ { "key": "method", "label": "Method", "type": "text" }, { "key": "access", "label": "Teacher access", "type": "text" }, { "key": "data", "label": "Data needed", "type": "text" }, { "key": "labels", "label": "Ground-truth labels?", "type": "text" }, { "key": "teacherCompute", "label": "Teacher compute during training", "type": "text" }, { "key": "difficulty", "label": "Difficulty (1-5)", "type": "number" } ], "rows": [ { "method": "Response-based KD (soft targets)", "access": "white-box", "data": "logits", "labels": "Optional (blended term)", "teacherCompute": "One forward pass per batch, cacheable", "difficulty": 1, "_source": "https://arxiv.org/abs/1503.02531" }, { "method": "Feature / hint KD (FitNets)", "access": "white-box", "data": "features", "labels": "Yes, in stage two", "teacherCompute": "One forward pass; plus a trained projector", "difficulty": 3, "_source": "https://arxiv.org/abs/1412.6550" }, { "method": "Attention transfer", "access": "white-box", "data": "features", "labels": "Yes", "teacherCompute": "One forward pass", "difficulty": 2, "_source": "https://arxiv.org/abs/1612.03928" }, { "method": "Relational KD (RKD)", "access": "white-box", "data": "features", "labels": "Optional", "teacherCompute": "One forward pass; O(batch^2) or O(batch^3) relation terms", "difficulty": 3, "_source": "https://arxiv.org/abs/1904.05068" }, { "method": "Contrastive representation distillation (CRD)", "access": "white-box", "data": "features", "labels": "Optional", "teacherCompute": "Forward pass plus a negatives memory bank", "difficulty": 4, "_source": "https://arxiv.org/abs/1910.10699" }, { "method": "Transformer-layer KD (TinyBERT / DistilBERT)", "access": "white-box", "data": "features", "labels": "Yes for the task stage", "teacherCompute": "Forward pass at both pretraining and task stages", "difficulty": 4, "_source": "https://arxiv.org/abs/1909.10351" }, { "method": "Sequence-level KD", "access": "black-box", "data": "outputs", "labels": "No", "teacherCompute": "One-time beam-search generation over the corpus", "difficulty": 1, "_source": "https://arxiv.org/abs/1606.07947" }, { "method": "Black-box output distillation (Alpaca/Vicuna style)", "access": "black-box", "data": "outputs", "labels": "No", "teacherCompute": "One-time API generation bill", "difficulty": 1, "_source": "https://crfm.stanford.edu/2023/03/13/alpaca.html" }, { "method": "Chain-of-thought / rationale distillation", "access": "black-box", "data": "outputs", "labels": "Optional (multi-task with labels)", "teacherCompute": "One-time generation, longer outputs so higher token bill", "difficulty": 2, "_source": "https://arxiv.org/abs/2305.02301" }, { "method": "Reasoning-trace distillation (R1-Distill, s1)", "access": "black-box", "data": "outputs", "labels": "Answers used for filtering", "teacherCompute": "Very high one-time cost: long traces, rejection sampling", "difficulty": 2, "_source": "https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" }, { "method": "Reverse-KL distillation (MiniLLM)", "access": "white-box", "data": "logits", "labels": "No", "teacherCompute": "Student generation plus teacher scoring each step", "difficulty": 5, "_source": "https://arxiv.org/abs/2306.08543" }, { "method": "Generalized KD (GKD, on-policy)", "access": "white-box", "data": "logits", "labels": "No", "teacherCompute": "Student generation plus teacher scoring; dominates step cost", "difficulty": 4, "_source": "https://arxiv.org/abs/2306.13649" }, { "method": "Speculative KD (SKD)", "access": "white-box", "data": "logits", "labels": "No", "teacherCompute": "Interleaved student proposal + teacher verification", "difficulty": 5, "_source": "https://arxiv.org/abs/2410.11325" }, { "method": "Self-distillation (Born-Again Networks)", "access": "white-box", "data": "logits", "labels": "Yes", "teacherCompute": "One full training generation per round", "difficulty": 2, "_source": "https://arxiv.org/abs/1805.04770" }, { "method": "Teacher-assistant KD (TAKD)", "access": "white-box", "data": "logits", "labels": "Yes", "teacherCompute": "Extra full training run for each intermediate model", "difficulty": 2, "_source": "https://arxiv.org/abs/1902.03393" }, { "method": "Multi-teacher KD", "access": "white-box", "data": "logits", "labels": "Optional", "teacherCompute": "K forward passes per batch, or K cached logit sets", "difficulty": 3, "_source": "https://www.isca-archive.org/interspeech_2017/fukuda17_interspeech.html" }, { "method": "Deep mutual learning", "access": "none (peers)", "data": "logits", "labels": "Yes", "teacherCompute": "No teacher; K students trained simultaneously", "difficulty": 2, "_source": "https://arxiv.org/abs/1706.00384" }, { "method": "Prune-then-distill (Minitron, Sheared LLaMA)", "access": "white-box", "data": "logits", "labels": "No", "teacherCompute": "Teacher forward pass over a re-training corpus (<1% of original tokens; 94B vs 15T)", "difficulty": 5, "_source": "https://arxiv.org/abs/2408.11796" }, { "method": "Quantization-aware distillation", "access": "white-box", "data": "logits", "labels": "No (LLM-QAT is data-free)", "teacherCompute": "Full-precision teacher forward pass each step", "difficulty": 4, "_source": "https://arxiv.org/abs/2305.17888" }, { "method": "Dataset distillation / condensation", "access": "none", "data": "none (synthesises data)", "labels": "Yes, for the real set", "teacherCompute": "No teacher; bi-level optimisation over inner training steps", "difficulty": 5, "_source": "https://arxiv.org/abs/1811.10959" }, { "method": "Preference distillation / RLAIF", "access": "black-box", "data": "outputs", "labels": "No", "teacherCompute": "Teacher ranks candidate completions", "difficulty": 4, "_source": "https://arxiv.org/abs/2310.16944" }, { "method": "DPO on teacher preferences (dDPO)", "access": "black-box", "data": "outputs", "labels": "No", "teacherCompute": "One-time preference labelling; no reward model, no PPO loop", "difficulty": 2, "_source": "https://arxiv.org/abs/2305.18290" }, { "method": "Cross-vocabulary logit KD (ULD, DSKD)", "access": "white-box", "data": "logits", "labels": "Optional", "teacherCompute": "Teacher forward pass plus an optimal-transport or projection step", "difficulty": 5, "_source": "https://arxiv.org/abs/2402.12030" }, { "method": "Draft-model distillation (EAGLE, Medusa)", "access": "white-box", "data": "features", "labels": "No", "teacherCompute": "Target-model features over a modest corpus; target weights frozen", "difficulty": 4, "_source": "https://arxiv.org/abs/2401.15077" }, { "method": "Retriever distillation (cross-encoder to bi-encoder)", "access": "white-box or grey", "data": "outputs (scores)", "labels": "No, teacher scores replace labels", "teacherCompute": "Cross-encoder scoring over query-document pairs; usually precomputed", "difficulty": 3, "_source": "https://arxiv.org/abs/2010.02666" }, { "method": "Diffusion distillation (progressive, consistency, LCM, ADD)", "access": "white-box", "data": "features", "labels": "No", "teacherCompute": "Teacher sampler queries; LCM reports 32 A100-hours end to end", "difficulty": 4, "_source": "https://arxiv.org/abs/2310.04378" }, { "method": "Context / prompt distillation", "access": "white-box (self)", "data": "logits", "labels": "No", "teacherCompute": "The same model prompted with the context acts as teacher", "difficulty": 2, "_source": "https://arxiv.org/abs/2209.15189" } ], "notes": "Difficulty is an editorial 1-5 implementation-effort rating, not a measured quantity. 'Data needed' uses the SCHEMA vocabulary: logits | outputs | features | none. The 'difficulty' column is an editorial 1-5 rating; each row's _source is the method's own paper, not a source for that rating.", "sources": [ "https://arxiv.org/abs/2402.13116", "https://huggingface.co/docs/trl/gkd_trainer", "https://pytorch.org/blog/llama-into-torchtune/" ] }, { "id": "loss-cheatsheet", "title": "Loss-function cheat sheet with the hyperparameters that actually matter", "description": "What you tune, and what a sensible starting value looks like according to the reference implementation.", "columns": [ { "key": "method", "label": "Method", "type": "text" }, { "key": "loss", "label": "Core objective", "type": "text" }, { "key": "knobs", "label": "Hyperparameters that matter", "type": "text" }, { "key": "defaults", "label": "Reference default", "type": "text" }, { "key": "impl", "label": "Reference implementation", "type": "text" } ], "rows": [ { "method": "Response-based KD", "loss": "a * T^2 * KL(teacher_T || student_T) + (1-a) * CE(labels)", "knobs": "Temperature T, mixing weight alpha", "defaults": "T in 2-10; alpha 0.5-0.9", "impl": "torchtune ForwardKLWithChunkedOutputLoss", "_source": "https://pytorch.org/blog/llama-into-torchtune/" }, { "method": "torchtune KD recipe", "loss": "CE + forward-KL on logits", "knobs": "kd_ratio, learning rate", "defaults": "kd_ratio 0.5, lr 3e-4; the blog finds kd_ratio 0.75-1.0 slightly better", "impl": "knowledge_distillation_single_device", "_source": "https://pytorch.org/blog/llama-into-torchtune/" }, { "method": "FitNets hint", "loss": "MSE(projector(student_feat), teacher_feat)", "knobs": "Which layer pair; projector width", "defaults": "Middle-layer hint, 1x1 conv regressor", "impl": "RepDistiller", "_source": "https://arxiv.org/abs/1412.6550" }, { "method": "Attention transfer", "loss": "L_p distance between L2-normalised attention maps", "knobs": "p in the map power sum; layer group set", "defaults": "p = 2, per residual-block group", "impl": "szagoruyko/attention-transfer", "_source": "https://arxiv.org/abs/1612.03928" }, { "method": "RKD", "loss": "Huber loss on distance ratios + angle cosines", "knobs": "Weighting of distance vs angle terms", "defaults": "Both terms, distance normalised by batch mean", "impl": "RepDistiller", "_source": "https://arxiv.org/abs/1904.05068" }, { "method": "CRD", "loss": "InfoNCE-style contrastive bound on mutual information", "knobs": "Number of negatives N, embedding dim", "defaults": "N in the thousands via a memory buffer", "impl": "HobbitLong/RepDistiller", "_source": "https://arxiv.org/abs/1910.10699" }, { "method": "TinyBERT", "loss": "Embedding MSE + attention MSE + hidden MSE + prediction CE", "knobs": "Layer mapping g(m); term weights; two-stage schedule", "defaults": "4-layer student, 312 hidden, uniform layer mapping", "impl": "huawei-noah/TinyBERT", "_source": "https://arxiv.org/abs/1909.10351" }, { "method": "Sequence-level KD", "loss": "NLL on the teacher's argmax (beam) output", "knobs": "Beam width for generation; whether to keep gold data", "defaults": "Beam 5; often mixed 50/50 with gold", "impl": "OpenNMT", "_source": "https://arxiv.org/abs/1606.07947" }, { "method": "MiniLLM", "loss": "Reverse KL, optimised on-policy with a policy-gradient estimator", "knobs": "rkl_advantage, single_step_decomposition, gamma, kd_temperature", "defaults": "TRL MiniLLMConfig: rkl_advantage True, gamma 0.0, temperature 1.0", "impl": "trl.experimental.minillm.MiniLLMTrainer", "_source": "https://huggingface.co/docs/trl/main/minillm" }, { "method": "GKD", "loss": "Generalized JSD, mixed on-policy/off-policy", "knobs": "lmbda (student-data fraction), beta (JSD interpolation), temperature, seq_kd", "defaults": "TRL GKDConfig: lmbda 0.5, beta 0.5, temperature 0.9", "impl": "trl.experimental.gkd.GKDTrainer", "_source": "https://huggingface.co/docs/trl/gkd_trainer" }, { "method": "On-policy distillation (Tinker form)", "loss": "Per-token reverse KL on student rollouts", "knobs": "Rollout length, teacher size", "defaults": "Qwen3-8B teacher, Qwen3-8B-Base student in the Thinking Machines run; the Qwen3 report's own run used a Qwen3-32B teacher.", "impl": "tinker-cookbook train_on_policy.py", "_source": "https://thinkingmachines.ai/blog/on-policy-distillation/" }, { "method": "dDPO", "loss": "Logistic loss on the implicit reward margin between chosen and rejected", "knobs": "beta (KL strength), reference model", "defaults": "Zephyr-7B used AI-ranked pairs and no human annotation", "impl": "trl DPOTrainer", "_source": "https://arxiv.org/abs/2310.16944" }, { "method": "Margin-MSE (retrieval)", "loss": "MSE between teacher and student score margins on (q, d+, d-)", "knobs": "Teacher ensemble size; negative sampling", "defaults": "3 BERT-cat teachers on MS MARCO passage", "impl": "sebastian-hofstaetter/neural-ranking-kd", "_source": "https://arxiv.org/abs/2010.02666" }, { "method": "Progressive distillation (diffusion)", "loss": "Student one step matches teacher's two DDIM steps", "knobs": "Number of halving rounds N", "defaults": "Halve step count repeatedly, e.g. 1024 -> 4", "impl": "diffusers", "_source": "https://arxiv.org/abs/2202.00512" }, { "method": "Consistency distillation", "loss": "Distance between f(x_{t+1}) and the EMA target at f(x_t)", "knobs": "EMA rate for the target network, discretisation schedule", "defaults": "OpenAI consistency_models repo settings", "impl": "openai/consistency_models", "_source": "https://arxiv.org/abs/2303.01469" }, { "method": "LLM-QAT", "loss": "KD on data generated by the model itself, with straight-through quantizers", "knobs": "Bit widths for W/A/KV-cache", "defaults": "Weights, activations and KV cache all quantized", "impl": "facebookresearch/LLM-QAT", "_source": "https://arxiv.org/abs/2305.17888" }, { "method": "EAGLE", "loss": "Regression on second-to-top-layer features + CE on tokens", "knobs": "Draft tree shape/depth; feature layer choice", "defaults": "One autoregressive draft head; EAGLE-3 fuses multiple layers", "impl": "SafeAILab/EAGLE, vLLM, SGLang", "_source": "https://arxiv.org/abs/2401.15077" }, { "method": "ULD", "loss": "Optimal-transport (Wasserstein) cost between sorted probability vectors", "knobs": "Truncation of the sorted vectors", "defaults": "No token alignment required", "impl": "Nicolas-BZRD/llm-recipes", "_source": "https://arxiv.org/abs/2402.12030" } ], "notes": "Formulas are written informally here; the exact LaTeX for each is in extras.methods[].lossFormula. TRL defaults quoted were read from the TRL v1.12.0 documented GKDConfig and MiniLLMConfig.", "sources": [ "https://huggingface.co/docs/trl/gkd_trainer", "https://huggingface.co/docs/trl/main/minillm", "https://pytorch.org/blog/llama-into-torchtune/" ] } ], "charts": [ { "id": "radar-six-methods", "title": "Six representative methods, six axes", "type": "radar", "xLabel": "Dimension", "yLabel": "Score (1-5, higher is better)", "unit": "score", "series": [ { "name": "Response-based KD (soft targets)", "data": [ { "x": "Quality retention", "y": 3 }, { "x": "Data efficiency", "y": 3 }, { "x": "Implementation simplicity", "y": 5 }, { "x": "Works without teacher internals", "y": 1 }, { "x": "Low training compute", "y": 4 }, { "x": "Breadth of applicability", "y": 4 } ] }, { "name": "Black-box output distillation", "data": [ { "x": "Quality retention", "y": 3 }, { "x": "Data efficiency", "y": 2 }, { "x": "Implementation simplicity", "y": 5 }, { "x": "Works without teacher internals", "y": 5 }, { "x": "Low training compute", "y": 4 }, { "x": "Breadth of applicability", "y": 5 } ] }, { "name": "Reasoning-trace distillation", "data": [ { "x": "Quality retention", "y": 5 }, { "x": "Data efficiency", "y": 4 }, { "x": "Implementation simplicity", "y": 4 }, { "x": "Works without teacher internals", "y": 5 }, { "x": "Low training compute", "y": 2 }, { "x": "Breadth of applicability", "y": 3 } ] }, { "name": "Generalized KD (on-policy)", "data": [ { "x": "Quality retention", "y": 5 }, { "x": "Data efficiency", "y": 5 }, { "x": "Implementation simplicity", "y": 2 }, { "x": "Works without teacher internals", "y": 1 }, { "x": "Low training compute", "y": 2 }, { "x": "Breadth of applicability", "y": 3 } ] }, { "name": "Prune-then-distill", "data": [ { "x": "Quality retention", "y": 4 }, { "x": "Data efficiency", "y": 5 }, { "x": "Implementation simplicity", "y": 1 }, { "x": "Works without teacher internals", "y": 1 }, { "x": "Low training compute", "y": 3 }, { "x": "Breadth of applicability", "y": 3 } ] }, { "name": "Diffusion distillation", "data": [ { "x": "Quality retention", "y": 4 }, { "x": "Data efficiency", "y": 5 }, { "x": "Implementation simplicity", "y": 2 }, { "x": "Works without teacher internals", "y": 1 }, { "x": "Low training compute", "y": 5 }, { "x": "Breadth of applicability", "y": 2 } ] } ], "notes": "Editorial scores; the URLs listed under sources are the papers whose reported trade-offs these scores summarise, not sources for the numbers themselves. Editorial 1-5 ratings, not measured quantities. They summarise the trade-offs argued in each method's cited paper: 'Works without teacher internals' is 5 for black-box methods and 1 for white-box; 'Low training compute' is scored on the marginal cost of the student run, so reasoning-trace distillation scores low because generating long traces from a frontier teacher dominates the bill.", "sources": [ "https://arxiv.org/abs/1503.02531", "https://arxiv.org/abs/2306.13649", "https://arxiv.org/abs/2408.11796", "https://arxiv.org/abs/2310.04378", "https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" ] }, { "id": "methods-per-year", "title": "Distillation methods in this library, by year introduced", "type": "bar", "xLabel": "Year of the originating paper", "yLabel": "Methods introduced", "unit": "methods", "series": [ { "name": "Methods catalogued", "data": [ { "x": "2014", "y": 1 }, { "x": "2015", "y": 1 }, { "x": "2016", "y": 2 }, { "x": "2017", "y": 2 }, { "x": "2018", "y": 2 }, { "x": "2019", "y": 4 }, { "x": "2020", "y": 1 }, { "x": "2021", "y": 1 }, { "x": "2022", "y": 1 }, { "x": "2023", "y": 8 }, { "x": "2024", "y": 3 }, { "x": "2025", "y": 1 } ] } ], "notes": "Counts the 27 entries in extras.methods by the `year` field, which is the year of the originating arXiv preprint (not the conference year — FitNets is 2014 on arXiv, ICLR 2015). The 2023 spike is the instruction-tuning and reasoning-distillation wave; 2024-2025 counts are lower partly because recent work refines these families rather than founding new ones.", "sources": [ "https://arxiv.org/abs/2402.13116", "https://arxiv.org/abs/2503.12067" ] }, { "id": "r1-distill-aime", "title": "Reasoning-trace distillation: AIME 2024 pass@1 by student size", "type": "bar", "xLabel": "DeepSeek-R1-Distill student", "yLabel": "AIME 2024 pass@1", "unit": "%", "series": [ { "name": "AIME 2024 pass@1", "data": [ { "x": "Qwen-1.5B", "y": 28.9 }, { "x": "Llama-8B", "y": 50.4 }, { "x": "Qwen-7B", "y": 55.5 }, { "x": "Qwen-14B", "y": 69.7 }, { "x": "Llama-70B", "y": 70 }, { "x": "Qwen-32B", "y": 72.6 } ] }, { "name": "MATH-500 pass@1", "data": [ { "x": "Qwen-1.5B", "y": 83.9 }, { "x": "Llama-8B", "y": 89.1 }, { "x": "Qwen-7B", "y": 92.8 }, { "x": "Qwen-14B", "y": 93.9 }, { "x": "Llama-70B", "y": 94.5 }, { "x": "Qwen-32B", "y": 94.3 } ] } ], "notes": "All students are plain supervised fine-tunes on 800k reasoning samples curated with DeepSeek-R1 — no reinforcement-learning stage. Figures from the official model card.", "sources": [ "https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" ] }, { "id": "onpolicy-vs-rl", "title": "Off-policy distillation vs RL vs on-policy distillation (Qwen3-8B student)", "type": "bar", "xLabel": "Training method", "yLabel": "AIME'24 pass@1", "unit": "%", "series": [ { "name": "AIME'24", "data": [ { "x": "Off-policy distillation", "y": 55 }, { "x": "Reinforcement learning", "y": 67.6 }, { "x": "On-policy distillation", "y": 74.4 } ] } ], "notes": "The compute is the real story: the RL result cost 17,920 GPU-hours and the on-policy distillation result cost 1,800 GPU-hours, a roughly 10x reduction for a 6.8-point gain. Both figures come from the Qwen3 technical report (Table 21) and are quoted by Thinking Machines; Thinking Machines' own Tinker reproduction started from a different checkpoint and reached roughly 70% AIME'24.", "sources": [ "https://thinkingmachines.ai/blog/on-policy-distillation/", "https://arxiv.org/html/2505.09388v1" ] }, { "id": "reported-speedups", "title": "Reported inference speedup of distilled students over their teachers", "type": "bar", "xLabel": "Student (method)", "yLabel": "Speedup vs teacher", "unit": "x", "series": [ { "name": "Speedup", "data": [ { "x": "DistilBERT (response+cosine)", "y": 1.6 }, { "x": "Llama-3.1-Minitron-4B width", "y": 1.8 }, { "x": "Llama-3.1-Minitron-4B depth", "y": 2.7 }, { "x": "EAGLE draft on Llama2-Chat-70B", "y": 3.1 }, { "x": "TinyBERT-4L", "y": 9.4 }, { "x": "Kim & Rush NMT student", "y": 10 } ] } ], "notes": "DistilBERT is quoted as '60% faster', rendered here as 1.6x. EAGLE is reported as a 2.7x-3.5x latency speedup ratio on LLaMA2-Chat 70B; 3.1 is the midpoint and is the least precise bar here. These numbers come from different hardware, batch sizes and tasks and are not directly comparable to each other — read them as within-paper claims.", "sources": [ "https://arxiv.org/abs/1910.01108", "https://arxiv.org/abs/1909.10351", "https://arxiv.org/abs/2408.11796", "https://arxiv.org/abs/2401.15077", "https://arxiv.org/abs/1606.07947" ] } ], "timeline": [ { "date": "2006-08", "title": "Model Compression (Buciluă, Caruana, Niculescu-Mizil)", "detail": "The pre-history: a large ensemble labels a synthetic unlabeled set, and a single small neural net is trained to mimic those labels. The mechanism is the same one used today for black-box distillation, nine years before the word 'distillation' was attached to it.", "category": "research", "source": "https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf" }, { "date": "2014-12", "title": "FitNets introduces intermediate 'hints'", "detail": "Romero et al. show a thin, deep student can be trained by first matching a teacher's mid-layer activations through a learned regressor, then applying standard KD. This is the birth of feature-based distillation.", "category": "research", "source": "https://arxiv.org/abs/1412.6550" }, { "date": "2015-03-09", "title": "Hinton, Vinyals & Dean publish 'Distilling the Knowledge in a Neural Network'", "detail": "Names the technique, introduces the temperature-softened softmax and the T^2-scaled two-term objective, and argues soft targets carry more information per example than hard labels.", "category": "research", "source": "https://arxiv.org/abs/1503.02531" }, { "date": "2016-06", "title": "Sequence-level KD (Kim & Rush)", "detail": "Distillation escapes classification. Training an NMT student on the teacher's beam-search outputs yields a student 10x faster than its teacher with little BLEU loss, and 13x fewer parameters when combined with pruning at a 0.4 BLEU cost.", "category": "research", "source": "https://arxiv.org/abs/1606.07947" }, { "date": "2016-12", "title": "Attention transfer (Zagoruyko & Komodakis)", "detail": "Instead of matching raw features, match spatial attention maps derived from channel-wise activation energy. Cheaper and more robust than FitNets hints across architectures.", "category": "research", "source": "https://arxiv.org/abs/1612.03928" }, { "date": "2017-06", "title": "Deep Mutual Learning", "detail": "Zhang et al. show a cohort of untrained peers teaching each other outperforms distillation from a fixed, more powerful teacher — the first strong evidence that no pretrained teacher is strictly necessary.", "category": "research", "source": "https://arxiv.org/abs/1706.00384" }, { "date": "2018-05", "title": "Born-Again Neural Networks", "detail": "Furlanello et al. distil a network into an identically-parameterised student and the student beats the teacher, decoupling distillation from compression entirely.", "category": "research", "source": "https://arxiv.org/abs/1805.04770" }, { "date": "2018-11", "title": "Dataset Distillation", "detail": "Wang, Zhu, Torralba and Efros invert the problem: fix the model and compress the dataset into a handful of synthetic examples.", "category": "research", "source": "https://arxiv.org/abs/1811.10959" }, { "date": "2019-02-09", "title": "Teacher-Assistant KD", "detail": "Mirzadeh et al. document the capacity-gap failure — a very large teacher transfers poorly to a very small student — and insert intermediate models to bridge it.", "category": "research", "source": "https://arxiv.org/abs/1902.03393" }, { "date": "2019-04-10", "title": "Relational Knowledge Distillation", "detail": "Park et al. transfer the geometry between examples rather than per-example outputs, using distance-wise and angle-wise losses. In metric learning the students beat their teachers.", "category": "research", "source": "https://arxiv.org/abs/1904.05068" }, { "date": "2019-09", "title": "TinyBERT", "detail": "Layer-wise transformer distillation at both the pretraining and task stages: embedding, attention-matrix, hidden-state and prediction losses. A 4-layer student keeps 96.8% of BERT-base on GLUE at 7.5x smaller and 9.4x faster.", "category": "research", "source": "https://arxiv.org/abs/1909.10351" }, { "date": "2019-10", "title": "DistilBERT ships in the transformers library", "detail": "Hugging Face releases a 40% smaller, 60% faster BERT that keeps 97% of language-understanding performance, trained with a triple loss (CE + masked LM + cosine embedding). Distillation becomes a default deployment step, not a research technique.", "category": "product", "source": "https://arxiv.org/abs/1910.01108" }, { "date": "2019-10-23", "title": "Contrastive Representation Distillation", "detail": "Tian, Krishnan and Isola reframe distillation as maximising mutual information between teacher and student representations, and beat KL-based KD across transfer tasks.", "category": "research", "source": "https://arxiv.org/abs/1910.10699" }, { "date": "2020-10-06", "title": "Margin-MSE brings distillation to retrieval", "detail": "Hofstätter et al. distil a cross-encoder ensemble into a fast bi-encoder by matching score margins, making dense retrieval practical at web scale.", "category": "research", "source": "https://arxiv.org/abs/2010.02666" }, { "date": "2021-12", "title": "Context distillation used for alignment", "detail": "Askell et al. internalise a long few-shot HHH prompt into model weights, showing that a prompt can be compiled into parameters — later formalised by Snell, Klein and Zhong.", "category": "research", "source": "https://arxiv.org/abs/2112.00861" }, { "date": "2022-02-01", "title": "Progressive Distillation for diffusion samplers", "detail": "Salimans and Ho repeatedly halve the number of DDIM steps by training a student to take one step where the teacher took two, opening the whole field of sampler distillation.", "category": "research", "source": "https://arxiv.org/abs/2202.00512" }, { "date": "2023-03-13", "title": "Stanford Alpaca", "detail": "52k instructions generated from text-davinci-003 for under $500, then a 3-hour finetune of LLaMA-7B on 8x A100s for under $100. The template for black-box output distillation — and the start of the API terms-of-service fight.", "category": "product", "source": "https://crfm.stanford.edu/2023/03/13/alpaca.html" }, { "date": "2023-03", "title": "Consistency Models", "detail": "Song, Dhariwal, Chen and Sutskever define models that map noise directly to data in one step, reaching FID 3.55 on CIFAR-10 for one-step generation.", "category": "research", "source": "https://arxiv.org/abs/2303.01469" }, { "date": "2023-05-03", "title": "Distilling Step-by-Step", "detail": "Hsieh et al. add teacher rationales as a second supervised task; a 770M T5 beats a few-shot 540B PaLM using 80% of the available data.", "category": "research", "source": "https://arxiv.org/abs/2305.02301" }, { "date": "2023-05-29", "title": "LLM-QAT", "detail": "Liu et al. show data-free quantization-aware training driven by the model's own generations, quantizing weights, activations and the KV cache.", "category": "research", "source": "https://arxiv.org/abs/2305.17888" }, { "date": "2023-06", "title": "MiniLLM and GKD land within days of each other", "detail": "Two independent groups converge on the same diagnosis — forward KL and off-policy data are wrong for generative students — and on the same fix: sample from the student, score with the teacher, and use a mode-seeking divergence.", "category": "research", "source": "https://arxiv.org/abs/2306.08543" }, { "date": "2023-10-10", "title": "Sheared LLaMA", "detail": "Xia et al. combine targeted structured pruning with dynamic batch loading to prune LLaMA2-7B to 1.3B and 2.7B students that beat same-size models trained from scratch.", "category": "research", "source": "https://arxiv.org/abs/2310.06694" }, { "date": "2023-10", "title": "Latent Consistency Models", "detail": "Luo et al. distil Stable Diffusion into a 2-4 step sampler at 768x768 for 32 A100 GPU-hours.", "category": "research", "source": "https://arxiv.org/abs/2310.04378" }, { "date": "2023-10-25", "title": "Zephyr-7B and dDPO", "detail": "Tunstall et al. distil alignment itself: AI-generated preference pairs plus direct preference optimisation, no human annotation, beating Llama2-Chat-70B on MT-Bench.", "category": "research", "source": "https://arxiv.org/abs/2310.16944" }, { "date": "2023-11", "title": "Orca 2", "detail": "Microsoft teaches small models to choose reasoning strategies rather than merely imitate traces.", "category": "research", "source": "https://arxiv.org/abs/2311.11045" }, { "date": "2023-11", "title": "Adversarial Diffusion Distillation (SDXL-Turbo)", "detail": "Stability AI distils SDXL with a combined adversarial and score-distillation objective and ships SDXL-Turbo, the first real-time single-step foundation image model.", "category": "research", "source": "https://arxiv.org/abs/2311.17042" }, { "date": "2024-01", "title": "EAGLE and Medusa", "detail": "Draft-model distillation becomes standard inference infrastructure: EAGLE drafts autoregressively at the feature level for a 2.7x-3.5x speedup on LLaMA2-Chat 70B, Medusa adds parallel decoding heads with tree attention.", "category": "product", "source": "https://arxiv.org/abs/2401.15077" }, { "date": "2024-02-19", "title": "Universal Logit Distillation", "detail": "Boizard et al. use optimal transport to distil across tokenizers, removing the shared-vocabulary requirement that had confined white-box KD to single model families.", "category": "research", "source": "https://arxiv.org/abs/2402.12030" }, { "date": "2024-07", "title": "NVIDIA Minitron", "detail": "Pruning plus distillation with under 1% of the original token budget (94B vs 15T tokens, ~160x fewer) produces Llama-3.1-Minitron-4B at 1.8x-2.7x speedup, with 'teacher correction' as a named prerequisite step.", "category": "product", "source": "https://arxiv.org/abs/2408.11796" }, { "date": "2024-10-01", "title": "OpenAI ships Model Distillation in the API", "detail": "Stored Completions (free) plus Evals plus fine-tuning become one managed pipeline, so a developer can capture GPT-4-class outputs in production and train a cheaper student on them without leaving the platform. Announced at DevDay on 1 October 2024; the openai.com announcement page blocks non-browser clients, so the OpenAI developer-community announcement thread is listed in sources as a reachable corroborating record.", "category": "product", "source": "https://openai.com/index/api-model-distillation/" }, { "date": "2024-10-15", "title": "Speculative Knowledge Distillation", "detail": "Xu et al. interleave student proposals with teacher corrections, taking the middle path between supervised KD's distribution mismatch and on-policy KD's low-quality samples.", "category": "research", "source": "https://arxiv.org/abs/2410.11325" }, { "date": "2025-01-20", "title": "DeepSeek-R1 and the R1-Distill family", "detail": "Six open-weight students fine-tuned on 800k R1-generated reasoning traces, topping out at 72.6% AIME 2024 pass@1 for the 32B Qwen student, released under MIT for the Qwen-based variants. Reasoning-trace distillation becomes the most-copied recipe of the year.", "category": "product", "source": "https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B" }, { "date": "2025-01-31", "title": "s1: 1,000 examples plus budget forcing", "detail": "Muennighoff et al. show that a curated 1,000-sample trace set distilled from Gemini Thinking, plus forcing the model to keep thinking by appending 'Wait', exceeds o1-preview on competition maths by up to 27%.", "category": "research", "source": "https://arxiv.org/abs/2501.19393" }, { "date": "2025-02", "title": "Distillation Scaling Laws", "detail": "Apple gives the field its first compute-allocation rule: distillation beats supervised learning when a teacher already exists or serves many students, and loses when one teacher must be trained for one student.", "category": "research", "source": "https://arxiv.org/abs/2502.08606" }, { "date": "2025-03", "title": "Gemma 3: every size distilled", "detail": "Google DeepMind reports that all Gemma 3 models are trained with knowledge distillation at both pretraining and instruction-tuning, sampling 256 logits per token weighted by teacher probability and renormalising.", "category": "product", "source": "https://arxiv.org/abs/2503.19786" }, { "date": "2025-05-01", "title": "Amazon Bedrock Model Distillation reaches general availability", "detail": "Managed distillation across Amazon Nova, Anthropic Claude and Meta Llama families, with AWS claiming up to 500% faster and up to 75% cheaper students at under 2% accuracy loss for RAG use cases.", "category": "product", "source": "https://aws-news.com/article/2025-05-01-amazon-bedrock-model-distillation-is-now-generally-available" }, { "date": "2025-05", "title": "Qwen3 documents strong-to-weak distillation", "detail": "Alibaba's report describes a two-phase off-policy then on-policy pipeline for its 0.6B-30B models, achieving better results than reinforcement learning at roughly one tenth of the GPU hours.", "category": "product", "source": "https://arxiv.org/html/2505.09388v1" }, { "date": "2025-09-17", "title": "DeepSeek-R1 published in Nature after peer review", "detail": "The first major open-weight LLM to pass peer review. The published version addresses the distillation allegations directly; Nature's report states that R1's success did not hinge on being trained on the output of its rivals.", "category": "research", "source": "https://www.nature.com/articles/d41586-025-03015-6" }, { "date": "2025-10-27", "title": "Thinking Machines publishes an on-policy distillation recipe", "detail": "A public, reproducible account using the Tinker API. It quotes the Qwen3 technical report's own numbers (Table 21) for a Qwen3-8B student: 55.0 off-policy to 74.4 on AIME'24 at 1,800 GPU-hours, versus 67.6 for RL at 17,920. Thinking Machines' own run uses a Qwen3-8B teacher and a Qwen3-8B-Base SFT-400K student, reaching roughly 70% AIME'24 in about 150 steps. Reverse-KL per-token grading of student rollouts becomes a mainstream recipe.", "category": "research", "source": "https://thinkingmachines.ai/blog/on-policy-distillation/" }, { "date": "2026-02-23", "title": "Anthropic discloses industrial-scale distillation attacks", "detail": "Anthropic reports roughly 24,000 fraudulent accounts and over 16 million exchanges attributed to DeepSeek (150k+), Moonshot AI (3.4M+) and MiniMax (13M+), aimed at agentic reasoning, tool use and coding, and describes classifier and behavioural-fingerprinting defences. Distillation becomes a named category of platform abuse.", "category": "legal", "source": "https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks" } ], "glossary": [ { "term": "Knowledge distillation (KD)", "definition": "Training a student model to reproduce the behaviour of a teacher model, using the teacher's outputs, internal states or generated data as the supervision signal instead of (or alongside) ground-truth labels." }, { "term": "Teacher", "definition": "The model whose behaviour is being copied. Usually larger, slower or more expensive than the student; occasionally the same size (self-distillation) or even a peer (mutual learning)." }, { "term": "Student", "definition": "The model being trained. Its capacity ceiling, not the loss function, is usually what limits how much of the teacher survives the transfer." }, { "term": "Soft targets", "definition": "The teacher's full probability distribution over classes or tokens, as opposed to a one-hot label. Carries information about which wrong answers were nearly right." }, { "term": "Dark knowledge", "definition": "Hinton's informal name for the information in the relative probabilities the teacher assigns to incorrect classes — the part of the teacher's judgement that a hard label throws away." }, { "term": "Temperature (T)", "definition": "A divisor applied to logits before the softmax. Higher T flattens the distribution and exposes small probability differences. Because gradients through the softened softmax scale as 1/T^2, the soft-target loss term is multiplied by T^2." }, { "term": "Forward KL", "definition": "KL(teacher || student). Mass-covering: it punishes the student for assigning low probability anywhere the teacher assigns mass, so an under-capacity student smears probability across modes it cannot represent." }, { "term": "Reverse KL", "definition": "KL(student || teacher). Mode-seeking: it punishes the student only where the student itself puts mass, so a small student concentrates on the teacher's dominant modes instead of hedging. The core of MiniLLM." }, { "term": "Generalized JSD", "definition": "A beta-interpolated Jensen-Shannon divergence used in GKD. Beta = 0 approximates forward KL, beta = 1 approximates reverse KL, and intermediate values trade off between them." }, { "term": "On-policy distillation", "definition": "Sampling training sequences from the student and having the teacher score them token by token, so the student is corrected exactly on the distribution it will actually produce at inference." }, { "term": "Off-policy distillation", "definition": "Training on a fixed corpus of teacher-generated (or human) sequences. Simpler and cheaper, but the student never sees its own mistakes during training." }, { "term": "Exposure bias", "definition": "The compounding error that occurs when an autoregressive model trained only on teacher-forced sequences meets its own generated prefixes at inference. The problem on-policy distillation exists to solve." }, { "term": "White-box distillation", "definition": "Distillation that requires access to teacher internals — logits, hidden states, attention maps or gradients. Only possible when you can run the teacher's weights yourself." }, { "term": "Black-box distillation", "definition": "Distillation from sampled teacher outputs only, as returned by an API. The dominant mode in practice, and the mode governed by API terms of service." }, { "term": "Sequence-level KD", "definition": "Replacing token-level distribution matching with plain maximum-likelihood training on complete sequences generated by the teacher, typically its beam-search output." }, { "term": "Rationale distillation", "definition": "Training the student to reproduce the teacher's reasoning text as an auxiliary target, not just its final answer. Distilling Step-by-Step frames this as a second task in a multi-task objective." }, { "term": "Reasoning-trace distillation", "definition": "Supervised fine-tuning on long chain-of-thought traces from a reasoning teacher, as in the DeepSeek-R1-Distill family (800k samples) and s1 (1,000 samples). No RL stage is required." }, { "term": "Budget forcing", "definition": "An inference-time control from s1 that either cuts off a model's thinking or extends it by appending 'Wait', trading test-time compute for accuracy on a model already distilled on reasoning traces." }, { "term": "Capacity gap", "definition": "The empirical finding that distillation degrades when teacher and student are too far apart in size, motivating teacher-assistant chains and mode-seeking divergences." }, { "term": "Teacher correction", "definition": "Lightly fine-tuning the teacher on the distillation corpus before distilling, so the teacher's distribution matches the transfer data. Named by NVIDIA in the Minitron work and independently confirmed in the torchtune case study." }, { "term": "Hint layer", "definition": "The teacher's intermediate layer chosen as a feature-matching target in FitNets, paired with a learned regressor that maps the narrower student layer into the teacher's dimensionality." }, { "term": "Attention map", "definition": "A spatial energy map derived by summing the absolute powers of a layer's channel activations. Attention transfer matches L2-normalised versions of these between teacher and student." }, { "term": "Relation-based knowledge", "definition": "Structure that lives between examples rather than within one — pairwise distances and triplet angles in embedding space. What RKD transfers." }, { "term": "Cross-tokenizer distillation", "definition": "Distilling between models with different vocabularies. Requires either an optimal-transport cost over sorted probabilities (ULD) or a shared projected output space with cross-model attention (DSKD)." }, { "term": "Dataset distillation", "definition": "Compressing a training set into a small synthetic set that trains a model to comparable accuracy, via bi-level optimisation. The model is fixed; the data is the thing being distilled." }, { "term": "Preference distillation", "definition": "Using a teacher (or an AI judge) to rank candidate completions and training the student on those preferences, via a reward model plus RL (RLAIF) or directly (dDPO)." }, { "term": "Speculative decoding", "definition": "An exact-inference speedup where a cheap draft model proposes several tokens and the target model verifies them in one pass. Distillation enters as the method for training draft heads (EAGLE, Medusa) that get accepted more often." }, { "term": "Progressive distillation", "definition": "Repeatedly halving a diffusion sampler's step count by training a student to take in one step what the teacher takes in two, compounding to large speedups over several rounds." }, { "term": "Consistency model", "definition": "A generative model trained so that all points on a probability-flow ODE trajectory map to the same origin, permitting one-step sampling. Can be distilled from a diffusion teacher or trained standalone." }, { "term": "Context distillation", "definition": "Compiling a prompt — instructions, few-shot examples or scratchpad reasoning — into model weights by training the model without the context to match its own predictions with the context." }, { "term": "Distillation attack", "definition": "Anthropic's term for coordinated, large-scale extraction of a hosted model's outputs across fraudulent accounts in order to train a competing model, as distinct from a legitimate managed distillation workflow." } ], "sources": [ { "title": "Distilling the Knowledge in a Neural Network", "url": "https://arxiv.org/abs/1503.02531", "publisher": "Hinton, Vinyals & Dean (Google)", "date": "2015-03-09", "type": "paper" }, { "title": "Distilling the Knowledge in a Neural Network (author PDF)", "url": "https://www.cs.toronto.edu/~hinton/absps/distillation.pdf", "publisher": "University of Toronto", "date": "2015-03-09", "type": "paper" }, { "title": "FitNets: Hints for Thin Deep Nets", "url": "https://arxiv.org/abs/1412.6550", "publisher": "Romero et al., ICLR 2015", "date": "2014-12", "type": "paper" }, { "title": "Paying More Attention to Attention", "url": "https://arxiv.org/abs/1612.03928", "publisher": "Zagoruyko & Komodakis, ICLR 2017", "date": "2016-12", "type": "paper" }, { "title": "Relational Knowledge Distillation", "url": "https://arxiv.org/abs/1904.05068", "publisher": "Park, Kim, Lu & Cho, CVPR 2019", "date": "2019-04-10", "type": "paper" }, { "title": "Contrastive Representation Distillation", "url": "https://arxiv.org/abs/1910.10699", "publisher": "Tian, Krishnan & Isola, ICLR 2020", "date": "2019-10-23", "type": "paper" }, { "title": "TinyBERT: Distilling BERT for Natural Language Understanding", "url": "https://arxiv.org/abs/1909.10351", "publisher": "Jiao et al., Huawei Noah's Ark Lab", "date": "2019-09", "type": "paper" }, { "title": "DistilBERT, a distilled version of BERT", "url": "https://arxiv.org/abs/1910.01108", "publisher": "Sanh, Debut, Chaumond & Wolf, Hugging Face", "date": "2019-10", "type": "paper" }, { "title": "Sequence-Level Knowledge Distillation", "url": "https://arxiv.org/abs/1606.07947", "publisher": "Kim & Rush, EMNLP 2016", "date": "2016-06", "type": "paper" }, { "title": "Born Again Neural Networks", "url": "https://arxiv.org/abs/1805.04770", "publisher": "Furlanello et al., ICML 2018", "date": "2018-05", "type": "paper" }, { "title": "Improved Knowledge Distillation via Teacher Assistant", "url": "https://arxiv.org/abs/1902.03393", "publisher": "Mirzadeh et al., AAAI 2020", "date": "2019-02-09", "type": "paper" }, { "title": "Deep Mutual Learning", "url": "https://arxiv.org/abs/1706.00384", "publisher": "Zhang, Xiang, Hospedales & Lu, CVPR 2018", "date": "2017-06", "type": "paper" }, { "title": "Knowledge Distillation: A Survey", "url": "https://arxiv.org/abs/2006.05525", "publisher": "Gou, Yu, Maybank & Tao, IJCV", "date": "2020-06", "type": "paper" }, { "title": "A Comprehensive Survey on Knowledge Distillation", "url": "https://arxiv.org/abs/2503.12067", "publisher": "Mansourian et al., Sharif University of Technology", "date": "2025-03-15", "type": "paper" }, { "title": "A Survey on Knowledge Distillation of Large Language Models", "url": "https://arxiv.org/abs/2402.13116", "publisher": "Xu et al.", "date": "2024-02-20", "type": "paper" }, { "title": "Dataset Distillation", "url": "https://arxiv.org/abs/1811.10959", "publisher": "Wang, Zhu, Torralba & Efros", "date": "2018-11", "type": "paper" }, { "title": "Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (Margin-MSE)", "url": "https://arxiv.org/abs/2010.02666", "publisher": "Hofstätter et al., TU Wien", "date": "2020-10-06", "type": "paper" }, { "title": "Learning by Distilling Context", "url": "https://arxiv.org/abs/2209.15189", "publisher": "Snell, Klein & Zhong, UC Berkeley", "date": "2022-09", "type": "paper" }, { "title": "A General Language Assistant as a Laboratory for Alignment", "url": "https://arxiv.org/abs/2112.00861", "publisher": "Askell et al., Anthropic", "date": "2021-12", "type": "paper" }, { "title": "Alpaca: A Strong, Replicable Instruction-Following Model", "url": "https://crfm.stanford.edu/2023/03/13/alpaca.html", "publisher": "Stanford CRFM", "date": "2023-03-13", "type": "blog" }, { "title": "Orca 2: Teaching Small Language Models How to Reason", "url": "https://arxiv.org/abs/2311.11045", "publisher": "Microsoft Research", "date": "2023-11", "type": "paper" }, { "title": "Orca: Progressive Learning from Complex Explanation Traces of GPT-4", "url": "https://arxiv.org/abs/2306.02707", "publisher": "Microsoft Research", "date": "2023-06", "type": "paper" }, { "title": "Distilling Step-by-Step!", "url": "https://arxiv.org/abs/2305.02301", "publisher": "Hsieh et al., Findings of ACL 2023", "date": "2023-05-03", "type": "paper" }, { "title": "MiniLLM: On-Policy Distillation of Large Language Models", "url": "https://arxiv.org/abs/2306.08543", "publisher": "Gu, Dong, Wei & Huang, ICLR 2024", "date": "2023-06", "type": "paper" }, { "title": "On-Policy Distillation of Language Models (GKD)", "url": "https://arxiv.org/abs/2306.13649", "publisher": "Agarwal et al., Google DeepMind, ICLR 2024", "date": "2023-06", "type": "paper" }, { "title": "Speculative Knowledge Distillation", "url": "https://arxiv.org/abs/2410.11325", "publisher": "Xu et al., ICLR 2025", "date": "2024-10-15", "type": "paper" }, { "title": "Zephyr: Direct Distillation of LM Alignment", "url": "https://arxiv.org/abs/2310.16944", "publisher": "Tunstall et al., Hugging Face", "date": "2023-10-25", "type": "paper" }, { "title": "Direct Preference Optimization", "url": "https://arxiv.org/abs/2305.18290", "publisher": "Rafailov et al., NeurIPS 2023", "date": "2023-05", "type": "paper" }, { "title": "Constitutional AI: Harmlessness from AI Feedback", "url": "https://arxiv.org/abs/2212.08073", "publisher": "Anthropic", "date": "2022-12", "type": "paper" }, { "title": "Compact Language Models via Pruning and Knowledge Distillation", "url": "https://arxiv.org/abs/2407.14679", "publisher": "NVIDIA, NeurIPS 2024", "date": "2024-07", "type": "paper" }, { "title": "LLM Pruning and Distillation in Practice: The Minitron Approach", "url": "https://arxiv.org/abs/2408.11796", "publisher": "NVIDIA", "date": "2024-08", "type": "paper" }, { "title": "Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning", "url": "https://arxiv.org/abs/2310.06694", "publisher": "Xia, Gao, Zeng & Chen, Princeton, ICLR 2024", "date": "2023-10-10", "type": "paper" }, { "title": "LLM-QAT: Data-Free Quantization Aware Training for Large Language Models", "url": "https://arxiv.org/abs/2305.17888", "publisher": "Liu et al., Meta", "date": "2023-05-29", "type": "paper" }, { "title": "Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss", "url": "https://arxiv.org/abs/2402.12030", "publisher": "Boizard et al.", "date": "2024-02-19", "type": "paper" }, { "title": "Dual-Space Knowledge Distillation for Large Language Models", "url": "https://arxiv.org/abs/2406.17328", "publisher": "Zhang et al., EMNLP 2024", "date": "2024-06", "type": "paper" }, { "title": "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty", "url": "https://arxiv.org/abs/2401.15077", "publisher": "Li, Wei, Zhang & Zhang, ICML 2024", "date": "2024-01", "type": "paper" }, { "title": "EAGLE-3: Scaling up Inference Acceleration via Training-Time Test", "url": "https://arxiv.org/abs/2503.01840", "publisher": "Li et al.", "date": "2025-03", "type": "paper" }, { "title": "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", "url": "https://arxiv.org/abs/2401.10774", "publisher": "Cai et al.", "date": "2024-01", "type": "paper" }, { "title": "Progressive Distillation for Fast Sampling of Diffusion Models", "url": "https://arxiv.org/abs/2202.00512", "publisher": "Salimans & Ho, Google, ICLR 2022", "date": "2022-02-01", "type": "paper" }, { "title": "Consistency Models", "url": "https://arxiv.org/abs/2303.01469", "publisher": "Song, Dhariwal, Chen & Sutskever, OpenAI, ICML 2023", "date": "2023-03", "type": "paper" }, { "title": "Latent Consistency Models", "url": "https://arxiv.org/abs/2310.04378", "publisher": "Luo et al., Tsinghua", "date": "2023-10", "type": "paper" }, { "title": "Adversarial Diffusion Distillation (SDXL-Turbo)", "url": "https://arxiv.org/abs/2311.17042", "publisher": "Sauer, Lorenz, Blattmann & Rombach, Stability AI", "date": "2023-11", "type": "paper" }, { "title": "s1: Simple test-time scaling", "url": "https://arxiv.org/abs/2501.19393", "publisher": "Muennighoff et al., Stanford", "date": "2025-01-31", "type": "paper" }, { "title": "Distillation Scaling Laws", "url": "https://arxiv.org/abs/2502.08606", "publisher": "Busbridge et al., Apple, ICML 2025", "date": "2025-02", "type": "paper" }, { "title": "DeepSeek-R1-Distill-Qwen-32B model card", "url": "https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B", "publisher": "DeepSeek-AI on Hugging Face", "date": "2025-01", "type": "docs" }, { "title": "Secrets of DeepSeek AI model revealed in landmark paper", "url": "https://www.nature.com/articles/d41586-025-03015-6", "publisher": "Nature", "date": "2025-09-17", "type": "news" }, { "title": "Qwen3 Technical Report", "url": "https://arxiv.org/html/2505.09388v1", "publisher": "Qwen Team, Alibaba", "date": "2025-05", "type": "paper" }, { "title": "Gemma 3 Technical Report", "url": "https://arxiv.org/abs/2503.19786", "publisher": "Google DeepMind", "date": "2025-03", "type": "paper" }, { "title": "On-Policy Distillation", "url": "https://thinkingmachines.ai/blog/on-policy-distillation/", "publisher": "Thinking Machines Lab", "date": "2025-10-27", "type": "blog" }, { "title": "Distilling Llama3.1 8B into 1B in torchtune", "url": "https://pytorch.org/blog/llama-into-torchtune/", "publisher": "PyTorch", "date": "2025-02", "type": "blog" }, { "title": "TRL Generalized Knowledge Distillation Trainer documentation", "url": "https://huggingface.co/docs/trl/gkd_trainer", "publisher": "Hugging Face (TRL v1.12.0)", "date": "2026", "type": "docs" }, { "title": "TRL MiniLLM Trainer documentation", "url": "https://huggingface.co/docs/trl/main/minillm", "publisher": "Hugging Face (TRL v1.12.0)", "date": "2026", "type": "docs" }, { "title": "DistillKit: An Open Source Toolkit For LLM Distillation", "url": "https://github.com/arcee-ai/DistillKit", "publisher": "Arcee AI", "date": "2024-08", "type": "docs" }, { "title": "Model Distillation in the API", "url": "https://openai.com/index/api-model-distillation/", "publisher": "OpenAI", "date": "2024-10-01", "type": "blog" }, { "title": "Amazon Bedrock Model Distillation is now generally available", "url": "https://aws-news.com/article/2025-05-01-amazon-bedrock-model-distillation-is-now-generally-available", "publisher": "AWS News", "date": "2025-05-01", "type": "news" }, { "title": "Amazon Bedrock Model Distillation: boost function calling accuracy while reducing cost and latency", "url": "https://aws.amazon.com/blogs/machine-learning/amazon-bedrock-model-distillation-boost-function-calling-accuracy-while-reducing-cost-and-latency/", "publisher": "Amazon Web Services", "date": "2025", "type": "blog" }, { "title": "Customize a model with distillation in Amazon Bedrock", "url": "https://docs.aws.amazon.com/bedrock/latest/userguide/model-distillation.html", "publisher": "AWS Documentation", "date": "2025", "type": "docs" }, { "title": "Detecting and preventing distillation attacks", "url": "https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks", "publisher": "Anthropic", "date": "2026-02-23", "type": "blog" }, { "title": "Anthropic accuses DeepSeek, Moonshot and MiniMax of distillation attacks", "url": "https://www.cnbc.com/2026/02/24/anthropic-openai-china-firms-distillation-deepseek.html", "publisher": "CNBC", "date": "2026-02-24", "type": "news" }, { "title": "Model Compression (KDD 2006)", "url": "https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf", "publisher": "Buciluă, Caruana & Niculescu-Mizil, Cornell", "date": "2006-08", "type": "paper" }, { "title": "RepDistiller reference implementations", "url": "https://github.com/HobbitLong/RepDistiller", "publisher": "Yonglong Tian", "date": "2019-10", "type": "docs" }, { "title": "attention-transfer reference implementation", "url": "https://github.com/szagoruyko/attention-transfer", "publisher": "Sergey Zagoruyko", "date": "2017", "type": "docs" }, { "title": "DSKD reference implementation", "url": "https://github.com/songmzhang/DSKD", "publisher": "Songming Zhang", "date": "2024-06", "type": "docs" }, { "title": "torchtune: PyTorch native post-training library", "url": "https://github.com/meta-pytorch/torchtune", "publisher": "Meta / PyTorch", "date": "2025", "type": "docs" }, { "title": "Efficient Knowledge Distillation from an Ensemble of Teachers", "url": "https://www.isca-archive.org/interspeech_2017/fukuda17_interspeech.html", "publisher": "Fukuda, Suzuki, Kurata, Thomas, Cui & Ramabhadran, Interspeech 2017, pp. 3697-3701", "date": "2017-08", "type": "paper" }, { "title": "Model Distillation, including Evals and Stored Completions", "url": "https://community.openai.com/t/model-distillation-including-evals-and-stored-completions/964021", "publisher": "OpenAI Developer Community", "date": "2024-10-01", "type": "news" } ], "extras": { "methods": [ { "id": "response-kd", "name": "Response-based KD (soft targets with temperature)", "family": "Response-based", "year": 2015, "paper": "Distilling the Knowledge in a Neural Network", "url": "https://arxiv.org/abs/1503.02531", "description": "The founding method. Train the student to match the teacher's temperature-softened output distribution, optionally blended with ordinary cross-entropy against ground-truth labels.", "howItWorks": "Start with what a hard label throws away. A well-trained image classifier shown a handwritten 2 does not output a clean one-hot vector; it might assign 0.9 to '2', 0.05 to '7', 0.04 to '3' and roughly nothing to '5'. Those small numbers are a compressed statement about the shape of the input and about how the teacher's learned feature space is organised. Hinton, Vinyals and Dean called this the *dark knowledge*, and the whole method is machinery for making it visible and usable.\n\nThe problem is that after a confident softmax, those informative numbers are numerically tiny — 0.05 versus 0.0001 is a huge ratio but a negligible difference in the loss. The fix is a temperature parameter T inserted into the softmax: divide every logit by T before exponentiating. At T = 1 you recover the normal distribution; at T = 5 or T = 10 the distribution flattens and the ratios between wrong-class probabilities become large enough to produce real gradients. Crucially you apply the same T to the student, so the two softened distributions are compared on equal footing.\n\nThe objective is a weighted sum of two terms. The first is KL divergence (equivalently, cross-entropy) between the softened teacher and softened student distributions. The second is ordinary cross-entropy between the student's T = 1 output and the true label, which anchors the student when the teacher is wrong. The paper notes a subtlety that trips up most reimplementations: gradients through the softened softmax scale as 1/T², so the soft-target term must be multiplied by T² or its contribution silently vanishes as you raise the temperature.\n\nA worked intuition for why this beats label training: imagine training a small net on 1,000 MNIST digits. With hard labels each example gives you log2(10) ≈ 3.3 bits of supervision. With a soft target you get a full 10-dimensional distribution, and the differences between examples of the same class — this 2 is 7-ish, that 2 is 3-ish — give the student a much richer, lower-variance gradient signal. The paper's most striking demonstration is that a distilled net can learn to recognise a digit class whose examples were entirely removed from its transfer set, purely from the shape of the teacher's soft targets on other digits.\n\nIn the LLM era the exact same objective operates per token position over the vocabulary, which is what torchtune's `ForwardKLWithChunkedOutputLoss` and every logit-matching trainer implement. The two failure modes to know about: forward KL is mass-covering, so an under-capacity student smears probability over modes it cannot represent (the motivation for MiniLLM's reverse KL), and the teacher must share the student's tokenizer (the motivation for ULD and DSKD).", "lossFormula": "\\mathcal{L} = (1-\\alpha)\\,\\mathcal{H}\\big(y,\\ \\sigma(z_s)\\big) \\;+\\; \\alpha\\,T^{2}\\,\\mathrm{KL}\\big(\\sigma(z_t/T)\\ \\|\\ \\sigma(z_s/T)\\big),\\qquad \\sigma(z)_i = \\frac{\\exp(z_i/T)}{\\sum_j \\exp(z_j/T)}", "pros": [ "Simplest possible implementation: one extra forward pass and two lines of loss code", "Teacher logits can be precomputed and cached, making the marginal training cost near zero", "Architecture-agnostic — the student need share nothing with the teacher but the output space", "Works with or without ground-truth labels, so it turns unlabelled data into training signal", "Well-understood regularisation effect: soft targets reduce gradient variance and act like label smoothing with structure" ], "cons": [ "Requires teacher logits, so it is unavailable against any closed API", "Requires an identical vocabulary or class set", "Forward KL is mass-covering: a small student hedges across teacher modes instead of committing, which is exactly wrong for text generation", "Off-policy — the student is never corrected on its own generated prefixes", "Temperature and alpha need tuning per task and interact with the T² correction in ways that are easy to get wrong" ], "whenToUse": "Use it first whenever you host both models and they share a tokenizer or label set. It is the right baseline for classifiers, encoders, and same-family LLM pairs (Llama 3.1 8B into Llama 3.2 1B), and the right first thing to try before reaching for on-policy methods.", "tools": [ "torchtune knowledge_distillation recipes", "Hugging Face TRL", "Arcee DistillKit (logit-based mode)", "NVIDIA TensorRT Model Optimizer", "PyTorch (hand-rolled, ~10 lines)" ], "examples": [ "DistilBERT (with an added cosine-embedding term)", "Gemma 3 pretraining, sampling 256 logits per token weighted by teacher probability", "torchtune's Llama 3.1 8B -> Llama 3.2 1B case study" ], "relatedMethods": [ "feature-kd", "reverse-kl-minillm", "generalized-kd", "self-distillation", "cross-vocabulary-kd" ], "difficulty": 1, "dataNeeded": "logits", "teacherAccess": "white-box" }, { "id": "feature-kd", "name": "Feature / intermediate-layer KD (FitNets hints)", "family": "Feature-based", "year": 2014, "paper": "FitNets: Hints for Thin Deep Nets", "url": "https://arxiv.org/abs/1412.6550", "description": "Supervise a student's hidden layer directly against a teacher's hidden layer, using a learned projector to bridge the width mismatch, then finish with ordinary response-based KD.", "howItWorks": "Response-based KD only constrains the student at the very end of the network. For a student that is much *deeper* than it is wide — the shape FitNets set out to train — that is far too little supervision arriving far too late: gradients from a single output-layer loss have to propagate through many thin layers, and optimisation stalls. Romero et al.'s answer is to plant an additional supervision point in the middle.\n\nPick a layer in the teacher (the *hint* layer) and a layer in the student (the *guided* layer). Because the student is narrower, its activation tensor has fewer channels than the teacher's, so you cannot compare them directly. FitNets inserts a learned regressor — in practice a 1x1 convolution — that maps the student's guided-layer output into the teacher's hint-layer dimensionality. The first training stage minimises the squared error between the projected student features and the teacher features, updating only the student's layers up to the guided layer plus the regressor. The second stage discards the regressor and trains the whole student with the standard Hinton KD objective.\n\nThe intuition is a curriculum. Stage one gives the student's early half a well-posed, dense target — 'produce a representation from which the teacher's mid-level representation is linearly predictable' — which is a far easier optimisation problem than 'produce a representation from which the final answer is predictable after ten more layers'. Once the bottom half is in a good basin, stage two only has to fit the top half and refine.\n\nA worked example: teacher is a wide 5-layer CNN with 128 channels at layer 3; student is a thin 11-layer CNN with 32 channels at layer 6. You add a 1x1 conv mapping 32 -> 128 channels, minimise MSE between the projected student layer-6 map and the teacher layer-3 map, then throw the conv away and run KD. Romero et al. reported students with roughly 10x fewer parameters beating the teacher on CIFAR-10.\n\nThe fragile part is layer choice. There is no principled way to decide which teacher layer corresponds to which student layer, and a bad pairing forces the student to imitate representations it structurally cannot hold, hurting final accuracy. This sensitivity is precisely what attention transfer (match cheap summary statistics instead of raw features) and RKD (match relations instead of absolute vectors) were invented to sidestep.", "lossFormula": "\\mathcal{L}_{\\text{hint}} = \\tfrac{1}{2}\\Big\\| \\, r\\big(F_s^{\\,h};\\,W_r\\big) \\;-\\; F_t^{\\,g} \\Big\\|_2^2 \\qquad\\text{then}\\qquad \\mathcal{L}_{\\text{KD}} = (1-\\alpha)\\mathcal{H}(y,\\sigma(z_s)) + \\alpha T^2\\mathrm{KL}(\\sigma(z_t/T)\\|\\sigma(z_s/T))", "pros": [ "Gives dense supervision deep inside the student, which unlocks thin-and-deep architectures that pure logit KD cannot train", "Substantially more information transferred than logits alone — features are high-dimensional", "Composes cleanly with response-based KD as a two-stage schedule", "Effective when teacher and student share an inductive bias (both CNNs, both transformers)" ], "cons": [ "Requires full white-box access to activations, not just logits", "Layer-pairing is a hyperparameter with no good default and large effect", "The projector adds parameters and its own optimisation dynamics", "Poor across architecture families — forcing a transformer student to match CNN features rarely helps", "Raw-feature MSE over-constrains: it demands the student reproduce the teacher's coordinate system, not just its information" ], "whenToUse": "Use when the student is much deeper or narrower than the teacher and logit-only KD is failing to converge, and when teacher and student are architecturally similar enough that a layer correspondence is meaningful. In transformer land, prefer the structured version (TinyBERT) over hand-picked hint layers.", "tools": [ "RepDistiller", "Arcee DistillKit (hidden-states mode)", "torchvision + custom hooks", "NVIDIA Model Optimizer (intermediate-state distillation)" ], "examples": [ "FitNets on CIFAR-10/100 and SVHN", "TinyBERT's hidden-state loss", "Minitron's intermediate-state distillation during pruning recovery" ], "relatedMethods": [ "response-kd", "attention-transfer", "relation-kd", "transformer-layer-kd", "contrastive-rep-kd" ], "difficulty": 3, "dataNeeded": "features", "teacherAccess": "white-box" }, { "id": "attention-transfer", "name": "Attention transfer", "family": "Attention-based", "year": 2016, "paper": "Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer", "url": "https://arxiv.org/abs/1612.03928", "description": "Instead of matching full activation tensors, match a cheap 2-D summary of where each layer is 'looking' — the channel-collapsed activation energy map — after L2 normalisation.", "howItWorks": "FitNets asks the student to reproduce a teacher's feature tensor exactly, which is a much stronger demand than necessary: two networks can encode the same information in permuted or rotated coordinates. Zagoruyko and Komodakis observed that what you actually want to transfer is *where* in the input the teacher is allocating capacity, and that this is captured by a much smaller object.\n\nTake a layer's activation tensor A with C channels and spatial size H x W. Collapse the channel dimension by summing the elementwise absolute values raised to a power p: Q = Σ_i |A_i|^p, giving an H x W map. With p = 2 this is the per-position activation energy. High values mark the regions the network's filters are responding to strongly. Flatten and L2-normalise this map, do the same for the corresponding student layer, and minimise the distance between them. Because both maps are normalised and channel-free, the student is free to organise its channels however it likes as long as its spatial focus matches.\n\nYou apply this at several depths, typically at the end of each residual-block group, so the student inherits the teacher's coarse-to-fine attention progression. The full objective is the ordinary task loss (plus optionally KD) with the attention terms added at a small weight.\n\nWorked intuition: a teacher classifying a dog image will have high activation energy over the muzzle and ears at mid depth. A student trained only on labels might latch onto grass texture and still get many training images right. The attention term makes the grass shortcut expensive, because the teacher's map has no energy there. You are transferring an inductive prior about *what to attend to*, which is far more architecture-portable than 'produce these exact 256 numbers'.\n\nThe transformer analogue is direct and is where the idea now does most of its work: match the teacher's attention probability matrices head-by-head, which is one of the four losses in TinyBERT. The caveat is that attention maps are a lossy summary — they say where, not what — so attention transfer alone typically gives a smaller gain than a well-tuned feature or logit method, and is usually stacked rather than used alone.", "lossFormula": "\\mathcal{L}_{AT} = \\sum_{j\\in\\mathcal{I}} \\left\\| \\frac{Q_s^{\\,j}}{\\|Q_s^{\\,j}\\|_2} - \\frac{Q_t^{\\,j}}{\\|Q_t^{\\,j}\\|_2} \\right\\|_p ,\\qquad Q^{\\,j} = \\mathrm{vec}\\Big(\\sum_{i=1}^{C} |A_i^{\\,j}|^{\\,p}\\Big)", "pros": [ "Channel-count agnostic, so no projector is needed and width mismatch is free", "Cheap: the summary map is H x W, orders of magnitude smaller than the activation tensor", "More robust to architecture differences than raw-feature matching", "Consistent gains reported across several datasets and CNN families", "Transfers an interpretable quantity — you can visualise exactly what is being copied" ], "cons": [ "Lossy: discards all channel-identity information", "Weaker standalone effect than logit or feature KD; usually needs to be combined", "Still white-box, still needs a layer correspondence (though a coarser one)", "The power p and the layer-group set are extra hyperparameters", "For transformers, per-head attention matching can force spurious head alignment when head counts differ" ], "whenToUse": "Use as an add-on term when teacher and student are both convolutional or both transformer but differ in width, and when FitNets-style feature matching is proving brittle. In NLP, use it as one component of a TinyBERT-style multi-loss recipe rather than on its own.", "tools": [ "szagoruyko/attention-transfer", "RepDistiller", "TinyBERT (attention-matrix MSE)", "custom forward hooks" ], "examples": [ "Wide ResNet teacher to thin ResNet student on CIFAR/ImageNet", "TinyBERT attention distillation", "Cross-modal attention transfer in video models" ], "relatedMethods": [ "feature-kd", "transformer-layer-kd", "relation-kd", "response-kd" ], "difficulty": 2, "dataNeeded": "features", "teacherAccess": "white-box" }, { "id": "relation-kd", "name": "Relational KD (RKD)", "family": "Relation-based", "year": 2019, "paper": "Relational Knowledge Distillation", "url": "https://arxiv.org/abs/1904.05068", "description": "Transfer the geometry *between* examples — pairwise distances and triplet angles in embedding space — rather than the representation of any single example.", "howItWorks": "Every method so far is what Park, Kim, Lu and Cho call *individual* knowledge distillation: for each input, the student is told what the teacher produced for that input. RKD's premise is that the useful structure is relational. What matters is not that a particular photo of a golden retriever maps to a particular 512-dimensional vector, but that it lands close to other retrievers, further from huskies, and much further from cars. That relational structure is invariant to any rotation or rescaling of the embedding space, so a student can honour it without being forced into the teacher's arbitrary coordinate frame.\n\nRKD instantiates this with two losses. The **distance-wise** loss takes every pair of examples in the mini-batch, computes the Euclidean distance between their teacher embeddings and between their student embeddings, normalises each by the mean pairwise distance in that batch (so absolute scale does not matter), and penalises the difference with a Huber loss. The **angle-wise** loss takes every triplet, computes the cosine of the angle at the middle example in the teacher's space and in the student's space, and penalises the difference. Angles are a higher-order relation than distances and empirically transfer more.\n\nA worked intuition: suppose the teacher's embedding puts three images at the corners of a long thin triangle. A student with half the dimensionality cannot place points at the teacher's exact coordinates. But it can absolutely reproduce 'these two are close, the third is far, and the angle at the near pair is acute' — and for retrieval, re-identification or any downstream nearest-neighbour use, that is the only thing that was ever being used.\n\nThe cost is combinatorial: distance terms are O(B²) per batch and angle terms are O(B³), so large batches get expensive and small batches give noisy relation estimates. The payoff is the headline result of the paper: in metric learning, RKD students *outperform their teachers* on standard benchmarks, which individual-KD methods essentially never do. That is the signature of a method transferring something more robust than the teacher's own point estimates.\n\nRKD is also the natural choice when teacher and student embedding dimensionalities differ and you refuse to add a projector, since relations are dimension-free.", "lossFormula": "\\mathcal{L}_{\\text{RKD-D}} = \\sum_{(i,j)} l_{\\delta}\\Big(\\psi_D(t_i,t_j),\\ \\psi_D(s_i,s_j)\\Big),\\quad \\psi_D(a,b) = \\frac{\\|a-b\\|_2}{\\mu} \\qquad\\text{and}\\qquad \\mathcal{L}_{\\text{RKD-A}} = \\sum_{(i,j,k)} l_{\\delta}\\Big(\\psi_A(t_i,t_j,t_k),\\ \\psi_A(s_i,s_j,s_k)\\Big),\\quad \\psi_A = \\cos\\angle\\, t_i t_j t_k", "pros": [ "Invariant to rotation, reflection and scaling of the embedding space — no coordinate-frame tax", "Works when teacher and student embedding dimensions differ, with no projector", "Students can exceed their teachers in metric-learning settings", "Directly optimises the property that retrieval and clustering actually consume", "Composable with logit KD as an extra term" ], "cons": [ "O(B²) and O(B³) terms make large batches costly and small batches statistically noisy", "Needs a meaningful embedding layer, so it does not apply to bare classifiers without one", "Two loss weights to balance against each other and against the task loss", "Less effective than logit KD for plain classification, where absolute class scores are the target", "Still white-box" ], "whenToUse": "Use for embedding models: image retrieval, face and person re-identification, recommendation towers, and sentence encoders — anywhere the deployed operation is a nearest-neighbour search rather than an argmax over classes.", "tools": [ "RepDistiller", "sentence-transformers (custom losses)", "PyTorch Metric Learning" ], "examples": [ "Metric learning on CUB-200, Cars-196 and Stanford Online Products", "Face-recognition backbone compression", "Embedding-tower distillation in retrieval stacks" ], "relatedMethods": [ "contrastive-rep-kd", "feature-kd", "retriever-distillation", "attention-transfer" ], "difficulty": 3, "dataNeeded": "features", "teacherAccess": "white-box" }, { "id": "contrastive-rep-kd", "name": "Contrastive representation distillation (CRD)", "family": "Relation-based", "year": 2019, "paper": "Contrastive Representation Distillation", "url": "https://arxiv.org/abs/1910.10699", "description": "Reframe distillation as maximising the mutual information between teacher and student representations, optimised with a contrastive (InfoNCE-style) objective over positive and negative pairs.", "howItWorks": "Tian, Krishnan and Isola start from a criticism of KL-based distillation: minimising KL between output distributions treats each dimension independently and therefore ignores structural, higher-order dependencies in the teacher's representation. Their alternative is information-theoretic — maximise I(T; S), the mutual information between the teacher's and student's representations of the same input.\n\nMutual information is not directly computable, so CRD maximises a tractable lower bound using a contrastive setup. Build a *positive* pair by taking the teacher and student representations of the same input. Build many *negative* pairs by pairing the teacher representation of one input with the student representation of a different input. Train a critic to distinguish positives from negatives; the value of that discrimination problem lower-bounds the mutual information, and backpropagating it into the student pushes its representation to be maximally predictive of the teacher's.\n\nThe practical machinery is a memory buffer holding thousands of representations so you can draw many negatives per positive without recomputing them, exactly as in contrastive self-supervised learning. The bound tightens with the number of negatives N, which is why CRD uses large N.\n\nWorked intuition for why this beats KL: consider a teacher whose penultimate layer encodes 'object is furry' in one direction and 'object is metallic' in another, and suppose these two attributes are anti-correlated in the data. KL on the output logits can be satisfied by a student that gets the marginal class probabilities right while scrambling the internal correlation. The contrastive objective cannot be satisfied that way — to tell 'this student vector goes with that teacher vector' apart from 200 impostors, the student must preserve the joint structure, correlations included.\n\nCRD is one of the few methods that transfers cleanly across *modalities* (distilling a depth network into an RGB network, say), because mutual information does not care what the inputs are. The paper reports state of the art on model compression, ensemble distillation and cross-modal transfer, sometimes beating the teacher when combined with plain KD. The cost is real: a memory bank, a critic network, negative-sampling hyperparameters, and a heavier training loop than any of the simpler feature methods.", "lossFormula": "\\mathcal{L}_{\\text{CRD}} = -\\,\\mathbb{E}_{q(T,S\\mid C=1)}\\big[\\log h(T,S)\\big] \\;-\\; N\\,\\mathbb{E}_{q(T,S\\mid C=0)}\\big[\\log\\!\\big(1-h(T,S)\\big)\\big],\\qquad h(T,S)=\\frac{e^{\\,g_T(T)^\\top g_S(S)/\\tau}}{e^{\\,g_T(T)^\\top g_S(S)/\\tau} + \\tfrac{N}{M}}", "pros": [ "Captures higher-order structure and correlations that per-dimension KL discards", "Principled: an explicit lower bound on mutual information", "Works cross-modally and for ensemble distillation, not just same-modality compression", "Strong empirical results; frequently the best single feature-family method in benchmark sweeps", "Stacks with response-based KD for further gains" ], "cons": [ "Heaviest implementation in the feature family: memory bank, critic, projection heads", "Sensitive to the number of negatives and the temperature of the critic", "Extra memory footprint during training", "Gains over simpler methods are modest relative to the added complexity for straightforward compression tasks", "White-box only" ], "whenToUse": "Use when you have exhausted logit and simple feature methods and the representation quality itself is the deliverable — cross-modal transfer, ensemble compression, or a backbone that will feed many downstream heads.", "tools": [ "HobbitLong/RepDistiller", "PyTorch (custom InfoNCE + memory bank)" ], "examples": [ "CIFAR-100 and ImageNet compression benchmarks", "RGB-to-depth cross-modal transfer", "Ensemble-into-single-model distillation" ], "relatedMethods": [ "relation-kd", "feature-kd", "response-kd", "retriever-distillation" ], "difficulty": 4, "dataNeeded": "features", "teacherAccess": "white-box" }, { "id": "transformer-layer-kd", "name": "Transformer-layer KD (TinyBERT / DistilBERT)", "family": "Feature-based", "year": 2019, "paper": "TinyBERT: Distilling BERT for Natural Language Understanding", "url": "https://arxiv.org/abs/1909.10351", "description": "The standard recipe for compressing encoder transformers: simultaneously match embeddings, attention matrices, hidden states and prediction logits, applied at both the pretraining and task-specific stages.", "howItWorks": "This is the method that turned distillation from a research technique into a deployment default. Jiao et al. define a layer mapping g(m) that assigns each student layer m to a teacher layer, then impose four losses at once. The **embedding loss** is an MSE between the student's embedding output (projected up to the teacher's width) and the teacher's. The **attention loss** is an MSE between the student's and teacher's attention score matrices, head by head — the paper argues these matrices encode substantial linguistic structure such as coreference and syntax. The **hidden-state loss** is an MSE between projected student hidden states and teacher hidden states at the mapped layer. The **prediction loss** is ordinary soft-target cross-entropy on the logits.\n\nThe second contribution is the two-stage schedule. *General distillation* runs the layer losses on a large unlabelled corpus, producing a general-purpose small encoder. *Task-specific distillation* then repeats the procedure on an augmented version of the downstream dataset, so the student picks up the teacher's task-adapted behaviour. Skipping either stage costs measurable accuracy.\n\nDistilBERT is the leaner sibling: a 6-layer student initialised from every other layer of BERT-base, trained on a triple loss of masked-language-modelling cross-entropy, soft-target distillation, and a cosine-embedding term that aligns hidden-state *directions* rather than exact values. It keeps 97% of BERT-base's GLUE performance at 40% fewer parameters and 60% faster inference. TinyBERT-4L goes further — 96.8% of BERT-base on GLUE while being 7.5x smaller and 9.4x faster — and beats other 4-layer distillation baselines with roughly 28% of their parameters.\n\nWorked intuition on why so many losses: an encoder's usefulness lives in intermediate representations that will be consumed by an arbitrary downstream head, so matching only the final logits under-determines the student. Each additional loss constrains a different part of the computation — what tokens mean (embeddings), what attends to what (attention), what the running state is (hidden), and what the answer is (logits) — and empirically the attention loss matters most.\n\nThe cost is a genuinely fiddly recipe: layer mapping, per-loss weights, projector matrices, two training stages, and data augmentation. Modern decoder-only LLM distillation has largely dropped the intermediate losses in favour of on-policy logit methods, but this recipe remains the best-understood path for encoder compression, which is still where most production embedding and classification models live.", "lossFormula": "\\mathcal{L} = \\sum_{m} \\lambda_m \\Big[ \\underbrace{\\mathrm{MSE}\\big(A_s^{m},\\,A_t^{\\,g(m)}\\big)}_{\\text{attention}} + \\underbrace{\\mathrm{MSE}\\big(H_s^{m}W_h,\\,H_t^{\\,g(m)}\\big)}_{\\text{hidden}} \\Big] \\;+\\; \\underbrace{\\mathrm{MSE}\\big(E_sW_e,\\,E_t\\big)}_{\\text{embedding}} \\;+\\; \\underbrace{\\mathrm{CE}\\big(\\sigma(z_t/T),\\,\\sigma(z_s/T)\\big)}_{\\text{prediction}}", "pros": [ "Best-validated recipe for encoder compression, with a decade of production use behind it", "Very large speedups at small accuracy cost: 9.4x faster at 96.8% of GLUE for TinyBERT-4L", "Attention matching transfers linguistic structure that logits alone do not", "Two-stage schedule yields a reusable general student plus a task-tuned one", "Reference implementations and pretrained students are widely available" ], "cons": [ "Many interacting hyperparameters: layer mapping, four loss weights, two projectors, two stages", "Requires a large unlabelled corpus for the general stage", "Attention matching assumes comparable head structure between teacher and student", "Less used for modern decoder-only LLMs, where on-policy logit methods dominate", "Expensive: two full distillation passes plus data augmentation" ], "whenToUse": "Use for encoder models you will deploy at scale — classification, NER, reranking, embeddings — where latency and cost matter and you can afford a two-stage training pipeline. For a quicker win, take DistilBERT's simpler triple loss instead.", "tools": [ "huawei-noah/TinyBERT", "Hugging Face transformers (DistilBERT)", "Arcee DistillKit (hidden-states mode)", "Sentence Transformers" ], "examples": [ "TinyBERT-4L and -6L on GLUE", "DistilBERT, DistilRoBERTa, DistilGPT-2", "MobileBERT and MiniLM as descendants of the same recipe" ], "relatedMethods": [ "feature-kd", "attention-transfer", "response-kd", "retriever-distillation", "prune-then-distill" ], "difficulty": 4, "dataNeeded": "features", "teacherAccess": "white-box" }, { "id": "sequence-level-kd", "name": "Sequence-level KD (Kim & Rush)", "family": "Sequence-level / data", "year": 2016, "paper": "Sequence-Level Knowledge Distillation", "url": "https://arxiv.org/abs/1606.07947", "description": "Replace token-level distribution matching with plain maximum-likelihood training on complete sequences generated by the teacher, typically its beam-search output.", "howItWorks": "Word-level distillation applies Hinton's objective at every decoding step: match the teacher's distribution over the vocabulary at position t. Kim and Rush observed that this is a mismatch with what a sequence model is actually judged on. The model is scored on whole outputs, and the space of possible sequences is exponentially large, so matching per-position marginals does not guarantee matching the distribution over sequences.\n\nSequence-level KD makes the target the sequence itself. The exact objective — cross-entropy against the teacher's full distribution over all possible output sequences — is intractable, so they approximate that distribution by its mode: run beam search with the teacher, take the single highest-scoring output, and treat it as a hard label. Training then reduces to ordinary maximum-likelihood on a corpus of (source, teacher-translation) pairs. That is the entire method, and its simplicity is the point.\n\nThe empirical effect is larger than the simplicity suggests. Teacher-generated targets are systematically less noisy and less diverse than human references: the teacher has already resolved the many-valid-translations ambiguity in a self-consistent way, so the student is fitting a unimodal, learnable function rather than an ambiguous multimodal one. A striking downstream consequence reported in the paper is that the resulting student needs *no beam search at inference* — greedy decoding is nearly as good, because the mode has effectively been baked into the training data. Their best student runs 10x faster than its state-of-the-art teacher with little loss, and adding weight pruning gives a model with 13x fewer parameters at a cost of 0.4 BLEU.\n\nWorked intuition: think of the teacher as a denoiser applied to the training set. Human reference translations for one German sentence might be 'the cat sat on the mat', 'a cat was sitting on the rug', and 'the cat is on the mat'. A small model trained on all three must spread probability across incompatible continuations and ends up hedging. Trained on the teacher's single canonical output for each source, it learns a sharp, consistent mapping.\n\nThis is also the conceptual ancestor of every black-box LLM distillation done today. Alpaca, Vicuna, Orca and the R1-Distill family are all sequence-level KD with a chat teacher instead of a translation teacher, which is why the method sits at the boundary between classical KD and the modern synthetic-data era. TRL exposes it directly as the `seq_kd` flag on GKDConfig.", "lossFormula": "\\mathcal{L}_{\\text{SeqKD}} = -\\sum_{y\\in\\mathcal{Y}} q(y\\mid x)\\,\\log p_\\theta(y\\mid x) \\;\\approx\\; -\\log p_\\theta(\\hat{y}\\mid x), \\qquad \\hat{y} = \\arg\\max_{y}\\, q(y\\mid x) \\ \\ \\text{(teacher beam search)}", "pros": [ "Black-box: needs only sampled teacher text, so it works against any API", "Trivial to implement — after generation it is ordinary supervised fine-tuning", "Teacher outputs are self-consistent, which removes reference ambiguity and helps small students disproportionately", "Often removes the need for beam search at inference, compounding the speedup", "Generation is a one-time cost and the resulting dataset is reusable and inspectable" ], "cons": [ "Off-policy: the student never sees its own generated prefixes, so exposure bias remains", "Approximating the teacher distribution by its mode discards all diversity and uncertainty", "Inherits and can amplify teacher errors, since there is no label to correct them", "Generation cost scales with corpus size and output length", "Against a commercial API this is the exact activity most terms of service prohibit for competing-model training" ], "whenToUse": "Use it as the default whenever the teacher is behind an API, or whenever you want a reusable, auditable training set. It is also the correct first step before any on-policy method, since it gives a strong initialisation cheaply.", "tools": [ "OpenNMT", "Hugging Face TRL (GKDConfig seq_kd=True)", "vLLM / SGLang for bulk generation", "OpenAI Model Distillation (Stored Completions)", "Amazon Bedrock Model Distillation" ], "examples": [ "WMT English-German NMT students 10x faster than the teacher", "Every Alpaca-lineage instruction dataset", "Bedrock and OpenAI managed distillation pipelines" ], "relatedMethods": [ "blackbox-output-distillation", "cot-rationale-distillation", "reasoning-trace-distillation", "generalized-kd", "response-kd" ], "difficulty": 1, "dataNeeded": "outputs", "teacherAccess": "black-box" }, { "id": "blackbox-output-distillation", "name": "Black-box output distillation (Alpaca / Vicuna / Orca style)", "family": "Sequence-level / data", "year": 2023, "paper": "Alpaca: A Strong, Replicable Instruction-Following Model; Orca: Progressive Learning from Complex Explanation Traces of GPT-4", "url": "https://crfm.stanford.edu/2023/03/13/alpaca.html", "description": "Generate a synthetic instruction-following dataset by prompting a strong API teacher, then supervised-fine-tune an open base model on it. The dominant form of distillation actually practised.", "howItWorks": "The recipe is three steps and no gradients ever touch the teacher. First, build a prompt set: Alpaca bootstrapped 52,000 instructions from 175 hand-written seed tasks using the self-instruct loop, where the teacher is asked to invent new instructions similar to the seeds; Vicuna instead harvested about 70,000 real user-shared ChatGPT conversations; Orca used curated system prompts designed to elicit step-by-step explanation traces rather than terse answers. Second, sample a completion from the teacher for each prompt. Third, run ordinary next-token supervised fine-tuning of an open base model on the (prompt, completion) pairs.\n\nThe economics are the reason this exploded. Stanford CRFM reports the Alpaca data generation cost under $500 of OpenAI API calls, and the fine-tune itself as three hours on eight 80GB A100s for under $100 — roughly $600 total to move LLaMA-7B from a raw base model to something that won 90 of 179 head-to-head comparisons against text-davinci-003 on the self-instruct evaluation set. That is a four-to-five order of magnitude discount on the teacher's own training cost.\n\nWhat actually transfers, and what does not, is the crux. Format, tone, instruction-following behaviour and refusal style transfer very efficiently, because they are surface regularities present in nearly every sample. Underlying capability transfers far less — a 7B student imitating GPT-4's confident prose can produce fluent, well-structured answers that are wrong, because it learned the shape of a good answer without the computation that produced it. Orca's response was to demand richer signal: system instructions that force the teacher to expose its reasoning, plus teacher-assistance from a mid-tier model, so the student learns the process rather than only the product. Orca 2 pushed further, arguing that pure imitation caps a small model's potential and that students should be taught to *select* a solution strategy rather than always copy the teacher's.\n\nThe legal and platform dimension is now inseparable from the technique. Every major API's terms restrict using outputs to develop competing models, and on 23 February 2026 Anthropic published a detailed account of what it called industrial-scale distillation attacks — roughly 24,000 fraudulent accounts and over 16 million exchanges attributed to DeepSeek, Moonshot AI and MiniMax, detected via classifiers, behavioural fingerprinting, IP correlation and request metadata. Meanwhile the same technique, applied to your own outputs from a model you pay for, is a first-class supported product on OpenAI and Bedrock. The method is identical; the licence is what differs.", "lossFormula": "\\mathcal{L} = -\\,\\mathbb{E}_{(x,y)\\sim\\mathcal{D}_{\\text{teacher}}}\\ \\sum_{t=1}^{|y|} \\log p_\\theta\\big(y_t \\mid x,\\ y_{ mid -> small)", "Orca's use of ChatGPT as teacher assistant alongside GPT-4" ], "relatedMethods": [ "response-kd", "reverse-kl-minillm", "multi-teacher-kd", "self-distillation", "prune-then-distill" ], "difficulty": 2, "dataNeeded": "logits", "teacherAccess": "white-box" }, { "id": "multi-teacher-kd", "name": "Multi-teacher KD", "family": "Response-based", "year": 2017, "paper": "Efficient Knowledge Distillation from an Ensemble of Teachers (Fukuda et al., Interspeech 2017); weighting variants surveyed in Knowledge Distillation: A Survey (Gou et al., 2020, arXiv 2006.05525)", "url": "https://www.isca-archive.org/interspeech_2017/fukuda17_interspeech.html", "description": "Distil from several teachers at once, combining their output distributions (or features) into a single target for the student.", "howItWorks": "Different teachers make different mistakes. That is the entire justification for ensembling, and multi-teacher KD is the attempt to keep the ensemble's accuracy while paying for only one model at inference. Train K teachers — different architectures, different seeds, different data slices, or different specialisms — and give the student a combined target.\n\nThe baseline combination is a uniform average of the softened teacher distributions, which is simple and reduces variance, since idiosyncratic teacher errors partially cancel while the shared signal reinforces. But the survey literature is blunt about the limitation: averaging treats every teacher as equally trustworthy and thereby erases the diversity that made the ensemble valuable. A biased teacher contributes as much as an unbiased one, and where teachers genuinely disagree the average is a compromise that may correspond to no correct answer at all. Later work therefore weights teachers adaptively — by per-sample confidence, by entropy, by measured correctness, or by learned gating — and some methods have the student sample one teacher per batch rather than blending.\n\nWorked intuition: three radiologists read a scan. Two say benign with high confidence, one says malignant with moderate confidence. Averaging their probability vectors produces a lukewarm 'probably benign' that no individual radiologist would endorse and that discards the fact that one specialist saw something. A confidence- or expertise-weighted combination preserves the disagreement structure; a flat mean does not.\n\nIn LLM practice this shows up as multi-source data distillation more often than as logit averaging: sample candidate answers from several strong models, keep the best per prompt by an automatic judge or a verifier, and fine-tune on the union. That variant is black-box, sidesteps the tokenizer-alignment problem entirely, and is what most open instruction datasets actually do. The white-box logit-averaging form requires all K teachers to share a vocabulary and to be resident (or their logits cached, which is K times the storage).", "lossFormula": "\\mathcal{L} = T^2\\,\\mathrm{KL}\\Big(\\textstyle\\sum_{k=1}^{K} w_k\\,\\sigma\\!\\big(z_{t_k}/T\\big)\\ \\Big\\|\\ \\sigma\\!\\big(z_s/T\\big)\\Big) + (1-\\alpha)\\mathcal{H}(y,\\sigma(z_s)),\\qquad \\textstyle\\sum_k w_k = 1,\\ w_k \\ \\text{uniform or adaptive}", "pros": [ "Captures ensemble-level accuracy at single-model inference cost", "Averaging reduces the variance and idiosyncratic bias of any single teacher", "Lets you combine specialists — a code teacher, a maths teacher, a safety teacher", "The black-box data variant needs no tokenizer alignment and works with API teachers", "Teacher logits can be precomputed once and reused across student runs" ], "cons": [ "Uniform averaging destroys the diversity that motivated the ensemble", "Conflicting teachers produce compromise targets that can be worse than any teacher", "K forward passes or K cached logit sets: K times the compute or storage", "Adaptive weighting adds real complexity and its own hyperparameters", "White-box form requires a shared vocabulary across all teachers" ], "whenToUse": "Use when you have several strong models with complementary strengths and want one deployable student. Prefer the black-box best-of-K data variant with a verifier or judge unless you specifically need per-token logit blending.", "tools": [ "Custom PyTorch loops", "Arcee DistillKit", "vLLM for multi-model bulk generation", "LLM-as-judge pipelines for candidate selection" ], "examples": [ "Acoustic-model ensembles distilled into one recogniser", "Margin-MSE's 3-teacher BERT-cat ensemble for retrieval", "Open instruction datasets mixing outputs from several frontier models" ], "relatedMethods": [ "response-kd", "deep-mutual-learning", "teacher-assistant-kd", "retriever-distillation", "blackbox-output-distillation" ], "difficulty": 3, "dataNeeded": "logits", "teacherAccess": "white-box" }, { "id": "deep-mutual-learning", "name": "Deep mutual learning (online distillation)", "family": "Response-based", "year": 2017, "paper": "Deep Mutual Learning", "url": "https://arxiv.org/abs/1706.00384", "description": "Train a cohort of untrained peer networks simultaneously, each learning from the labels and from the others' predictions. No pretrained teacher exists at any point — every peer is white-box to every other.", "howItWorks": "Zhang, Xiang, Hospedales and Lu asked whether distillation actually requires a teacher, and answered no. In deep mutual learning a pool of networks — which may be identical or different architectures — starts from random initialisation and trains together. Each network's loss has two terms: standard supervised cross-entropy against the labels, and a mimicry term that is the KL divergence to the average of its peers' current predictions. Every network is simultaneously a student and a teacher, and the roles never separate.\n\nThe surprising empirical result is that this beats distilling from a single powerful, static, pretrained teacher. The explanation the paper offers is about the *shape* of the learned solution rather than raw accuracy. Independently trained networks converge to different local minima and are confident about different secondary classes; the mimicry term makes each network account for its peers' plausible-but-different views, which raises the entropy of its posterior and steers the cohort toward wider, flatter minima that generalise better. Note the key line from the paper: in mutual learning the cohort effectively pools its collective estimate of the next most likely classes.\n\nWorked intuition: two students revising together who explain their reasoning to each other end up better than either would after reading the professor's polished lecture notes. The professor's notes are correct but already compressed past the point where the beginner can reconstruct the reasoning; the peer's half-formed explanation is at the right level and exposes exactly the alternatives worth considering.\n\nThe practical costs are that you train K models to deploy one, memory scales with K, and the cohort can converge to a shared wrong opinion (mutual reinforcement of a common error) with nothing external to correct it. Cohort size and the weight of the mimicry term are the tuning knobs. In modern practice this family — 'online' or 'collaborative' distillation — matters most when no strong teacher exists for your domain at all, which is common in specialised industrial and scientific settings.", "lossFormula": "\\mathcal{L}_{\\theta_1} = \\mathcal{H}\\big(y,\\sigma(z_1)\\big) \\;+\\; \\frac{1}{K-1}\\sum_{k\\neq 1} \\mathrm{KL}\\big(\\sigma(z_k)\\ \\|\\ \\sigma(z_1)\\big) \\qquad\\text{(symmetrically for every peer } k)", "pros": [ "No pretrained teacher required — usable when none exists for your domain", "Outperforms distillation from a stronger static teacher in the original experiments", "Peers may have different architectures, so you can co-train a deployable small model with a large one", "Produces an ensemble as a by-product if you want to keep all peers", "Empirically finds flatter, better-generalising minima" ], "cons": [ "Trains K models to deploy one — K times the compute and memory", "Cohorts can collectively converge on a shared error with no external correction", "Cohort size and mimicry weight are extra hyperparameters", "Harder to reason about and debug than a fixed teacher-student pipeline", "Rarely used at LLM scale, where training two frontier models in lockstep is impractical" ], "whenToUse": "Use when you must train from scratch in a domain with no strong pretrained teacher — specialised scientific, industrial or medical models — and you can afford to train a small cohort. Also useful when you want a large and a small model co-trained in one run.", "tools": [ "Custom PyTorch training loops", "timm (multi-model harness)", "MMPretrain / mmrazor" ], "examples": [ "CIFAR-100 and Market-1501 person re-identification in the original paper", "Online distillation variants in production vision stacks" ], "relatedMethods": [ "self-distillation", "multi-teacher-kd", "response-kd", "teacher-assistant-kd" ], "difficulty": 2, "dataNeeded": "logits", "teacherAccess": "none" }, { "id": "prune-then-distill", "name": "Pruning + distillation (Minitron, Sheared LLaMA)", "family": "Compression-composed", "year": 2023, "paper": "Compact Language Models via Pruning and Knowledge Distillation; LLM Pruning and Distillation in Practice: The Minitron Approach; Sheared LLaMA", "url": "https://arxiv.org/abs/2407.14679", "description": "Structurally prune a large model down to a target shape, then use distillation from the unpruned original to recover the lost accuracy on a small fraction of the original token budget.", "howItWorks": "Training a small model from scratch throws away everything the large model learned. Pruning keeps the weights but breaks the network. Composing them is now the strongest known recipe for producing a small model from a large one, and it is what NVIDIA, Meta and Princeton all converged on.\n\nThe Minitron pipeline runs in three moves. First, **importance estimation**: run calibration data through the teacher and score layers, attention heads, MLP neurons and embedding channels by their activation-based contribution. Second, **structured pruning**: remove the lowest-scoring structures to hit a target shape. NVIDIA reports two variants for Llama 3.1 8B — width pruning, which cuts hidden dimension from 4096 to 3072 and MLP dimension from 14336 to 9216, and depth pruning, which halves the layer count from 32 to 16. Third, **distillation-based retraining**: train the pruned model against the original using logit KD plus intermediate-state losses. The token budget is the striking number: 94B tokens versus the teacher's 15T, under 1% of the original token budget and a roughly 160x reduction, yielding average speedups of 2.7x (depth) and 1.8x (width).\n\nThe Minitron papers add a step called **teacher correction** — lightly fine-tuning the teacher on the distillation dataset before pruning and distilling, because when you lack access to the original training data the teacher's distribution no longer matches your corpus and the distillation signal degrades. The PyTorch torchtune case study independently found the same thing: using a LoRA-finetuned teacher gave lower KD loss and better hellaswag and commonsense scores than using the base teacher, whose KD loss stayed nearly flat.\n\nSheared LLaMA attacks the same problem from the pre-training side. It formulates pruning as a constrained optimisation that prunes LLaMA2-7B end-to-end to an exact target architecture, then adds **dynamic batch loading**, which reweights the domain mixture of each training batch according to which domains' losses are lagging. The resulting 1.3B and 2.7B models beat Pythia, INCITE, OpenLLaMA and TinyLlama at equivalent sizes.\n\nThe caveats are practical rather than conceptual: this is the most engineering-heavy method in the library, it needs the teacher weights and a large corpus, aggressive pruning can silently destroy narrow capabilities that broad benchmarks do not measure, and depth versus width pruning trade differently — depth gives better latency, width tends to preserve quality better.", "lossFormula": "\\hat{M} = \\mathrm{Prune}\\big(M;\\ \\mathcal{I}(\\text{layers, heads, neurons, channels})\\big)\\ \\ \\text{then}\\ \\ \\mathcal{L} = \\mathrm{KL}\\big(p_{M}\\,\\|\\,p_{\\hat{M}}\\big) + \\sum_m \\gamma_m\\,\\mathrm{MSE}\\big(H^{m}_{\\hat{M}},\\,H^{m}_{M}\\big)", "pros": [ "Retains the teacher's learned weights instead of starting over — the reason it needs under 1% of the original tokens", "Real deployment wins: 1.8x-2.7x measured speedups for Llama-3.1-Minitron-4B", "Lets you hit an exact target architecture and latency budget", "Sheared LLaMA students beat same-size models trained from scratch", "Open weights and reference recipes exist for both lineages" ], "cons": [ "The most complex pipeline here: importance scoring, pruning, teacher correction, distillation retraining", "Requires full teacher weights and a large re-training corpus", "Aggressive pruning can remove narrow capabilities that aggregate benchmarks will not reveal", "Depth versus width is a genuine trade (latency versus quality) with no universal answer", "Needs substantial GPU memory: teacher and student resident together over billions of tokens" ], "whenToUse": "Use when you own or can license the teacher weights and need a specific smaller architecture for a hard latency or memory budget. This is how you build a model family from a single large model rather than training each size independently.", "tools": [ "NVIDIA TensorRT Model Optimizer", "NVIDIA NeMo (pruning + distillation recipes)", "princeton-nlp/LLM-Shearing", "torch.nn.utils.prune for the basics" ], "examples": [ "Llama-3.1-Minitron-4B (width and depth variants)", "Mistral-NeMo-Minitron-8B from Mistral NeMo 12B", "Sheared-LLaMA-1.3B and -2.7B from LLaMA2-7B", "Llama 3.2 1B/3B, which used logits from Llama 3.1 8B and 70B to recover after pruning" ], "relatedMethods": [ "quantization-aware-distillation", "response-kd", "feature-kd", "teacher-assistant-kd", "generalized-kd" ], "difficulty": 5, "dataNeeded": "logits", "teacherAccess": "white-box" }, { "id": "quantization-aware-distillation", "name": "Quantization-aware distillation (QAD / LLM-QAT)", "family": "Compression-composed", "year": 2023, "paper": "LLM-QAT: Data-Free Quantization Aware Training for Large Language Models", "url": "https://arxiv.org/abs/2305.17888", "description": "Train a low-bit student with quantization simulated in the loop while distilling from the full-precision original, recovering the accuracy that post-training quantization loses.", "howItWorks": "Post-training quantization simply rounds a trained model's weights to fewer bits. It works acceptably at 8 bits and degrades sharply below 4, because the rounding error compounds through the network and nothing ever adapts to it. Quantization-aware distillation fixes this by making the model train *with* the rounding present and *against* its own full-precision self.\n\nMechanically: insert fake-quantize operations into the forward pass so activations and weights are rounded to the target bit-width during training, but pass gradients through those non-differentiable rounding operations with a straight-through estimator. Then set the loss to be distillation from the untouched full-precision model — KL between the FP teacher's output distribution and the quantized student's. The student therefore learns weights that are *robust to their own rounding*, redistributing information away from the values that quantization destroys. Note what QAD is and is not: it is an accuracy-recovery technique, not a way to teach the model anything new. Teacher and student are the same model at different precisions.\n\nLLM-QAT's specific contribution is that it is **data-free**. You often cannot get the original pretraining corpus, and fine-tuning a quantized LLM on a mismatched corpus damages it. So Liu et al. use the pretrained model's own generations as the training data, which by construction matches the original output distribution. They also quantize the KV cache, not just weights and activations — critical because at long sequence lengths the KV cache, not the weights, dominates memory and bandwidth.\n\nWorked intuition: a full-precision weight of 0.734 rounds to 0.75 at 4 bits, an error of about 2%. Across billions of weights those errors interact and the model's outputs drift. QAD lets the optimiser see that drift during training and compensate — moving to a value that rounds cleanly, or shifting the burden to a neighbouring weight — so the *quantized* network, not the full-precision one, is what gets optimised.\n\nThis is now standard production practice: NVIDIA's Model Optimizer ships QAD recipes, and it is the usual final step after pruning-and-distillation. The costs are that training a quantized model is slower per step than training a normal one, low-bit regimes (2-3 bits) remain genuinely hard, and the achieved speedup depends entirely on whether your inference kernels actually support the chosen format.", "lossFormula": "\\mathcal{L} = \\mathrm{KL}\\big(p_{\\mathrm{fp}}(\\cdot\\mid x)\\ \\big\\|\\ p_{Q(\\theta)}(\\cdot\\mid x)\\big),\\qquad Q(w) = s\\cdot\\mathrm{clip}\\Big(\\Big\\lfloor \\tfrac{w}{s} \\Big\\rceil,\\ -2^{\\,b-1},\\ 2^{\\,b-1}-1\\Big),\\qquad \\frac{\\partial Q}{\\partial w}\\ \\approx\\ \\mathbb{1}_{|w|\\le \\tau}\\ \\ \\text{(STE)}", "pros": [ "Recovers most of the accuracy lost by post-training quantization, especially below 8 bits", "Data-free variant needs no access to the original training corpus", "Quantizing the KV cache unlocks long-context throughput that weight-only quantization does not", "Applies to any generative model independent of its training data", "Stacks directly on top of pruning-and-distillation as the final compression step" ], "cons": [ "Training with fake-quantize ops is slower per step than normal training", "Straight-through gradients are biased, so optimisation is noisier", "2-3 bit regimes are still an open problem despite steady progress", "The realised speedup depends on kernel support for the chosen format, not on the bit-width alone", "Recovers accuracy but teaches nothing new — it cannot fix a weak model" ], "whenToUse": "Use as the last step before deployment when you are quantizing below 8 bits and post-training quantization has cost you more accuracy than you can accept, particularly on edge hardware or long-context serving where the KV cache dominates.", "tools": [ "NVIDIA TensorRT Model Optimizer", "facebookresearch/LLM-QAT", "torchao / torch.ao.quantization", "torchtune QAT recipes", "Intel Neural Compressor" ], "examples": [ "LLM-QAT 4-bit weight/activation/KV-cache LLaMA models", "NVIDIA NVFP4 accuracy-recovery pipelines", "On-device deployment of Minitron- and Gemma-class students" ], "relatedMethods": [ "prune-then-distill", "response-kd", "self-distillation", "draft-model-distillation" ], "difficulty": 4, "dataNeeded": "logits", "teacherAccess": "white-box" }, { "id": "dataset-distillation", "name": "Dataset distillation / condensation", "family": "Dataset / context", "year": 2018, "paper": "Dataset Distillation", "url": "https://arxiv.org/abs/1811.10959", "description": "Hold the model fixed and compress the *training set* into a tiny synthetic set that trains a network to comparable accuracy. The data, not the network, is the thing distilled.", "howItWorks": "Wang, Zhu, Torralba and Efros inverted the standard framing. Model distillation compresses a complex model into a simple one; dataset distillation keeps the model and asks how few synthetic examples can carry the information needed to train it. The answer, on MNIST-scale problems, is startlingly few — a handful of images per class, which typically do not look like natural images at all but like optimised interference patterns that happen to induce the right gradients.\n\nThe formulation is bi-level optimisation. The inner problem takes a randomly initialised network and performs one (or a few) gradient steps on the synthetic set. The outer problem evaluates the resulting network on the *real* training data and backpropagates that loss all the way through the inner update into the synthetic pixels. The synthetic images and their (often soft, learnable) labels are the parameters being optimised. Because you must differentiate through an optimisation step, the memory and compute cost is severe, and this is what has limited the field.\n\nSubsequent work replaced the expensive meta-gradient with cheaper surrogate objectives: **gradient matching** makes synthetic-data gradients match real-data gradients at each step; **distribution matching** aligns feature statistics; **trajectory matching** makes a network trained on synthetic data follow the same parameter path as one trained on real data. Each trades some fidelity for tractability.\n\nWorked intuition: think of it as designing an exam rather than a textbook. You cannot cover the whole subject in ten questions, but you can choose ten questions such that a student who can answer them must have learned the material — each one probing a decision boundary that nothing else probes. Dataset distillation searches for exactly those maximally informative synthetic items.\n\nThe practical applications are neural architecture search (evaluate hundreds of candidate architectures on a 100-image proxy set), continual learning (a condensed replay buffer that resists forgetting), federated learning (send distilled data instead of raw data), and privacy (the synthetic examples are not real records, though the privacy guarantee is empirical rather than formal). Scaling to ImageNet-size and to LLM corpora remains genuinely hard, and distilled sets are known to be architecture-specific — a set condensed for a ConvNet often transfers poorly to a ResNet.", "lossFormula": "\\tilde{\\mathcal{D}}^{*} = \\arg\\min_{\\tilde{\\mathcal{D}}}\\ \\mathcal{L}\\big(\\mathcal{D};\\ \\theta_1(\\tilde{\\mathcal{D}})\\big) \\qquad\\text{s.t.}\\qquad \\theta_1(\\tilde{\\mathcal{D}}) = \\theta_0 - \\eta\\,\\nabla_{\\theta}\\,\\mathcal{L}\\big(\\tilde{\\mathcal{D}};\\,\\theta_0\\big),\\quad \\theta_0\\sim p(\\theta_0)", "pros": [ "Extreme compression of the data axis — orders of magnitude fewer examples", "Makes architecture search and hyperparameter sweeps dramatically cheaper via proxy datasets", "Natural fit for continual-learning replay buffers and federated settings", "Synthetic examples are not literal records, which helps (informally) with data-sharing constraints", "Reusable: once condensed, the set trains many models" ], "cons": [ "Bi-level optimisation is expensive and memory-hungry; differentiating through training steps does not scale naively", "Distilled sets are often architecture-specific and transfer poorly", "Scaling beyond small image benchmarks remains an open problem", "Synthetic examples are uninterpretable, so you cannot audit what the set actually encodes", "Privacy protection is empirical, not a formal guarantee" ], "whenToUse": "Use for neural architecture search proxies, continual-learning replay buffers, and federated or data-constrained settings — not as a route to a better production model. Treat it as data-axis compression, orthogonal to every other method here.", "tools": [ "VICO-UoE/DatasetCondensation", "DC-BENCH", "torchvision + custom bi-level loops" ], "examples": [ "MNIST and CIFAR-10 condensed to a handful of images per class", "Proxy datasets for architecture search", "Condensed replay buffers in continual learning" ], "relatedMethods": [ "context-distillation", "self-distillation", "blackbox-output-distillation" ], "difficulty": 5, "dataNeeded": "none", "teacherAccess": "none" }, { "id": "preference-distillation-rlaif", "name": "Preference distillation / RLAIF", "family": "Preference", "year": 2023, "paper": "Constitutional AI: Harmlessness from AI Feedback; Zephyr: Direct Distillation of LM Alignment", "url": "https://arxiv.org/abs/2212.08073", "description": "Distil judgement rather than answers: a teacher model ranks candidate completions, those rankings train a reward model, and the student is optimised against it — human annotation replaced by AI feedback.", "howItWorks": "Everything earlier in this library transfers what the teacher *says*. Preference distillation transfers what the teacher *prefers*, which is a different and often more valuable quantity — a teacher can reliably identify the better of two answers in domains where it cannot reliably produce a great answer unprompted.\n\nAnthropic's Constitutional AI established the loop: generate a response, ask the model to critique it against an explicit written set of principles, ask it to revise, and use the resulting comparisons as preference data — reinforcement learning from AI feedback in place of RLHF's human labellers. The reward model trained on those AI-generated comparisons then drives a standard KL-regularised policy-optimisation objective, where the KL term to a reference model prevents the policy drifting into reward-model exploits.\n\nThe key economics: preference labels are far cheaper to obtain from a model than from humans and can be produced at a scale human annotation cannot approach, while the ranking task is easier than the generation task, so a teacher's preferences are often more reliable than its completions. The principles are also written down and auditable, which is a governance property no amount of human labelling gives you.\n\nWorked intuition: an editor who cannot write a bestselling novel can still tell you reliably which of two drafts is better. Distilling the editor's judgement into a writer produces a better writer than making the writer imitate the editor's own prose would.\n\nZephyr showed how far you can strip this down. Tunstall et al. take an SFT model, sample four completions per prompt from a set of open models, have GPT-4 rank them, and then skip the reward model and RL loop entirely in favour of direct preference optimisation on the resulting pairs (see the dDPO entry). Zephyr-7B set state of the art for 7B chat models and surpassed Llama2-Chat-70B on MT-Bench with no human annotation at all. The failure modes are inherited: the student adopts the teacher's biases and blind spots wholesale, reward models are exploitable, and full RLAIF with PPO is operationally heavy — which is exactly why most teams now use the direct variant.", "lossFormula": "r_\\phi \\leftarrow \\arg\\max_\\phi \\sum \\log\\sigma\\big(r_\\phi(x,y_w) - r_\\phi(x,y_l)\\big),\\ \\ (y_w,y_l)\\ \\text{ranked by the teacher};\\qquad \\max_\\theta\\ \\mathbb{E}_{y\\sim\\pi_\\theta}\\big[r_\\phi(x,y)\\big] - \\beta\\,\\mathrm{KL}\\big(\\pi_\\theta\\,\\|\\,\\pi_{\\text{ref}}\\big)", "pros": [ "Removes the human-annotation bottleneck, so preference data scales with compute", "Ranking is an easier task than generation, so teacher preferences are often more reliable than teacher completions", "The governing principles are explicit and auditable, unlike implicit annotator preferences", "Black-box: only the teacher's judgements are needed", "Zephyr-7B beat Llama2-Chat-70B on MT-Bench with no human labels" ], "cons": [ "The student inherits the teacher's values, biases and blind spots without correction", "Reward models are exploitable; policies find degenerate high-reward behaviours", "Full RLAIF with PPO is operationally heavy — reward model, policy, reference and value models in memory", "Judge models exhibit known artefacts such as position and verbosity bias", "Requires a KL anchor to a reference model or the policy drifts off distribution" ], "whenToUse": "Use for alignment, tone, helpfulness and safety behaviours where 'better' is a judgement call rather than a verifiable fact. If you do not need an explicit reward model, go straight to dDPO — it is far simpler for most of the benefit.", "tools": [ "Hugging Face TRL (RewardTrainer, PPOTrainer, GRPOTrainer)", "OpenRLHF", "NVIDIA NeMo Aligner", "alignment-handbook", "LLM-as-judge harnesses" ], "examples": [ "Anthropic Constitutional AI / RLAIF", "Zephyr-7B's AI-ranked preference stage", "UltraFeedback and the open AI-feedback dataset ecosystem" ], "relatedMethods": [ "dpo-teacher-preferences", "blackbox-output-distillation", "reasoning-trace-distillation", "reverse-kl-minillm" ], "difficulty": 4, "dataNeeded": "outputs", "teacherAccess": "black-box" }, { "id": "dpo-teacher-preferences", "name": "DPO on teacher preferences (dDPO)", "family": "Preference", "year": 2023, "paper": "Direct Preference Optimization: Your Language Model is Secretly a Reward Model; Zephyr: Direct Distillation of LM Alignment", "url": "https://arxiv.org/abs/2305.18290", "description": "Skip the reward model and the RL loop: optimise the student directly on teacher-ranked preference pairs with a simple classification-style loss.", "howItWorks": "Rafailov et al.'s result is that the RLHF objective has a closed-form solution relating the optimal policy to the reward, which can be inverted — the language model *is* implicitly a reward model. That means you never need to train an explicit reward model or run PPO. Instead you can directly optimise a logistic loss on the difference between the policy's and a frozen reference policy's log-probability ratios for the chosen and rejected responses, with a temperature beta controlling how far the policy may drift from the reference.\n\nApplied to distillation, the preference pairs come from a teacher rather than from humans, and the whole pipeline becomes almost embarrassingly simple. Zephyr's recipe is: distil-SFT on a teacher-generated instruction corpus, sample four candidate completions per prompt from a set of open models, have a strong teacher rank them, form (chosen, rejected) pairs, and run dDPO. No reward model, no value model, no rollouts, no online generation — it is a supervised objective over a static dataset, which means it trains on hardware where PPO would not fit and converges in hours. Zephyr-7B set state of the art among 7B chat models and surpassed the best open RLHF model of the time on MT-Bench, with zero human annotation.\n\nWorked intuition for the loss: it is binary classification. Given a prompt and two responses, the model must assign relatively more probability mass (compared to the reference model) to the one the teacher preferred. Beta controls the stakes — small beta lets the policy move far to satisfy preferences and risks degeneration, large beta keeps it close to the reference and limits how much preference signal can be absorbed.\n\nThe known limitations are worth stating plainly. DPO is off-policy: it only ever sees the fixed pairs in your dataset, so it cannot discover and correct behaviours that the student develops during training but that no pair covers. It is sensitive to beta and to the quality of the reference model. And because the loss only cares about the *relative* margin, it can achieve low loss by pushing down the rejected response rather than pushing up the chosen one, which degrades both — a well-documented pathology that later variants (IPO, KTO, ORPO, SimPO) were designed to address. Even so, dDPO is where most teams should start: it captures most of preference distillation's value at a small fraction of RLAIF's complexity.", "lossFormula": "\\mathcal{L}_{\\text{DPO}} = -\\,\\mathbb{E}_{(x,y_w,y_l)\\sim\\mathcal{D}_{\\text{teacher}}}\\left[\\log \\sigma\\!\\left(\\beta\\log\\frac{\\pi_\\theta(y_w\\mid x)}{\\pi_{\\text{ref}}(y_w\\mid x)} \\;-\\; \\beta\\log\\frac{\\pi_\\theta(y_l\\mid x)}{\\pi_{\\text{ref}}(y_l\\mid x)}\\right)\\right]", "pros": [ "No reward model, no value model, no rollouts — a plain supervised loss over a static dataset", "Trains in hours on hardware where PPO would not fit", "Zephyr-7B beat Llama2-Chat-70B on MT-Bench using only AI-ranked preferences", "Black-box: needs teacher rankings, not teacher internals", "First-class support in TRL and the alignment-handbook, with well-trodden recipes" ], "cons": [ "Off-policy — cannot correct behaviours that emerge during training but are absent from the pair set", "Sensitive to beta and to the choice of reference model", "Can lower the chosen response's probability as long as the rejected one falls faster, degrading both", "Inherits every bias of the ranking teacher", "Needs a good SFT checkpoint first; DPO on a weak base is unreliable" ], "whenToUse": "Use as the default alignment stage after any SFT-based distillation. If you cannot afford RLAIF's infrastructure — which is almost everyone — this captures most of the benefit. Escalate to online preference methods only if you can show DPO is plateauing.", "tools": [ "Hugging Face TRL DPOTrainer", "alignment-handbook", "Axolotl", "LLaMA-Factory", "OpenRLHF" ], "examples": [ "Zephyr-7B-beta (dDPO on GPT-4-ranked UltraFeedback)", "Tulu and Nous-family preference-tuned models", "Most open 7B-70B chat models released since late 2023" ], "relatedMethods": [ "preference-distillation-rlaif", "blackbox-output-distillation", "reasoning-trace-distillation", "sequence-level-kd" ], "difficulty": 2, "dataNeeded": "outputs", "teacherAccess": "black-box" }, { "id": "cross-vocabulary-kd", "name": "Cross-vocabulary logit KD (ULD, DSKD)", "family": "Response-based", "year": 2024, "paper": "Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs; Dual-Space Knowledge Distillation for Large Language Models", "url": "https://arxiv.org/abs/2402.12030", "description": "Make white-box logit distillation work when teacher and student have different tokenizers, either by comparing sorted probability vectors with an optimal-transport cost or by projecting both models into a shared output space.", "howItWorks": "Standard logit KD computes a divergence between two vectors indexed by vocabulary position. If teacher and student have different tokenizers, position 4,721 means a different token in each and the comparison is meaningless. This single constraint has confined white-box distillation to within-family pairs — you can distil Llama 70B into Llama 8B but not into Qwen 7B, no matter how good a fit the student would be.\n\n**ULD** (Boizard et al.) removes the alignment requirement by refusing to align. Sort each probability vector in descending order and compare the sorted distributions with an optimal-transport (Wasserstein) cost under a uniform ground metric. The intuition is that the *shape* of the distribution — how peaked it is, how fast mass falls off across the ranked alternatives — carries most of the transferable information about the teacher's confidence and uncertainty, and that shape is well-defined regardless of which symbols occupy which ranks. You lose token identity but keep the calibration signal, and unlike KL the transport cost is well behaved when the two vocabularies have different sizes.\n\n**DSKD** (Zhang et al., EMNLP 2024) takes the opposite route: rather than discarding identity, recover it. The paper diagnoses a *space discrepancy* — teacher and student representations and distributions live in different spaces, so their similarity is low even for the same input — and unifies the two output spaces before applying any divergence. A cross-model attention mechanism automatically aligns representations across differing vocabularies, after which any standard distance (KL included) can be used. The result is a general white-box framework covering both same-tokenizer and cross-tokenizer distillation.\n\nWorked intuition for ULD: two people describe the same photograph in different languages. You cannot align their words, but you can align their *emphasis structure* — how much of the description goes to the most salient element, the second, the third. If one is highly confident about a single subject and the other hedges across five, that difference survives translation. ULD transfers exactly that.\n\nThe honest assessment: both methods work, both are strictly harder to tune than same-tokenizer KD, and neither has become a default. torchtune's roadmap still lists cross-tokenizer distillation as future work, and in practice most teams facing a tokenizer mismatch fall back to black-box sequence-level distillation, which sidesteps the problem entirely at the cost of signal density. This is an active area where the tooling has not yet caught up with the papers.", "lossFormula": "\\mathcal{L}_{\\text{ULD}} = \\sum_{t=1}^{T} \\mathcal{W}_1\\Big(\\mathrm{sort}\\big(p_t^{\\text{teacher}}\\big),\\ \\mathrm{sort}\\big(q_t^{\\text{student}}\\big)\\Big) \\qquad\\text{vs.}\\qquad \\mathcal{L}_{\\text{DSKD}} = \\mathrm{KL}\\Big(\\mathcal{P}_{\\text{shared}}\\big(p_t\\big)\\ \\Big\\|\\ \\mathcal{P}_{\\text{shared}}\\big(q_t\\big)\\Big)", "pros": [ "Unlocks white-box distillation across model families — a Llama teacher into a Qwen student", "ULD needs no token alignment at all and tolerates different vocabulary sizes", "DSKD is a general framework covering same-tokenizer and cross-tokenizer cases with one interface", "Much denser signal than falling back to black-box sequence-level KD", "Lets you pick the best available teacher rather than the best same-family teacher" ], "cons": [ "ULD discards token identity, so some information is provably lost", "DSKD's cross-model attention adds trainable parameters and optimisation complexity", "Both are consistently reported as less reliable than same-tokenizer KD", "Little first-class support in mainstream training libraries; expect to work from the reference repos", "Optimal-transport computation adds per-step cost" ], "whenToUse": "Use when the best available teacher is in a different model family from your required student and you host both sets of weights. If you do not strictly need per-token signal, black-box sequence-level distillation is simpler and more predictable.", "tools": [ "Nicolas-BZRD/llm-recipes (ULD)", "songmzhang/DSKD and DSKDv2", "Custom TRL trainers" ], "examples": [ "Cross-family distillation experiments in the ULD paper", "DSKD's same- and cross-vocabulary LLM benchmarks", "Multi-Level Optimal Transport and later cross-tokenizer variants" ], "relatedMethods": [ "response-kd", "generalized-kd", "sequence-level-kd", "multi-teacher-kd" ], "difficulty": 5, "dataNeeded": "logits", "teacherAccess": "white-box" }, { "id": "draft-model-distillation", "name": "Draft-model distillation for speculative decoding (EAGLE, Medusa)", "family": "Trajectory / sampler", "year": 2024, "paper": "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty; Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", "url": "https://arxiv.org/abs/2401.15077", "description": "Distil a tiny draft head from a frozen target model so that speculative decoding accepts more proposed tokens. The target model's output distribution is provably unchanged; only latency falls.", "howItWorks": "Autoregressive decoding is memory-bandwidth bound: generating one token requires streaming the entire weight set through the accelerator, so the arithmetic units sit idle. Speculative decoding fixes this by having a cheap drafter propose several tokens, then verifying all of them in a single target-model forward pass. Verification uses a rejection-sampling rule that guarantees the accepted tokens are distributed exactly as the target model would have produced them — this is a *lossless* speedup, not an approximation. The only lever is acceptance rate, and that is what distillation optimises.\n\n**EAGLE**'s insight is that drafting at the token level is unnecessarily hard. It drafts autoregressively at the *feature* level instead — predicting the target model's second-to-top-layer hidden state, which is far more regular and predictable than the token distribution. Feature-level autoregression introduces its own uncertainty because the sampled token at each step is hidden, so EAGLE additionally feeds the token sequence from the previous step (the actual sampling result) into the draft model, resolving the ambiguity. Training is a regression loss on features plus cross-entropy on the resulting token distribution, against a frozen target. Reported result: a 2.7x-3.5x latency speedup on LLaMA2-Chat 70B with doubled throughput and an unchanged output distribution. EAGLE-2 adds dynamic draft trees; EAGLE-3 abandons feature prediction for direct token prediction and fuses multiple layers via a 'training-time test' technique, which lets the drafter keep improving as training data scales.\n\n**Medusa** takes a simpler route: bolt several extra decoding heads onto the target model's last hidden state, each predicting the token k positions ahead. Each head emits multiple top candidates, and the resulting combinations are verified in parallel using tree attention. No separate draft model, no separate serving stack — just extra heads trained while the backbone stays frozen.\n\nWorked intuition: a court stenographer who can anticipate the next few words of a familiar phrase types ahead, and the speaker either confirms or corrects. If the guesses are usually right you finish much faster; if they are wrong you lose nothing but the wasted keystrokes, and the transcript is identical either way. Distillation is what makes the stenographer's guesses good.\n\nThis is now standard inference infrastructure rather than research — EAGLE is supported in vLLM and SGLang — and it is the clearest example of distillation that changes latency without changing the model's behaviour at all. The trade-offs: draft quality is workload-specific (a drafter trained on chat underperforms on code), tree verification adds real implementation complexity, and gains shrink at large batch sizes where the system is already compute-bound rather than bandwidth-bound.", "lossFormula": "\\mathcal{L} = \\underbrace{\\mathrm{SmoothL1}\\big(f_{\\text{draft}}(h_{