Methods catalogued here
27 methods
spanning 2014-2025
Counted in extras.methods; families extend the response / feature / relation trichotomy of Gou et al. (arXiv 2006.05525) with six additional families specific to generative and compression-composed KD; eleven families in total.
arxiv.org
DistilBERT GLUE retention
97 % of BERT-base
40% fewer params, 60% faster
The canonical compression datapoint for response+feature distillation on encoders.
arxiv.org
TinyBERT-4L GLUE retention
96.8 % of BERT-base
7.5x smaller, 9.4x faster
Adds attention-matrix and hidden-state losses on top of logit matching.
arxiv.org
DeepSeek-R1-Distill-Qwen-32B, AIME 2024
72.6 % pass@1
pure SFT on 800k R1 traces, no RL stage
Shows sequence-level reasoning-trace distillation alone can transplant frontier reasoning into a 32B open-weight student.
huggingface.co
Alpaca teacher-data generation cost
500 USD (upper bound)
52k instructions; <$100 more to finetune
Stanford CRFM reports under $500 of OpenAI API calls plus 3 hours on 8x A100 80GB.
crfm.stanford.edu
s1K reasoning-distillation dataset size
1,000 examples
beats o1-preview on AIME24/MATH by up to 27%
1,000 curated questions with traces distilled from Gemini Thinking Experimental, plus budget forcing at inference.
arxiv.org
On-policy distillation vs RL, GPU-hours
1,800 GPU-hours
vs 17,920 for RL at a lower score
Qwen3-8B student: 74.4% AIME'24 via on-policy distillation at 1,800 GPU-hours vs 67.6% via RL at 17,920.
thinkingmachines.ai
Latent Consistency Model training cost
32 A100 GPU-hours
50 steps -> 2-4 steps at 768x768
Diffusion distillation is the cheapest high-leverage distillation in this library.
arxiv.org
The first question is not 'which loss' but 'what does the teacher let you see'
Every method in this library sits on one side of a hard line. White-box methods need per-token logits or internal activations and are only available if you host the teacher's weights. Black-box methods need only sampled text and work against any API. That single constraint eliminates roughly half the catalogue before you compare anything else. If you are distilling from a commercial API you are restricted to sequence-level, chain-of-thought, preference and dataset methods — no soft targets, no feature hints, no reverse-KL.
Sources arxiv.org · arxiv.org
Soft targets carry more information per example than labels — that is the whole original insight
Hinton, Vinyals and Dean's 2015 argument is that a teacher's full probability vector over wrong classes (the 'dark knowledge': a 2 that looks slightly like a 7) has high entropy and therefore more bits per training case and lower gradient variance than a one-hot label. Raising the softmax temperature exposes those relative probabilities. Because gradients through the softened softmax scale as 1/T^2, the soft-target term must be multiplied by T^2 to keep the two objectives balanced when T is tuned.
Sources cs.toronto.edu · arxiv.org
On-policy methods fixed the exposure-bias problem and are now the default frontier recipe
Off-policy distillation trains the student on the teacher's own sequences, so at inference the student meets its own distribution for the first time and compounds errors. GKD (2023) and MiniLLM (2023) fix this by sampling from the student and having the teacher grade those tokens, with a divergence (reverse KL or generalized JSD) that tolerates a student too small to cover the teacher's modes. The Qwen3 technical report (Table 21) reports a Qwen3-8B student at 74.4 on AIME'24 for 1,800 GPU-hours of on-policy distillation versus 67.6 for RL at 17,920, up from 55.0 off-policy; Thinking Machines reproduced the method with the Tinker API and reached roughly 70% AIME'24 from a Qwen3-8B-Base SFT-400K checkpoint.
Sources arxiv.org · arxiv.org · arxiv.org · thinkingmachines.ai
Rationales beat labels: distilling the reasoning is worth more than distilling the answer
Distilling Step-by-Step extracts teacher rationales as a second supervised task alongside the label, and reports a 770M T5 outperforming a few-shot-prompted 540B PaLM using only 80% of the available data — roughly a 700x parameter reduction. DeepSeek pushed the same idea to its limit by fine-tuning Qwen and Llama students on 800k long chain-of-thought traces from R1, with no RL stage, reaching 72.6% pass@1 on AIME 2024 at 32B.
Sources arxiv.org · huggingface.co
Distillation is not free: scaling laws say it only wins when the teacher cost is amortised
Apple's 2025 distillation scaling law finds that when a teacher already exists, or when one teacher serves many students, distillation beats supervised learning up to a compute level that scales predictably with student size. But if you must train a teacher to produce exactly one student, plain supervised learning is generally preferable. This is the single most useful economic result in the field and it contradicts the folk assumption that distillation is always the cheaper path.
Sources arxiv.org
Prune first, then distill — compression now routinely stacks
The strongest small-model recipes are compositions, not single methods. NVIDIA's Minitron prunes Llama 3.1 8B along depth or width and re-trains with distillation on under 1% of the original token budget (94B tokens vs 15T, ~160x fewer), yielding 1.8x-2.7x speedups. Sheared LLaMA combines targeted structured pruning with dynamic batch loading to beat same-size models trained from scratch. Quantization-aware distillation then recovers the accuracy that low-bit quantization costs.
Sources arxiv.org · arxiv.org · arxiv.org · arxiv.org
Tokenizer mismatch is the quiet blocker on white-box distillation
Logit-matching requires teacher and student to share a vocabulary, which rules out most cross-family pairs (Llama teacher, Qwen student). Two 2024 lines of work attack this: Universal Logit Distillation uses an optimal-transport cost between sorted probability vectors so no token alignment is needed, and Dual-Space KD projects both models into a shared output space with a cross-model attention mechanism. Both are still less reliable than same-tokenizer KD, and torchtune's own roadmap lists cross-tokenizer support as future work.
Sources arxiv.org · arxiv.org · pytorch.org
Distillation is now infrastructure, not research — and its abuse is now a public legal fight
OpenAI shipped Model Distillation into its API in late 2024 (stored completions plus evals plus fine-tuning), and Amazon Bedrock Model Distillation reached general availability on 1 May 2025 with claims of up to 500% faster and up to 75% cheaper distilled models at under 2% accuracy loss for RAG. The flip side arrived on 23 February 2026, when Anthropic disclosed what it called industrial-scale distillation attacks: roughly 24,000 fraudulent accounts and over 16 million exchanges attributed to DeepSeek, Moonshot AI and MiniMax, targeting agentic reasoning, tool use and coding.
Sources openai.com · aws.amazon.com · anthropic.com
A fine-tuned teacher distills better than a raw one
The PyTorch torchtune case study distilling Llama 3.1 8B into Llama 3.2 1B on Alpaca found that KD loss stayed nearly flat when the teacher was not first adapted to the transfer set, and that using a LoRA-finetuned teacher gave the best hellaswag and commonsense results. NVIDIA independently reports the same thing as 'teacher correction' — a light fine-tune of the teacher on the distillation corpus before pruning-and-distilling. The lesson: the teacher's distribution must match the transfer data or the distillation signal is nearly uninformative.
Sources pytorch.org · arxiv.org
Distillation has escaped model compression entirely
Four of the methods here do not shrink a model at all. Self-distillation (Born-Again Networks) trains a student identical to its teacher and still beats it. Dataset distillation compresses the training set rather than the network. Draft-model distillation (EAGLE, Medusa) trains a tiny head purely to make speculative decoding accept more tokens, leaving the target model's output distribution unchanged. Diffusion distillation compresses the number of sampling steps, not the parameter count — LCM reaches 2-4 step generation for 32 A100-hours.
Sources arxiv.org · arxiv.org · arxiv.org · arxiv.org
Response-based KD (soft targets with temperature)
Response-based · 2015The founding method. Train the student to match the teacher's temperature-softened output distribution, optionally blended with ordinary cross-entropy against ground-truth labels.
When to use it. Use it first whenever you host both models and they share a tokenizer or label set. It is the right baseline for classifiers, encoders, and same-family LLM pairs (Llama 3.1 8B into Llama 3.2 1B), and the right first thing to try before reaching for on-policy methods.
- Difficulty
- 1 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- torchtune knowledge_distillation recipes, Hugging Face TRL, Arcee DistillKit (logit-based mode), NVIDIA TensorRT Model Optimizer, PyTorch (hand-rolled, ~10 lines)
- Examples
- DistilBERT (with an added cosine-embedding term), Gemma 3 pretraining, sampling 256 logits per token weighted by teacher probability, torchtune's Llama 3.1 8B -> Llama 3.2 1B case study
- Strengths
- Simplest possible implementation: one extra forward pass and two lines of loss code; Teacher logits can be precomputed and cached, making the marginal training cost near zero; Architecture-agnostic — the student need share nothing with the teacher but the output space; Works with or without ground-truth labels, so it turns unlabelled data into training signal; Well-understood regularisation effect: soft targets reduce gradient variance and act like label smoothing with structure
- Limits
- Requires teacher logits, so it is unavailable against any closed API; Requires an identical vocabulary or class set; Forward KL is mass-covering: a small student hedges across teacher modes instead of committing, which is exactly wrong for text generation; Off-policy — the student is never corrected on its own generated prefixes; Temperature and alpha need tuning per task and interact with the T² correction in ways that are easy to get wrong
Paper: Distilling the Knowledge in a Neural Network
Feature / intermediate-layer KD (FitNets hints)
Feature-based · 2014Supervise a student's hidden layer directly against a teacher's hidden layer, using a learned projector to bridge the width mismatch, then finish with ordinary response-based KD.
When to use it. Use when the student is much deeper or narrower than the teacher and logit-only KD is failing to converge, and when teacher and student are architecturally similar enough that a layer correspondence is meaningful. In transformer land, prefer the structured version (TinyBERT) over hand-picked hint layers.
- Difficulty
- 3 of 5
- Teacher access
- white-box
- Data needed
- features
- Tools
- RepDistiller, Arcee DistillKit (hidden-states mode), torchvision + custom hooks, NVIDIA Model Optimizer (intermediate-state distillation)
- Examples
- FitNets on CIFAR-10/100 and SVHN, TinyBERT's hidden-state loss, Minitron's intermediate-state distillation during pruning recovery
- Strengths
- Gives dense supervision deep inside the student, which unlocks thin-and-deep architectures that pure logit KD cannot train; Substantially more information transferred than logits alone — features are high-dimensional; Composes cleanly with response-based KD as a two-stage schedule; Effective when teacher and student share an inductive bias (both CNNs, both transformers)
- Limits
- Requires full white-box access to activations, not just logits; Layer-pairing is a hyperparameter with no good default and large effect; The projector adds parameters and its own optimisation dynamics; Poor across architecture families — forcing a transformer student to match CNN features rarely helps; Raw-feature MSE over-constrains: it demands the student reproduce the teacher's coordinate system, not just its information
Paper: FitNets: Hints for Thin Deep Nets
Attention transfer
Attention-based · 2016Instead of matching full activation tensors, match a cheap 2-D summary of where each layer is 'looking' — the channel-collapsed activation energy map — after L2 normalisation.
When to use it. Use as an add-on term when teacher and student are both convolutional or both transformer but differ in width, and when FitNets-style feature matching is proving brittle. In NLP, use it as one component of a TinyBERT-style multi-loss recipe rather than on its own.
- Difficulty
- 2 of 5
- Teacher access
- white-box
- Data needed
- features
- Tools
- szagoruyko/attention-transfer, RepDistiller, TinyBERT (attention-matrix MSE), custom forward hooks
- Examples
- Wide ResNet teacher to thin ResNet student on CIFAR/ImageNet, TinyBERT attention distillation, Cross-modal attention transfer in video models
- Strengths
- Channel-count agnostic, so no projector is needed and width mismatch is free; Cheap: the summary map is H x W, orders of magnitude smaller than the activation tensor; More robust to architecture differences than raw-feature matching; Consistent gains reported across several datasets and CNN families; Transfers an interpretable quantity — you can visualise exactly what is being copied
- Limits
- Lossy: discards all channel-identity information; Weaker standalone effect than logit or feature KD; usually needs to be combined; Still white-box, still needs a layer correspondence (though a coarser one); The power p and the layer-group set are extra hyperparameters; For transformers, per-head attention matching can force spurious head alignment when head counts differ
Paper: Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
Relational KD (RKD)
Relation-based · 2019Transfer the geometry *between* examples — pairwise distances and triplet angles in embedding space — rather than the representation of any single example.
When to use it. Use for embedding models: image retrieval, face and person re-identification, recommendation towers, and sentence encoders — anywhere the deployed operation is a nearest-neighbour search rather than an argmax over classes.
- Difficulty
- 3 of 5
- Teacher access
- white-box
- Data needed
- features
- Tools
- RepDistiller, sentence-transformers (custom losses), PyTorch Metric Learning
- Examples
- Metric learning on CUB-200, Cars-196 and Stanford Online Products, Face-recognition backbone compression, Embedding-tower distillation in retrieval stacks
- Strengths
- Invariant to rotation, reflection and scaling of the embedding space — no coordinate-frame tax; Works when teacher and student embedding dimensions differ, with no projector; Students can exceed their teachers in metric-learning settings; Directly optimises the property that retrieval and clustering actually consume; Composable with logit KD as an extra term
- Limits
- O(B²) and O(B³) terms make large batches costly and small batches statistically noisy; Needs a meaningful embedding layer, so it does not apply to bare classifiers without one; Two loss weights to balance against each other and against the task loss; Less effective than logit KD for plain classification, where absolute class scores are the target; Still white-box
Paper: Relational Knowledge Distillation
Contrastive representation distillation (CRD)
Relation-based · 2019Reframe distillation as maximising the mutual information between teacher and student representations, optimised with a contrastive (InfoNCE-style) objective over positive and negative pairs.
When to use it. Use when you have exhausted logit and simple feature methods and the representation quality itself is the deliverable — cross-modal transfer, ensemble compression, or a backbone that will feed many downstream heads.
- Difficulty
- 4 of 5
- Teacher access
- white-box
- Data needed
- features
- Tools
- HobbitLong/RepDistiller, PyTorch (custom InfoNCE + memory bank)
- Examples
- CIFAR-100 and ImageNet compression benchmarks, RGB-to-depth cross-modal transfer, Ensemble-into-single-model distillation
- Strengths
- Captures higher-order structure and correlations that per-dimension KL discards; Principled: an explicit lower bound on mutual information; Works cross-modally and for ensemble distillation, not just same-modality compression; Strong empirical results; frequently the best single feature-family method in benchmark sweeps; Stacks with response-based KD for further gains
- Limits
- Heaviest implementation in the feature family: memory bank, critic, projection heads; Sensitive to the number of negatives and the temperature of the critic; Extra memory footprint during training; Gains over simpler methods are modest relative to the added complexity for straightforward compression tasks; White-box only
Paper: Contrastive Representation Distillation
Transformer-layer KD (TinyBERT / DistilBERT)
Feature-based · 2019The standard recipe for compressing encoder transformers: simultaneously match embeddings, attention matrices, hidden states and prediction logits, applied at both the pretraining and task-specific stages.
When to use it. Use for encoder models you will deploy at scale — classification, NER, reranking, embeddings — where latency and cost matter and you can afford a two-stage training pipeline. For a quicker win, take DistilBERT's simpler triple loss instead.
- Difficulty
- 4 of 5
- Teacher access
- white-box
- Data needed
- features
- Tools
- huawei-noah/TinyBERT, Hugging Face transformers (DistilBERT), Arcee DistillKit (hidden-states mode), Sentence Transformers
- Examples
- TinyBERT-4L and -6L on GLUE, DistilBERT, DistilRoBERTa, DistilGPT-2, MobileBERT and MiniLM as descendants of the same recipe
- Strengths
- Best-validated recipe for encoder compression, with a decade of production use behind it; Very large speedups at small accuracy cost: 9.4x faster at 96.8% of GLUE for TinyBERT-4L; Attention matching transfers linguistic structure that logits alone do not; Two-stage schedule yields a reusable general student plus a task-tuned one; Reference implementations and pretrained students are widely available
- Limits
- Many interacting hyperparameters: layer mapping, four loss weights, two projectors, two stages; Requires a large unlabelled corpus for the general stage; Attention matching assumes comparable head structure between teacher and student; Less used for modern decoder-only LLMs, where on-policy logit methods dominate; Expensive: two full distillation passes plus data augmentation
Paper: TinyBERT: Distilling BERT for Natural Language Understanding
Sequence-level KD (Kim & Rush)
Sequence-level / data · 2016Replace token-level distribution matching with plain maximum-likelihood training on complete sequences generated by the teacher, typically its beam-search output.
When to use it. Use it as the default whenever the teacher is behind an API, or whenever you want a reusable, auditable training set. It is also the correct first step before any on-policy method, since it gives a strong initialisation cheaply.
- Difficulty
- 1 of 5
- Teacher access
- black-box
- Data needed
- outputs
- Tools
- OpenNMT, Hugging Face TRL (GKDConfig seq_kd=True), vLLM / SGLang for bulk generation, OpenAI Model Distillation (Stored Completions), Amazon Bedrock Model Distillation
- Examples
- WMT English-German NMT students 10x faster than the teacher, Every Alpaca-lineage instruction dataset, Bedrock and OpenAI managed distillation pipelines
- Strengths
- Black-box: needs only sampled teacher text, so it works against any API; Trivial to implement — after generation it is ordinary supervised fine-tuning; Teacher outputs are self-consistent, which removes reference ambiguity and helps small students disproportionately; Often removes the need for beam search at inference, compounding the speedup; Generation is a one-time cost and the resulting dataset is reusable and inspectable
- Limits
- Off-policy: the student never sees its own generated prefixes, so exposure bias remains; Approximating the teacher distribution by its mode discards all diversity and uncertainty; Inherits and can amplify teacher errors, since there is no label to correct them; Generation cost scales with corpus size and output length; Against a commercial API this is the exact activity most terms of service prohibit for competing-model training
Paper: Sequence-Level Knowledge Distillation
Black-box output distillation (Alpaca / Vicuna / Orca style)
Sequence-level / data · 2023Generate a synthetic instruction-following dataset by prompting a strong API teacher, then supervised-fine-tune an open base model on it. The dominant form of distillation actually practised.
When to use it. Use when the teacher is API-only, when you need a task-specific student fast, or when you want a reusable dataset. Pair it with rejection sampling against a verifier (correct answers only) to convert it from style transfer into capability transfer.
- Difficulty
- 1 of 5
- Teacher access
- black-box
- Data needed
- outputs
- Tools
- OpenAI Model Distillation (Stored Completions + Evals + fine-tuning), Amazon Bedrock Model Distillation, Hugging Face TRL SFTTrainer, Axolotl, LLaMA-Factory, vLLM for bulk generation
- Examples
- Stanford Alpaca (52k self-instruct samples from text-davinci-003), Vicuna (70k ShareGPT conversations), Orca and Orca 2 (GPT-4 explanation traces), Zephyr's UltraChat SFT stage
- Strengths
- Works against any teacher on earth, including ones you cannot host; Extremely cheap: Alpaca's full pipeline cost roughly $600; The dataset is a durable, inspectable, filterable asset independent of the student; No tokenizer, architecture or vocabulary constraints; Directly supported as a managed product by OpenAI, AWS Bedrock and Google Vertex
- Limits
- Teaches style far more efficiently than capability — fluent-but-wrong students are the classic failure; One sample per prompt is a very sparse signal compared with per-token logits; Inherits every teacher bias, hallucination and refusal quirk with no correcting label; Off-policy, so exposure bias is untouched; Terms-of-service exposure: this is the exact activity at the centre of the 2025-2026 OpenAI/DeepSeek and Anthropic disputes
Paper: Alpaca: A Strong, Replicable Instruction-Following Model; Orca: Progressive Learning from Complex Explanation Traces of GPT-4
Chain-of-thought / rationale distillation
Rationale / reasoning · 2023Extract the teacher's reasoning text as a second supervised target alongside the label, training the student in a multi-task framework to produce both the rationale and the answer.
When to use it. Use for structured reasoning tasks with checkable answers — arithmetic, NLI, multi-hop QA, code, tool selection — especially when labelled data is scarce and you can afford a modest teacher generation budget.
- Difficulty
- 2 of 5
- Teacher access
- black-box
- Data needed
- outputs
- Tools
- google-research/distilling-step-by-step, Hugging Face TRL SFTTrainer with task prefixes, DSPy for programmatic rationale extraction, OpenAI Model Distillation
- Examples
- 770M T5 beating few-shot 540B PaLM on e-SNLI, ANLI, CQA and SVAMP, Orca's explanation-trace training, Tool-use trace distillation in agent stacks
- Strengths
- Dramatic data efficiency: outperforms a 540B few-shot teacher at 770M using 80% of available data; Rationales are free by-products of a single chain-of-thought API call; Black-box — no teacher internals needed; Rationale generation is training-only, so inference cost is unchanged; Filtering on answer correctness converts noisy rationales into a high-quality corpus
- Limits
- Teacher rationales may not reflect the teacher's true computation (post-hoc rationalisation); Longer teacher outputs mean a materially higher generation bill; The rationale loss weight lambda needs tuning and can dominate the label loss; Works best where reasoning is verbalisable; weak for perceptual or intuitive tasks; Small students can learn to produce plausible rationales that do not actually support their answers
Paper: Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Reasoning-trace distillation (DeepSeek-R1-Distill, s1)
Rationale / reasoning · 2025Supervised fine-tuning on long chain-of-thought traces from a dedicated reasoning teacher, with no reinforcement-learning stage. The single most-copied recipe of 2025.
When to use it. Use when you need frontier-level maths, code or multi-step reasoning in a self-hostable model and you have (or can license) a reasoning teacher plus a verifier. Start with a few thousand rigorously filtered traces before scaling the corpus.
- Difficulty
- 2 of 5
- Teacher access
- black-box
- Data needed
- outputs
- Tools
- Hugging Face TRL SFTTrainer, vLLM / SGLang for high-throughput trace generation, LLaMA-Factory, Open-R1 and OpenThoughts reproduction pipelines, s1 repository
- Examples
- DeepSeek-R1-Distill-Qwen-1.5B/7B/14B/32B and Llama-8B/70B, s1-32B from 1,000 Gemini Thinking traces, OpenThoughts-3 and the open reasoning-dataset ecosystem
- Strengths
- Transplants frontier reasoning into open-weight students with no RL machinery at all; Extraordinary sample efficiency — s1 used 1,000 examples; Black-box: only sampled text is required; Produces permissively licensed artefacts (the R1-Distill Qwen students are MIT); Composes with inference-time controls such as budget forcing for further gains
- Limits
- Very high teacher generation cost: traces are thousands of tokens each and rejection sampling multiplies it; Students inherit teacher verbosity, so inference cost rises even as parameter count falls; Reported to trail same-size RL-trained models on the hardest problems; Only works where answers are verifiable enough to filter on; Legally the most contested technique in the library
Paper: DeepSeek-R1 (Nature, 2025); s1: Simple test-time scaling
Reverse-KL distillation (MiniLLM)
On-policy / divergence-choice · 2023Swap forward KL for reverse KL so the student concentrates on the teacher's dominant modes instead of smearing probability over regions it cannot represent, optimised on the student's own samples.
When to use it. Use when the student is much smaller than the teacher, the task is open-ended generation, and forward-KL distillation is producing bland or incoherent output. Do a cheap off-policy SFT pass first, then switch to reverse KL for the finishing stage.
- Difficulty
- 5 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- trl.experimental.minillm.MiniLLMTrainer, microsoft/LMOps MiniLLM reference code, Tinker cookbook (train_on_policy.py)
- Examples
- MiniLLM students at 120M-13B on instruction following, Thinking Machines on-policy distillation (the alpha_1=1 special case), Qwen3's on-policy strong-to-weak stage
- Strengths
- Correct objective for an under-capacity generative student — sharp outputs instead of blurred averages; Reported lower exposure bias, better calibration and better long-form generation than forward KL; On-policy by construction, so the student is corrected on its own distribution; Scales across model families from 120M to 13B in the original experiments; Now first-class in TRL with a documented trainer and config
- Limits
- Policy-gradient optimisation is high-variance and needs the paper's stabilisers to train at all; Every step requires student generation plus teacher scoring, so it is far more expensive than off-policy KD; Mode-seeking by design: the student will silently drop teacher behaviours, which is bad if you needed coverage; White-box only; The hardest method here to get right from scratch
Paper: MiniLLM: On-Policy Distillation of Large Language Models
Generalized KD (GKD, on-policy)
On-policy / divergence-choice · 2023Train the student on its own generated sequences with teacher feedback on each token, using a beta-interpolated Jensen-Shannon divergence so you can dial continuously between forward and reverse KL.
When to use it. Use as the finishing stage for any generative student where you host the teacher. Bootstrap with sequence-level KD or SFT, then run GKD with lmbda high. This is the current default frontier recipe for small-model post-training.
- Difficulty
- 4 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- trl.experimental.gkd.GKDTrainer, Tinker (on-policy distillation cookbook), torchtune (custom on-policy loop), NVIDIA NeMo Aligner
- Examples
- Qwen3 0.6B-30B strong-to-weak on-policy stage, Qwen3 technical report Table 21 (Qwen3-32B teacher, Qwen3-8B student), Thinking Machines Tinker reproduction (Qwen3-8B teacher, Qwen3-8B-Base student), Gemma post-training distillation from a large instruction-tuned teacher
- Strengths
- Directly attacks exposure bias by correcting the student on its own outputs; One knob (lmbda) spans supervised, sequence-level and fully on-policy distillation; One knob (beta) spans forward KL to reverse KL, so you tune coverage against sharpness; Integrates cleanly with RLHF-style fine-tuning; Roughly 10x GPU-hour savings versus RL at equal or better quality in published runs; Shipped and documented in TRL as GKDTrainer
- Limits
- Student generation every step makes each step slow; the teacher must stay resident in memory; Two extra hyperparameters whose optimum is task-dependent; White-box, same-tokenizer only; Needs a good off-policy starting point or early on-policy samples are too poor to be informative; Known implementation gotcha: Gemma-family models need flash-attn kernels or logits go NaN under soft capping
Paper: On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Speculative knowledge distillation (SKD)
On-policy / divergence-choice · 2024The student proposes tokens and the teacher replaces the poorly ranked ones, producing training sequences that stay near the student's distribution while remaining high quality.
When to use it. Use when on-policy GKD is underperforming because the capacity gap is large and early student samples are too poor for the teacher's feedback to be meaningful. It is the right escalation from GKD, not a starting point.
- Difficulty
- 5 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- Reference implementation from the paper, Custom TRL trainer subclassing GKDTrainer, vLLM for the proposal step
- Examples
- Translation, summarisation, arithmetic and instruction-following experiments in the ICLR 2025 paper
- Strengths
- Gets on-policy relevance without the low-quality-sample problem that hurts pure on-policy KD; Self-annealing: the data distribution shifts from teacher-like to student-like as the student improves, with no schedule; Reported to beat both supervised and on-policy KD across four task families; Robust to different data sizes and student initialisations; Reuses well-understood speculative-decoding machinery
- Limits
- The most complex training loop in this library: interleaved generation plus verification plus loss; Highest per-step cost — both models active at every token; Top-K rejection threshold is a new hyperparameter with limited published guidance; White-box and same-tokenizer only; No first-class trainer in mainstream libraries yet; expect to implement from the paper
Paper: Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling
Self-distillation (Born-Again Networks)
Response-based · 2018Distil a trained model into a student with an identical architecture and parameter count. The student reliably beats the teacher, and repeating the process compounds the gain.
When to use it. Use when you have spare training compute, cannot change the deployed architecture, and want a few points of accuracy — or when you need to collapse an ensemble. Also the right mental model for any self-improvement loop on filtered self-generated data.
- Difficulty
- 2 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- Any standard training framework, torchtune / TRL (teacher = a copy of the model), timm for vision
- Examples
- Born-Again DenseNets and ResNets on CIFAR-10/100, Ensemble consolidation in production vision stacks, Iterated rejection-sampling fine-tuning in LLM post-training
- Strengths
- Free accuracy with no change to deployed architecture, latency or memory; No smaller student to design, no capacity gap to manage; Ensembling successive generations gives further gains; Provides a principled way to collapse an ensemble into a single model; The mechanism underlying iterated self-improvement and rejection-sampling fine-tuning loops
- Limits
- Each generation is a full training run — expensive for modest gains; Returns diminish quickly after two or three rounds; Risk of confirmation loops: the model's own biases are reinforced across generations; Gains are far smaller than what pruning or true compression buys you in deployment terms; The mechanism is still not fully explained, which makes it hard to predict where it will help
Paper: Born Again Neural Networks
Teacher-assistant KD (TAKD)
Response-based · 2019Insert one or more intermediate-sized models between a very large teacher and a very small student, distilling in a chain, to fix the capacity-gap failure.
When to use it. Use when the teacher-to-student parameter ratio is very large (say 20x or more) and direct distillation is measurably worse than label training. Try a mode-seeking divergence first; reach for a chain when that is not enough or unavailable.
- Difficulty
- 2 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- imirzadeh/Teacher-Assistant-Knowledge-Distillation, Any KD framework applied iteratively, torchtune / TRL for LLM size ladders
- Examples
- CIFAR and ImageNet CNN/ResNet chains in the original paper, Frontier-model size ladders (very large -> mid -> small), Orca's use of ChatGPT as teacher assistant alongside GPT-4
- Strengths
- Directly addresses a documented, reproducible failure mode of naive KD; Simple to implement — it is just KD applied twice; Backed by theoretical analysis and experiments on CIFAR-10/100 and ImageNet; The intermediate models are themselves useful deployable artefacts at other price points; Generalises naturally to longer chains when the gap is very large
- Limits
- Training cost is linear in the number of intermediates; Choosing the assistant's size is empirical, with no closed-form rule; Errors compound down the chain — each hop loses a little; Often superseded by simply choosing a better divergence (reverse KL / JSD) at one hop; Rarely worth it unless the size ratio is extreme
Paper: Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher
Multi-teacher KD
Response-based · 2017Distil from several teachers at once, combining their output distributions (or features) into a single target for the student.
When to use it. Use when you have several strong models with complementary strengths and want one deployable student. Prefer the black-box best-of-K data variant with a verifier or judge unless you specifically need per-token logit blending.
- Difficulty
- 3 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- Custom PyTorch loops, Arcee DistillKit, vLLM for multi-model bulk generation, LLM-as-judge pipelines for candidate selection
- Examples
- Acoustic-model ensembles distilled into one recogniser, Margin-MSE's 3-teacher BERT-cat ensemble for retrieval, Open instruction datasets mixing outputs from several frontier models
- Strengths
- Captures ensemble-level accuracy at single-model inference cost; Averaging reduces the variance and idiosyncratic bias of any single teacher; Lets you combine specialists — a code teacher, a maths teacher, a safety teacher; The black-box data variant needs no tokenizer alignment and works with API teachers; Teacher logits can be precomputed once and reused across student runs
- Limits
- Uniform averaging destroys the diversity that motivated the ensemble; Conflicting teachers produce compromise targets that can be worse than any teacher; K forward passes or K cached logit sets: K times the compute or storage; Adaptive weighting adds real complexity and its own hyperparameters; White-box form requires a shared vocabulary across all teachers
Paper: Efficient Knowledge Distillation from an Ensemble of Teachers (Fukuda et al., Interspeech 2017); weighting variants surveyed in Knowledge Distillation: A Survey (Gou et al., 2020, arXiv 2006.05525)
Deep mutual learning (online distillation)
Response-based · 2017Train a cohort of untrained peer networks simultaneously, each learning from the labels and from the others' predictions. No pretrained teacher exists at any point — every peer is white-box to every other.
When to use it. Use when you must train from scratch in a domain with no strong pretrained teacher — specialised scientific, industrial or medical models — and you can afford to train a small cohort. Also useful when you want a large and a small model co-trained in one run.
- Difficulty
- 2 of 5
- Teacher access
- none
- Data needed
- logits
- Tools
- Custom PyTorch training loops, timm (multi-model harness), MMPretrain / mmrazor
- Examples
- CIFAR-100 and Market-1501 person re-identification in the original paper, Online distillation variants in production vision stacks
- Strengths
- No pretrained teacher required — usable when none exists for your domain; Outperforms distillation from a stronger static teacher in the original experiments; Peers may have different architectures, so you can co-train a deployable small model with a large one; Produces an ensemble as a by-product if you want to keep all peers; Empirically finds flatter, better-generalising minima
- Limits
- Trains K models to deploy one — K times the compute and memory; Cohorts can collectively converge on a shared error with no external correction; Cohort size and mimicry weight are extra hyperparameters; Harder to reason about and debug than a fixed teacher-student pipeline; Rarely used at LLM scale, where training two frontier models in lockstep is impractical
Paper: Deep Mutual Learning
Pruning + distillation (Minitron, Sheared LLaMA)
Compression-composed · 2023Structurally prune a large model down to a target shape, then use distillation from the unpruned original to recover the lost accuracy on a small fraction of the original token budget.
When to use it. Use when you own or can license the teacher weights and need a specific smaller architecture for a hard latency or memory budget. This is how you build a model family from a single large model rather than training each size independently.
- Difficulty
- 5 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- NVIDIA TensorRT Model Optimizer, NVIDIA NeMo (pruning + distillation recipes), princeton-nlp/LLM-Shearing, torch.nn.utils.prune for the basics
- Examples
- Llama-3.1-Minitron-4B (width and depth variants), Mistral-NeMo-Minitron-8B from Mistral NeMo 12B, Sheared-LLaMA-1.3B and -2.7B from LLaMA2-7B, Llama 3.2 1B/3B, which used logits from Llama 3.1 8B and 70B to recover after pruning
- Strengths
- Retains the teacher's learned weights instead of starting over — the reason it needs under 1% of the original tokens; Real deployment wins: 1.8x-2.7x measured speedups for Llama-3.1-Minitron-4B; Lets you hit an exact target architecture and latency budget; Sheared LLaMA students beat same-size models trained from scratch; Open weights and reference recipes exist for both lineages
- Limits
- The most complex pipeline here: importance scoring, pruning, teacher correction, distillation retraining; Requires full teacher weights and a large re-training corpus; Aggressive pruning can remove narrow capabilities that aggregate benchmarks will not reveal; Depth versus width is a genuine trade (latency versus quality) with no universal answer; Needs substantial GPU memory: teacher and student resident together over billions of tokens
Paper: Compact Language Models via Pruning and Knowledge Distillation; LLM Pruning and Distillation in Practice: The Minitron Approach; Sheared LLaMA
Quantization-aware distillation (QAD / LLM-QAT)
Compression-composed · 2023Train a low-bit student with quantization simulated in the loop while distilling from the full-precision original, recovering the accuracy that post-training quantization loses.
When to use it. Use as the last step before deployment when you are quantizing below 8 bits and post-training quantization has cost you more accuracy than you can accept, particularly on edge hardware or long-context serving where the KV cache dominates.
- Difficulty
- 4 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- NVIDIA TensorRT Model Optimizer, facebookresearch/LLM-QAT, torchao / torch.ao.quantization, torchtune QAT recipes, Intel Neural Compressor
- Examples
- LLM-QAT 4-bit weight/activation/KV-cache LLaMA models, NVIDIA NVFP4 accuracy-recovery pipelines, On-device deployment of Minitron- and Gemma-class students
- Strengths
- Recovers most of the accuracy lost by post-training quantization, especially below 8 bits; Data-free variant needs no access to the original training corpus; Quantizing the KV cache unlocks long-context throughput that weight-only quantization does not; Applies to any generative model independent of its training data; Stacks directly on top of pruning-and-distillation as the final compression step
- Limits
- Training with fake-quantize ops is slower per step than normal training; Straight-through gradients are biased, so optimisation is noisier; 2-3 bit regimes are still an open problem despite steady progress; The realised speedup depends on kernel support for the chosen format, not on the bit-width alone; Recovers accuracy but teaches nothing new — it cannot fix a weak model
Paper: LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
Dataset distillation / condensation
Dataset / context · 2018Hold the model fixed and compress the *training set* into a tiny synthetic set that trains a network to comparable accuracy. The data, not the network, is the thing distilled.
When to use it. Use for neural architecture search proxies, continual-learning replay buffers, and federated or data-constrained settings — not as a route to a better production model. Treat it as data-axis compression, orthogonal to every other method here.
- Difficulty
- 5 of 5
- Teacher access
- none
- Data needed
- none
- Tools
- VICO-UoE/DatasetCondensation, DC-BENCH, torchvision + custom bi-level loops
- Examples
- MNIST and CIFAR-10 condensed to a handful of images per class, Proxy datasets for architecture search, Condensed replay buffers in continual learning
- Strengths
- Extreme compression of the data axis — orders of magnitude fewer examples; Makes architecture search and hyperparameter sweeps dramatically cheaper via proxy datasets; Natural fit for continual-learning replay buffers and federated settings; Synthetic examples are not literal records, which helps (informally) with data-sharing constraints; Reusable: once condensed, the set trains many models
- Limits
- Bi-level optimisation is expensive and memory-hungry; differentiating through training steps does not scale naively; Distilled sets are often architecture-specific and transfer poorly; Scaling beyond small image benchmarks remains an open problem; Synthetic examples are uninterpretable, so you cannot audit what the set actually encodes; Privacy protection is empirical, not a formal guarantee
Paper: Dataset Distillation
Preference distillation / RLAIF
Preference · 2023Distil judgement rather than answers: a teacher model ranks candidate completions, those rankings train a reward model, and the student is optimised against it — human annotation replaced by AI feedback.
When to use it. Use for alignment, tone, helpfulness and safety behaviours where 'better' is a judgement call rather than a verifiable fact. If you do not need an explicit reward model, go straight to dDPO — it is far simpler for most of the benefit.
- Difficulty
- 4 of 5
- Teacher access
- black-box
- Data needed
- outputs
- Tools
- Hugging Face TRL (RewardTrainer, PPOTrainer, GRPOTrainer), OpenRLHF, NVIDIA NeMo Aligner, alignment-handbook, LLM-as-judge harnesses
- Examples
- Anthropic Constitutional AI / RLAIF, Zephyr-7B's AI-ranked preference stage, UltraFeedback and the open AI-feedback dataset ecosystem
- Strengths
- Removes the human-annotation bottleneck, so preference data scales with compute; Ranking is an easier task than generation, so teacher preferences are often more reliable than teacher completions; The governing principles are explicit and auditable, unlike implicit annotator preferences; Black-box: only the teacher's judgements are needed; Zephyr-7B beat Llama2-Chat-70B on MT-Bench with no human labels
- Limits
- The student inherits the teacher's values, biases and blind spots without correction; Reward models are exploitable; policies find degenerate high-reward behaviours; Full RLAIF with PPO is operationally heavy — reward model, policy, reference and value models in memory; Judge models exhibit known artefacts such as position and verbosity bias; Requires a KL anchor to a reference model or the policy drifts off distribution
Paper: Constitutional AI: Harmlessness from AI Feedback; Zephyr: Direct Distillation of LM Alignment
DPO on teacher preferences (dDPO)
Preference · 2023Skip the reward model and the RL loop: optimise the student directly on teacher-ranked preference pairs with a simple classification-style loss.
When to use it. Use as the default alignment stage after any SFT-based distillation. If you cannot afford RLAIF's infrastructure — which is almost everyone — this captures most of the benefit. Escalate to online preference methods only if you can show DPO is plateauing.
- Difficulty
- 2 of 5
- Teacher access
- black-box
- Data needed
- outputs
- Tools
- Hugging Face TRL DPOTrainer, alignment-handbook, Axolotl, LLaMA-Factory, OpenRLHF
- Examples
- Zephyr-7B-beta (dDPO on GPT-4-ranked UltraFeedback), Tulu and Nous-family preference-tuned models, Most open 7B-70B chat models released since late 2023
- Strengths
- No reward model, no value model, no rollouts — a plain supervised loss over a static dataset; Trains in hours on hardware where PPO would not fit; Zephyr-7B beat Llama2-Chat-70B on MT-Bench using only AI-ranked preferences; Black-box: needs teacher rankings, not teacher internals; First-class support in TRL and the alignment-handbook, with well-trodden recipes
- Limits
- Off-policy — cannot correct behaviours that emerge during training but are absent from the pair set; Sensitive to beta and to the choice of reference model; Can lower the chosen response's probability as long as the rejected one falls faster, degrading both; Inherits every bias of the ranking teacher; Needs a good SFT checkpoint first; DPO on a weak base is unreliable
Paper: Direct Preference Optimization: Your Language Model is Secretly a Reward Model; Zephyr: Direct Distillation of LM Alignment
Cross-vocabulary logit KD (ULD, DSKD)
Response-based · 2024Make white-box logit distillation work when teacher and student have different tokenizers, either by comparing sorted probability vectors with an optimal-transport cost or by projecting both models into a shared output space.
When to use it. Use when the best available teacher is in a different model family from your required student and you host both sets of weights. If you do not strictly need per-token signal, black-box sequence-level distillation is simpler and more predictable.
- Difficulty
- 5 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- Nicolas-BZRD/llm-recipes (ULD), songmzhang/DSKD and DSKDv2, Custom TRL trainers
- Examples
- Cross-family distillation experiments in the ULD paper, DSKD's same- and cross-vocabulary LLM benchmarks, Multi-Level Optimal Transport and later cross-tokenizer variants
- Strengths
- Unlocks white-box distillation across model families — a Llama teacher into a Qwen student; ULD needs no token alignment at all and tolerates different vocabulary sizes; DSKD is a general framework covering same-tokenizer and cross-tokenizer cases with one interface; Much denser signal than falling back to black-box sequence-level KD; Lets you pick the best available teacher rather than the best same-family teacher
- Limits
- ULD discards token identity, so some information is provably lost; DSKD's cross-model attention adds trainable parameters and optimisation complexity; Both are consistently reported as less reliable than same-tokenizer KD; Little first-class support in mainstream training libraries; expect to work from the reference repos; Optimal-transport computation adds per-step cost
Paper: Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs; Dual-Space Knowledge Distillation for Large Language Models
Draft-model distillation for speculative decoding (EAGLE, Medusa)
Trajectory / sampler · 2024Distil a tiny draft head from a frozen target model so that speculative decoding accepts more proposed tokens. The target model's output distribution is provably unchanged; only latency falls.
When to use it. Use whenever you self-host a large model for latency-sensitive interactive serving at low to moderate batch size. It is a pure win with no quality cost, which makes it unusual in this library.
- Difficulty
- 4 of 5
- Teacher access
- white-box
- Data needed
- features
- Tools
- vLLM (EAGLE / EAGLE-3 support), SGLang, SafeAILab/EAGLE, FasterDecoding/Medusa, NVIDIA TensorRT-LLM
- Examples
- EAGLE on LLaMA2-Chat 7B/13B/70B, EAGLE-3 with multi-layer feature fusion, Medusa heads on 7B coding models, where the speedups are largest
- Strengths
- Provably lossless: the verification rule preserves the target model's exact output distribution; Large real speedups — 2.7x-3.5x latency on LLaMA2-Chat 70B with doubled throughput for EAGLE; The draft head is tiny, so training is cheap and the memory overhead at serving is small; Supported out of the box in vLLM and SGLang; Medusa needs no separate draft model at all; Orthogonal to every other method here — stack it on top of a pruned, quantized, distilled student
- Limits
- Acceptance rate is workload-specific; a drafter tuned on chat underperforms on code or long-context; Tree attention and dynamic draft trees are non-trivial to implement correctly; Benefits shrink at large batch sizes where serving is compute-bound rather than bandwidth-bound; Requires white-box access to the target's hidden states to train the drafter; Adds a second artefact to version, evaluate and keep in sync with the target model
Paper: EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty; Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
Retriever distillation (cross-encoder into bi-encoder)
Relation-based · 2020Distil a slow, accurate cross-encoder ranker into a fast bi-encoder or late-interaction model by matching relevance-score margins, making high-quality dense retrieval affordable at web scale.
When to use it. Use whenever you are building dense retrieval or RAG at a scale where a cross-encoder cannot run over the corpus. Distil for first-stage recall, and keep the cross-encoder as a reranker over the top few dozen results.
- Difficulty
- 3 of 5
- Teacher access
- white-box
- Data needed
- outputs
- Tools
- sentence-transformers (MarginMSELoss), sebastian-hofstaetter/neural-ranking-kd, ColBERT / RAGatouille, Tevatron, PyTerrier
- Examples
- distilbert-dot-margin_mse-T2-msmarco and the ColBERT variant, RocketQA / RocketQAv2 listwise distillation, Most modern open embedding models' training recipes
- Strengths
- Bridges an architecture gap that no amount of scaling closes — cross-encoder quality at bi-encoder latency; Margin matching removes the score-scale mismatch that breaks naive score MSE; Teacher scores can be precomputed once and reused across many student runs; Needs no human relevance labels — the teacher's scores replace them; Works for both dot-product bi-encoders and late-interaction models such as ColBERT
- Limits
- Teacher scoring over candidate pairs is expensive; listwise variants multiply it further; Hard-negative mining is essential and is itself a tuning problem; Margin loss discards absolute calibration, so scores are not comparable across queries; The student inherits the teacher's domain biases, which matters for out-of-domain retrieval; Listwise KL beats margin-MSE in several published comparisons but costs more
Paper: Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (Margin-MSE)
Diffusion distillation (progressive, consistency, LCM, adversarial)
Trajectory / sampler · 2022Compress the number of sampling steps rather than the parameter count: train a student to jump along the denoising trajectory in one step where the teacher took many.
When to use it. Use whenever generation latency is the constraint: interactive canvases, real-time video, on-device image generation. Start with an off-the-shelf LCM-LoRA before training anything yourself.
- Difficulty
- 4 of 5
- Teacher access
- white-box
- Data needed
- features
- Tools
- Hugging Face diffusers (LCM, consistency, Turbo pipelines), openai/consistency_models, Stability AI SDXL-Turbo weights, ComfyUI (LCM samplers)
- Examples
- Progressive distillation of DDIM samplers to 4-8 steps, Consistency models: FID 3.55 one-step on CIFAR-10, Latent Consistency Models and LCM-LoRA for Stable Diffusion, SDXL-Turbo single-step real-time synthesis
- Strengths
- Order-of-magnitude latency wins: 50 steps to 1-4, enabling real-time and on-device generation; Parameter count is usually unchanged, so no serving-memory change; Extremely cheap to train — LCM reports 32 A100 GPU-hours for a 768x768 model; LCM-LoRA style adapters make the speedup portable across fine-tuned checkpoints; Consistency models also permit multi-step sampling, so quality is a runtime dial
- Limits
- Sample diversity falls as step count falls — the classic mode-collapse-adjacent trade; One-step students degrade visibly on complex compositional prompts; Adversarial variants (ADD) inherit GAN training instability; Progressive distillation needs several sequential training rounds; Distilled samplers are tied to their teacher checkpoint and must be redone when it changes
Paper: Progressive Distillation for Fast Sampling of Diffusion Models; Consistency Models; Latent Consistency Models; Adversarial Diffusion Distillation
Context / prompt distillation
Dataset / context · 2021Compile a prompt into the weights: train the model without the context to match the predictions of the same model given the context, so instructions, examples and scratchpad reasoning become behaviour.
When to use it. Use when a long system prompt is a material fraction of your per-request cost or context budget, when you want a persona or protocol to survive without prompt-injection risk, or when you need a small model to behave as if it had a scaffold it cannot afford to carry.
- Difficulty
- 2 of 5
- Teacher access
- white-box
- Data needed
- logits
- Tools
- Hugging Face TRL (SFT against self-generated logits), torchtune, Tinker cookbook prompt-distillation recipe, Custom KL-on-logits training loops
- Examples
- Askell et al.'s fourteen-conversation alignment prompt distilled into weights, Snell, Klein & Zhong's instruction, example and scratchpad internalisation, Knowledge injection via prompt distillation as an alternative to document fine-tuning
- Strengths
- Removes a recurring per-request token cost permanently; Frees context window and reduces time-to-first-token; No separate teacher model needed — the model teaches itself; Internalises instructions, few-shot examples and scratchpad reasoning alike; Lets a small model carry a complex persona or protocol that would not fit in its context
- Limits
- The behaviour becomes fixed; changing it requires retraining rather than editing a prompt; Generalises only as far as the distribution of inputs used during distillation; Risk of catastrophic forgetting if the distillation input set is narrow; Harder to audit than a prompt — the instruction is no longer visible anywhere; Needs white-box access, so it only applies to models you can train
Paper: A General Language Assistant as a Laboratory for Alignment (Askell et al.); Learning by Distilling Context (Snell, Klein & Zhong)
Six representative methods, six axes
scoreThe values plotted in “Six representative methods, six axes”, in score.| Dimension | Response-based KD (soft targets) score | Black-box output distillation score | Reasoning-trace distillation score | Generalized KD (on-policy) score | Prune-then-distill score | Diffusion distillation score |
|---|
| Quality retention | 3 | 3 | 5 | 5 | 4 | 4 |
|---|
| Data efficiency | 3 | 2 | 4 | 5 | 5 | 5 |
|---|
| Implementation simplicity | 5 | 5 | 4 | 2 | 1 | 2 |
|---|
| Works without teacher internals | 1 | 5 | 5 | 1 | 1 | 1 |
|---|
| Low training compute | 4 | 4 | 2 | 2 | 3 | 5 |
|---|
| Breadth of applicability | 4 | 5 | 3 | 3 | 3 | 2 |
|---|
Editorial scores; the URLs listed under sources are the papers whose reported trade-offs these scores summarise, not sources for the numbers themselves. Editorial 1-5 ratings, not measured quantities. They summarise the trade-offs argued in each method's cited paper: 'Works without teacher internals' is 5 for black-box methods and 1 for white-box; 'Low training compute' is scored on the marginal cost of the student run, so reasoning-trace distillation scores low because generating long traces from a frontier teacher dominates the bill.
Sources: arxiv.org · arxiv.org · arxiv.org · arxiv.org · huggingface.co
Distillation methods in this library, by year introduced
methodsThe values plotted in “Distillation methods in this library, by year introduced”, in methods.| Year of the originating paper | Methods introduced methods |
|---|
| 2014 | 1 |
| 2015 | 1 |
| 2016 | 2 |
| 2017 | 2 |
| 2018 | 2 |
| 2019 | 4 |
| 2020 | 1 |
| 2021 | 1 |
| 2022 | 1 |
| 2023 | 8 |
| 2024 | 3 |
| 2025 | 1 |
Counts the 27 entries in extras.methods by the `year` field, which is the year of the originating arXiv preprint (not the conference year — FitNets is 2014 on arXiv, ICLR 2015). The 2023 spike is the instruction-tuning and reasoning-distillation wave; 2024-2025 counts are lower partly because recent work refines these families rather than founding new ones.
Sources: arxiv.org · arxiv.org
Reasoning-trace distillation: AIME 2024 pass@1 by student size
%The values plotted in “Reasoning-trace distillation: AIME 2024 pass@1 by student size”, in %.| DeepSeek-R1-Distill student | AIME 2024 pass@1 % | MATH-500 pass@1 % |
|---|
| Qwen-1.5B | 28.9 | 83.9 |
|---|
| Llama-8B | 50.4 | 89.1 |
|---|
| Qwen-7B | 55.5 | 92.8 |
|---|
| Qwen-14B | 69.7 | 93.9 |
|---|
| Llama-70B | 70 | 94.5 |
|---|
| Qwen-32B | 72.6 | 94.3 |
|---|
All students are plain supervised fine-tunes on 800k reasoning samples curated with DeepSeek-R1 — no reinforcement-learning stage. Figures from the official model card.
Sources: huggingface.co
Off-policy distillation vs RL vs on-policy distillation (Qwen3-8B student)
%The values plotted in “Off-policy distillation vs RL vs on-policy distillation (Qwen3-8B student)”, in %.| Training method | AIME'24 pass@1 % |
|---|
| Off-policy distillation | 55 |
| Reinforcement learning | 67.6 |
| On-policy distillation | 74.4 |
The compute is the real story: the RL result cost 17,920 GPU-hours and the on-policy distillation result cost 1,800 GPU-hours, a roughly 10x reduction for a 6.8-point gain. Both figures come from the Qwen3 technical report (Table 21) and are quoted by Thinking Machines; Thinking Machines' own Tinker reproduction started from a different checkpoint and reached roughly 70% AIME'24.
Sources: thinkingmachines.ai · arxiv.org
Reported inference speedup of distilled students over their teachers
xThe values plotted in “Reported inference speedup of distilled students over their teachers”, in x.| Student (method) | Speedup vs teacher x |
|---|
| DistilBERT (response+cosine) | 1.6 |
| Llama-3.1-Minitron-4B width | 1.8 |
| Llama-3.1-Minitron-4B depth | 2.7 |
| EAGLE draft on Llama2-Chat-70B | 3.1 |
| TinyBERT-4L | 9.4 |
| Kim & Rush NMT student | 10 |
DistilBERT is quoted as '60% faster', rendered here as 1.6x. EAGLE is reported as a 2.7x-3.5x latency speedup ratio on LLaMA2-Chat 70B; 3.1 is the midpoint and is the least precise bar here. These numbers come from different hardware, batch sizes and tasks and are not directly comparable to each other — read them as within-paper claims.
Sources: arxiv.org · arxiv.org · arxiv.org · arxiv.org · arxiv.org
- Knowledge distillation (KD)
- Training a student model to reproduce the behaviour of a teacher model, using the teacher's outputs, internal states or generated data as the supervision signal instead of (or alongside) ground-truth labels.
- Teacher
- The model whose behaviour is being copied. Usually larger, slower or more expensive than the student; occasionally the same size (self-distillation) or even a peer (mutual learning).
- Student
- The model being trained. Its capacity ceiling, not the loss function, is usually what limits how much of the teacher survives the transfer.
- Soft targets
- The teacher's full probability distribution over classes or tokens, as opposed to a one-hot label. Carries information about which wrong answers were nearly right.
- Dark knowledge
- Hinton's informal name for the information in the relative probabilities the teacher assigns to incorrect classes — the part of the teacher's judgement that a hard label throws away.
- Temperature (T)
- A divisor applied to logits before the softmax. Higher T flattens the distribution and exposes small probability differences. Because gradients through the softened softmax scale as 1/T^2, the soft-target loss term is multiplied by T^2.
- Forward KL
- KL(teacher || student). Mass-covering: it punishes the student for assigning low probability anywhere the teacher assigns mass, so an under-capacity student smears probability across modes it cannot represent.
- Reverse KL
- KL(student || teacher). Mode-seeking: it punishes the student only where the student itself puts mass, so a small student concentrates on the teacher's dominant modes instead of hedging. The core of MiniLLM.
- Generalized JSD
- A beta-interpolated Jensen-Shannon divergence used in GKD. Beta = 0 approximates forward KL, beta = 1 approximates reverse KL, and intermediate values trade off between them.
- On-policy distillation
- Sampling training sequences from the student and having the teacher score them token by token, so the student is corrected exactly on the distribution it will actually produce at inference.
- Off-policy distillation
- Training on a fixed corpus of teacher-generated (or human) sequences. Simpler and cheaper, but the student never sees its own mistakes during training.
- Exposure bias
- The compounding error that occurs when an autoregressive model trained only on teacher-forced sequences meets its own generated prefixes at inference. The problem on-policy distillation exists to solve.
- White-box distillation
- Distillation that requires access to teacher internals — logits, hidden states, attention maps or gradients. Only possible when you can run the teacher's weights yourself.
- Black-box distillation
- Distillation from sampled teacher outputs only, as returned by an API. The dominant mode in practice, and the mode governed by API terms of service.
- Sequence-level KD
- Replacing token-level distribution matching with plain maximum-likelihood training on complete sequences generated by the teacher, typically its beam-search output.
- Rationale distillation
- Training the student to reproduce the teacher's reasoning text as an auxiliary target, not just its final answer. Distilling Step-by-Step frames this as a second task in a multi-task objective.
- Reasoning-trace distillation
- Supervised fine-tuning on long chain-of-thought traces from a reasoning teacher, as in the DeepSeek-R1-Distill family (800k samples) and s1 (1,000 samples). No RL stage is required.
- Budget forcing
- An inference-time control from s1 that either cuts off a model's thinking or extends it by appending 'Wait', trading test-time compute for accuracy on a model already distilled on reasoning traces.
- Capacity gap
- The empirical finding that distillation degrades when teacher and student are too far apart in size, motivating teacher-assistant chains and mode-seeking divergences.
- Teacher correction
- Lightly fine-tuning the teacher on the distillation corpus before distilling, so the teacher's distribution matches the transfer data. Named by NVIDIA in the Minitron work and independently confirmed in the torchtune case study.
- Hint layer
- The teacher's intermediate layer chosen as a feature-matching target in FitNets, paired with a learned regressor that maps the narrower student layer into the teacher's dimensionality.
- Attention map
- A spatial energy map derived by summing the absolute powers of a layer's channel activations. Attention transfer matches L2-normalised versions of these between teacher and student.
- Relation-based knowledge
- Structure that lives between examples rather than within one — pairwise distances and triplet angles in embedding space. What RKD transfers.
- Cross-tokenizer distillation
- Distilling between models with different vocabularies. Requires either an optimal-transport cost over sorted probabilities (ULD) or a shared projected output space with cross-model attention (DSKD).
- Dataset distillation
- Compressing a training set into a small synthetic set that trains a model to comparable accuracy, via bi-level optimisation. The model is fixed; the data is the thing being distilled.
- Preference distillation
- Using a teacher (or an AI judge) to rank candidate completions and training the student on those preferences, via a reward model plus RL (RLAIF) or directly (dDPO).
- Speculative decoding
- An exact-inference speedup where a cheap draft model proposes several tokens and the target model verifies them in one pass. Distillation enters as the method for training draft heads (EAGLE, Medusa) that get accepted more often.
- Progressive distillation
- Repeatedly halving a diffusion sampler's step count by training a student to take in one step what the teacher takes in two, compounding to large speedups over several rounds.
- Consistency model
- A generative model trained so that all points on a probability-flow ODE trajectory map to the same origin, permitting one-step sampling. Can be distilled from a diffusion teacher or trained standalone.
- Context distillation
- Compiling a prompt — instructions, few-shot examples or scratchpad reasoning — into model weights by training the model without the context to match its own predictions with the context.
- Distillation attack
- Anthropic's term for coordinated, large-scale extraction of a hosted model's outputs across fraudulent accounts in order to train a competing model, as distinct from a legitimate managed distillation workflow.
Every figure on this page comes from one of these primary sources. Compiled 4 September 2026.