Academic view of AI distillation: from dark knowledge to on-policy reasoning transfer

Key figures · 8 figures

Key findings · 10 findings

1. Distillation is now measurably better than RL at instilling reasoning in mid-size models

   DeepSeek's own ablation is the cleanest evidence: applying large-scale RL directly to Qwen2.5-32B (DeepSeek-R1-Zero-Qwen-32B) reached AIME 2024 pass@1 of 47.0 and MATH-500 of 91.6, while plain supervised fine-tuning on 800k traces sampled from the 671B DeepSeek-R1 teacher reached 72.6 and 94.3 on the same base model. Qwen3 reproduced the pattern a few months later with on-policy logit distillation: 74.4 vs 67.6 AIME'24 at one tenth the GPU-hours. The 2025 follow-up by Kim et al. refines the claim — RL with verifiable rewards raises pass@1 but often not pass@k, whereas distillation can raise both when it injects genuinely new knowledge.

   Sources arxiv.org · arxiv.org · arxiv.org

2. The field's centre of gravity moved from off-policy traces to on-policy token-level grading

   Off-policy distillation trains the student on the teacher's own perfect outputs, so errors compound at inference — exposure bias that the 2026 survey by Song and Zheng argues scales roughly with the square of sequence length. On-policy distillation instead samples trajectories from the student and has the teacher score each token, combining RL's distribution match with distillation's dense signal. arXiv mentions went from 38 in 2025 to 359 in the first eight months of 2026, and by mid-2026 Qwen3, DeepSeek-V4 and NVIDIA's Nemotron 3 Ultra (and GLM-5, per a Hugging Face community write-up) had all made it a primary post-training stage.

   Sources arxiv.org · thinkingmachines.ai · huggingface.co

3. Sample efficiency collapsed by three orders of magnitude in a single year

   DeepSeek-R1-Distill used 800,000 curated reasoning traces. Within weeks, Berkeley's Sky-T1 matched o1-preview-class math with 17k traces for under $450, Bespoke-Stratos hit AIME 2024 63.3 with the same 17k budget, Stanford's s1 reached AIME 56.7 and MATH-500 93.0 on 1,000 examples in 26 minutes on 16 H100s, and LIMO reported 63.3/95.6 from 817 samples — roughly 1% of the data used by prior approaches. The academic lesson is that a strong base model already contains most of the reasoning capability; the traces mainly teach a format and a search policy.

   Sources novasky-ai.github.io · huggingface.co · arxiv.org · arxiv.org

4. Retention degrades sharply below ~14B parameters on hard reasoning, but barely at all on easier benchmarks

   Across the DeepSeek-R1-Distill family the same teacher yields very different retention depending on task difficulty and student size. On MATH-500 the 1.5B student already retains 86.2% of the 671B teacher and the 32B student retains 96.9%. On AIME 2024 the 1.5B student retains only 36.2% while the 32B retains 91.0%. Retention is therefore not a property of the method but of the interaction between benchmark difficulty and student capacity — the 'capacity gap' that Cho and Hariharan identified in vision in 2019 and that Kajitsuka et al. revisited for chain-of-thought distillation in April 2026.

   Sources huggingface.co · arxiv.org · arxiv.org

5. Distillation now has a scaling law, and it says teachers can be too strong

   Apple's Distillation Scaling Laws (ICML 2025) is a large-scale controlled study of distillation — students from 143M to 12.6B parameters, teachers spanning a similar range, up to 512B training tokens, figures taken from the paper body and the Apple ML Research write-up rather than the abstract — and produced a law predicting student cross-entropy from the compute split between teacher and student. The headline practical result: distillation beats supervised learning only up to a compute level that scales predictably with student size, and a teacher that is too capable for the student's budget makes things worse rather than better.

   Sources arxiv.org · machinelearning.apple.com

6. Pretraining-time distillation is now standard at the frontier, not just a fine-tuning trick

   Gemma 2's 2B and 9B models replaced next-token prediction with distillation from a larger teacher and were trained on more than 50x the compute-optimal token count; the paper's ablation shows a 2B model trained on 500B tokens scoring 60.3 average from scratch versus 67.7 distilled. Gemma 3 refined this by sampling 256 teacher logits per token, renormalising, and using cross-entropy over that sample. Meta pruned Llama 3.1 8B in one shot and used logits from Llama 3.1 8B and 70B as token-level targets during pretraining of Llama 3.2 1B and 3B. NVIDIA's Minitron showed pruning-plus-distillation needs up to 40x fewer tokens per model than training from scratch.

   Sources arxiv.org · arxiv.org · ai.meta.com · arxiv.org

7. The divergence you minimise determines what kind of student you get

   Classical KD minimises forward KL, which is mode-covering: the student spreads mass over regions the teacher barely visits, which is fine for classification but produces hallucinated low-probability text in generation. MiniLLM (ICLR 2024) swapped in reverse KL, which is mode-seeking, and reported lower exposure bias, better calibration and stronger long-text generation across 120M-13B students. GKD generalised this to a JSD family evaluated on student-sampled sequences, and by 2026 reverse KL over on-policy rollouts had become the default objective in industrial post-training pipelines.

   Sources arxiv.org · arxiv.org · thinkingmachines.ai

8. A small student can beat a much larger teacher when the transferred signal is rationales, not labels

   Distilling Step-by-Step extracted natural-language rationales alongside labels and trained a 770M T5 that outperformed few-shot-prompted 540B PaLM while using only 80% of the available data — a ~700x parameter reduction. Orca (13B) learned from GPT-4 explanation traces and beat Vicuna-13B by over 100% on Big-Bench Hard and 42% on AGIEval. MobileBERT even exceeds the same-size BERT-base baseline on SQuAD (its actual teacher is a custom IB-BERT-LARGE, which it does not beat) (F1 90.0 vs 88.5 on v1.1, 79.2 vs 77.1 on v2.0) at 4.3x smaller and 5.5x faster. The pattern: intermediate supervision, not just the final answer, is what closes the gap.

   Sources arxiv.org · arxiv.org · arxiv.org

9. Distillation has become measurable — and the measurements suggest widespread homogenisation

   The Quantification of Large Language Model Distillation framework (ACL 2025) proposes Response Similarity Evaluation and Identity Consistency Evaluation to estimate how heavily a model was distilled from another, and reports that most well-known closed- and open-source LLMs exhibit high distillation degrees, with base models more distilled than aligned ones. Read alongside Shumailov et al.'s Nature result that recursive training on generated data destroys distribution tails, this defines a genuine research risk: a literature that trains overwhelmingly on frontier-model outputs may be narrowing the diversity it depends on.

   Sources arxiv.org · nature.com

10. Reproducibility improved in 2025-2026 — distillation is one of the few frontier techniques the open literature can actually replicate

    Hugging Face's Open-R1 reproduced DeepSeek's reported MATH-500 result for R1-Distill-Qwen-32B (95.6 vs 94.3 reported, per the Open-R1 repository evaluation table) and released OpenR1-Math-220k and Mixture-of-Thoughts as fully open training data. OpenThoughts ran 1,000+ controlled ablations to build OpenThoughts3-1.2M, whose 7B student beat DeepSeek-R1-Distill-Qwen-7B by 15.3 points on AIME 2025, 17.2 on LiveCodeBench and 20.5 on GPQA Diamond. DeepSeek-R1 itself became the first major open-weight LLM published after independent peer review, in Nature in September 2025.

    Sources github.com · huggingface.co · arxiv.org · nature.com

Charts · 6 charts

Tables · 6 tables

Timeline · 37 events

1. 20 August 2006 research

   Buciluă, Caruana & Niculescu-Mizil: Model Compression

   KDD 2006 paper compresses a large ensemble into a single small neural network by labelling a synthetic transfer set. The first statement of the teacher-student idea; ~2,900 citations.

   Source: dl.acm.org

2. 9 March 2015 research

   Hinton, Vinyals & Dean: Distilling the Knowledge in a Neural Network

   Introduces temperature-softened soft targets and the term 'distillation'. Presented at the NIPS 2014 Deep Learning Workshop; now the field's canonical reference at ~25,900 citations.

   Source: arxiv.org

3. 27 March 2015 research

   FitNets: hints from intermediate layers

   Romero et al. show that matching intermediate feature maps, not just outputs, lets thin deep students beat their teachers. Opens the feature-based branch of the taxonomy.

   Source: arxiv.org

4. 25 June 2016 research

   Kim & Rush: Sequence-Level Knowledge Distillation

   Extends KD to sequence generation by training on teacher-generated output sequences rather than per-token distributions — the direct ancestor of every black-box LLM distillation recipe.

   Source: arxiv.org

5. 27 November 2018 research

   Dataset Distillation

   Wang et al. compress a dataset rather than a model, synthesising a handful of images that train a network to near-full accuracy. Spawns a sub-field that reached ~104 arXiv papers in 2025.

   Source: arxiv.org

6. 2 October 2019 research

   DistilBERT

   Sanh et al. distil BERT during pretraining: 40% smaller, 60% faster, 97% of language-understanding capability retained. The result that made distillation standard practice in NLP.

   Source: arxiv.org

7. 3 October 2019 research

   Cho & Hariharan: On the Efficacy of Knowledge Distillation

   ICCV 2019 paper shows larger teachers are often worse teachers because of capacity mismatch, and that stopping teacher training early mitigates it. First rigorous statement of the capacity gap.

   Source: arxiv.org

8. 6 April 2020 research

   MobileBERT

   A 4.3x smaller, 5.5x faster BERT that runs in 62 ms on a Pixel 4 and beats BERT-base on SQuAD F1 — an early demonstration that a student can exceed its reference on some tasks.

   Source: arxiv.org

9. 1 June 2021 research

   Knowledge Distillation: A Survey (IJCV)

   Gou et al. codify the response/feature/relation and offline/online/self taxonomies that the field still uses. ~4,570 citations.

   Source: arxiv.org

10. 13 March 2023 research

    Stanford Alpaca

    LLaMA-7B fine-tuned on 52k self-instruct demonstrations generated by text-davinci-003 for under $600 total. The moment black-box distillation of a commercial API became a mainstream academic method — and a terms-of-service question.

    Source: crfm.stanford.edu

11. 3 May 2023 research

    Distilling Step-by-Step

    Hsieh et al. extract rationales as an extra supervision signal; a 770M T5 outperforms few-shot 540B PaLM using 80% of the data. The founding paper of chain-of-thought distillation.

    Source: arxiv.org

12. 5 June 2023 research

    Orca

    Microsoft distils GPT-4 explanation traces into a 13B student that beats Vicuna-13B by >100% on Big-Bench Hard and 42% on AGIEval, establishing trace imitation as a serious method.

    Source: arxiv.org

13. 14 June 2023 research

    MiniLLM: reverse-KL distillation

    Gu et al. replace forward KL with reverse KL to stop the student overestimating low-probability regions of the teacher, and derive an on-policy optimisation for it. Scales 120M-13B.

    Source: arxiv.org

14. 23 June 2023 research

    GKD: on-policy distillation of language models

    Agarwal et al. train on student-generated sequences graded by the teacher, with a generalized JSD family of losses — the template every 2026 industrial pipeline now follows.

    Source: arxiv.org

15. 25 October 2023 research

    Zephyr-7B: distilled DPO

    Hugging Face show alignment itself can be distilled — dDPO on AI-ranked preferences gives MT-Bench 7.34, surpassing Llama2-Chat-70B, with no human annotation.

    Source: arxiv.org

16. 19 July 2024 research

    Minitron: pruning + distillation

    NVIDIA compress Nemotron-4 15B into 8B and 4B using up to 40x fewer training tokens per model and under 3% of the original data, with up to 16% higher MMLU than training from scratch.

    Source: arxiv.org

17. 24 July 2024 research

    Shumailov et al., Nature: model collapse

    Recursive training on generated data irreversibly destroys the tails of the original distribution. The strongest published caution against a purely distillation-fed research ecosystem.

    Source: nature.com

18. 31 July 2024 research

    Gemma 2 makes distillation a pretraining objective

    Google trains the 2B and 9B models with distillation instead of next-token prediction on >50x the compute-optimal token count; the ablation shows 60.3 to 67.7 average improvement at 500B tokens.

    Source: arxiv.org

19. 25 September 2024 product

    Llama 3.2 1B/3B: one-shot pruning plus logit distillation

    Meta prunes Llama 3.1 8B and uses logits from Llama 3.1 8B and 70B as token-level targets during pretraining — distillation as a first-class step in a frontier lab's small-model pipeline.

    Source: ai.meta.com

20. 10 January 2025 research

    Sky-T1-32B-Preview trained for under $450

    Berkeley's NovaSky distils QwQ-32B-Preview traces into Qwen2.5-32B-Instruct in 19 hours on 8 H100s, matching o1-preview on math and coding and releasing data, code and weights.

    Source: novasky-ai.github.io

21. 22 January 2025 research

    DeepSeek-R1 and the R1-Distill series

    Six students (Qwen 1.5B/7B/14B/32B, Llama 8B/70B) fine-tuned on 800k traces from the 671B teacher. The 32B student hits AIME 2024 72.6 vs 47.0 for large-scale RL on the same base. The single most consequential distillation release to date.

    Source: arxiv.org

22. 22 January 2025 research

    Quantification of LLM Distillation

    Response Similarity and Identity Consistency Evaluation give the first quantitative estimate of how distilled a model is, and report high distillation degrees across well-known open and closed LLMs.

    Source: arxiv.org

23. 22 January 2025 research

    Bespoke-Stratos-32B: 17k samples, 47x less data

    Bespoke Labs distil DeepSeek-R1 through a modified Sky-T1 pipeline, generating the dataset in ~1.5 hours and nearly matching R1-Distill-Qwen-32B (AIME 63.3, MATH-500 93.0).

    Source: bespokelabs.ai

24. 28 January 2025 research

    Hugging Face launches Open-R1

    A fully open reproduction effort; its evaluation table later scores DeepSeek-R1-Distill-Qwen-32B at 95.6 on MATH-500 against the 94.3 DeepSeek reported, and the project releases OpenR1-Math-220k and Mixture-of-Thoughts.

    Source: github.com

25. 31 January 2025 research

    s1: 1,000 samples, 26 minutes, 16 H100s

    Muennighoff et al. distil Gemini Flash Thinking traces and add budget forcing; s1-32B reaches AIME 2024 56.7 and MATH-500 93.0, exceeding o1-preview on AIME by up to 27%.

    Source: arxiv.org

26. 5 February 2025 research

    LIMO: 817 samples

    Ye et al. reach AIME24 63.3 and MATH500 95.6 with 1% of the data used by prior approaches, arguing that reasoning is elicited rather than taught.

    Source: arxiv.org

27. 12 February 2025 research

    Apple publishes Distillation Scaling Laws

    Busbridge et al. fit student loss to the teacher/student compute split across 143M-12.6B students and up to 512B tokens, giving compute-optimal recipes and showing distillation's advantage over supervised learning is bounded.

    Source: arxiv.org

28. 12 March 2025 research

    Gemma 3 refines pretraining distillation

    All Gemma 3 sizes are trained with KD by sampling 256 teacher logits per token, renormalising over the sample, and applying cross-entropy — a cheap approximation to full-vocabulary logit matching. (The model shipped 12 March 2025; the technical report followed on 25 March.)

    Source: arxiv.org

29. 14 May 2025 research

    Qwen3 formalises strong-to-weak distillation

    Off-policy trace distillation followed by on-policy logit KL against Qwen3-32B / 235B-A22B teachers, reported at 1,800 GPU-hours vs 17,920 for RL, with AIME'24 74.4 vs 67.6.

    Source: arxiv.org

30. 4 June 2025 research

    OpenThoughts3-1.2M and OpenThinker3-7B

    1,000+ controlled ablations produce a 1.2M-example recipe whose 7B student beats DeepSeek-R1-Distill-Qwen-7B by 15.3 (AIME25), 17.2 (LiveCodeBench) and 20.5 (GPQA Diamond) points.

    Source: arxiv.org

31. 17 September 2025 research

    DeepSeek-R1 published in Nature

    The first major open-weight LLM to appear after independent peer review, with a 64-page reviewer exchange. Legitimises reasoning distillation as peer-reviewed science rather than preprint folklore.

    Source: nature.com

32. 27 October 2025 research

    Thinking Machines Lab popularises on-policy distillation

    A widely-read technical post frames on-policy distillation as RL's relevance plus distillation's dense reward, and reports reaching teacher parity 7-10x faster than RL for Qwen3-8B.

    Source: thinkingmachines.ai

33. 1 April 2026 research

    A Survey of On-Policy Distillation for Large Language Models

    Song and Zheng formalise OPD as minimising an f-divergence over student-sampled trajectories, organise the field along optimisation target / signal source / stabilisation, and connect it to KL-constrained RL. Revised to v4 by June 2026.

    Source: arxiv.org

34. 10 April 2026 research

    Capacity gap revisited for chain-of-thought distillation

    Kajitsuka, Honda and Takase show CoT distillation can make students worse than baseline under the standard evaluation protocol, propose a corrected protocol, and give practical teacher-student pairing rules.

    Source: arxiv.org

35. 26 April 2026 research

    DeepSeek-V4 replaces mixed RL with on-policy distillation

    V4-Pro (1.6T, 49B active) and V4-Flash (284B, 13B active) train independent domain experts, then consolidate them into a single student via on-policy distillation instead of a further multi-domain RL stage.

    Source: arxiv.org

36. 12 June 2026 research

    Nemotron 3 Ultra introduces multi-teacher on-policy distillation

    NVIDIA train more than ten domain-specialised teachers and consolidate them into a 550B/55B-active hybrid Mamba-Transformer student via dense token-level guidance on student rollouts (MOPD).

    Source: arxiv.org

37. 8 July 2026 research

    A community survey reports distillation across the 2026 frontier

    A Hugging Face community post surveying 2026 frontier releases reports that Qwen3, DeepSeek-V4, GLM-5, Nemotron 3 Ultra and Cursor Composer 2.5 use some form of distillation — spanning black-box traces, white-box logits, on-policy grading and self-distillation — and infers from its post-training recipe that Gemma 4 likely does too.

    Source: huggingface.co

Glossary · 22 terms

Knowledge distillation (KD)

Training a compact 'student' model to reproduce the behaviour of a larger 'teacher' model, using the teacher's outputs as the training signal instead of, or alongside, ground-truth labels.

Soft targets / dark knowledge

The teacher's full probability distribution over classes or tokens. The relative probabilities assigned to wrong answers encode similarity structure that one-hot labels throw away — Hinton's term for this is 'dark knowledge'.

Temperature (T)

A divisor applied to logits before the softmax. Higher T flattens the distribution, exposing more of the teacher's relative ordering over low-probability classes. Gradients are typically rescaled by T-squared to keep loss magnitudes comparable.

Response-based / logit distillation

Transferring only the teacher's output distribution. The original and still most common form; requires white-box access to logits.

Feature-based distillation

Matching the student's intermediate activations to the teacher's, usually through a learned projection. Introduced by FitNets as 'hints'.

Relation-based distillation

Transferring the structure between examples — pairwise distances, angles or attention relations — rather than per-example outputs. Relational KD and CRD are the canonical examples.

Sequence-level KD

For generative models, training on complete teacher-generated output sequences treated as hard targets, rather than matching per-token distributions. Kim & Rush, 2016.

White-box vs black-box distillation

White-box distillation needs the teacher's internal logits or activations, so it requires open weights. Black-box distillation uses only the teacher's sampled text, so it works through any API — which is why it dominates the reasoning-distillation literature and why it raises terms-of-service questions.

Offline, online and self-distillation

The three sampling regimes. Offline uses a fixed pretrained teacher; online trains teacher and student together (or trains peers mutually); self-distillation uses the model's own earlier checkpoint or a stronger-context version of itself as the teacher.

On-policy distillation

The student generates the trajectories and the teacher scores each token of the student's own output. Combines RL's on-policy distribution match with distillation's dense per-token signal; the dominant frontier post-training technique of 2026.

Exposure bias

The mismatch between training on the teacher's perfect prefixes and generating your own at inference, so early errors compound. The 2026 on-policy distillation survey argues the resulting error scales roughly with the square of sequence length.

Forward vs reverse KL

Forward KL (teacher-to-student) is mode-covering: the student spreads probability mass over regions the teacher rarely visits, which produces plausible-sounding but unsupported generations. Reverse KL (student-to-teacher) is mode-seeking: the student concentrates on regions the teacher genuinely supports. MiniLLM made reverse KL standard for generative distillation.

Capacity gap

The observation that a teacher can be too strong for a given student: beyond some size ratio, distillation quality degrades rather than improves. Identified by Cho & Hariharan (2019) and revisited for chain-of-thought distillation in 2026.

Chain-of-thought (CoT) distillation

Training a small model on the teacher's step-by-step reasoning traces rather than only its final answers. The mechanism behind Distilling Step-by-Step, Orca, and the entire DeepSeek-R1-Distill wave.

Rationale distillation

A multi-task variant of CoT distillation in which the student is trained to predict both the label and a natural-language rationale, treating the rationale as auxiliary supervision rather than as the output format.

Pruning + distillation

Structurally removing width, depth, attention heads or MLP channels from a trained model, then using the original model as a teacher to recover the lost accuracy. NVIDIA's Minitron and Meta's Llama 3.2 1B/3B both use this pattern.

Draft-model distillation

Aligning a small 'draft' model to a large target model so that speculative decoding accepts more of the draft's proposed tokens. DistillSpec reports 10-45% speedups over standard speculative decoding.

Dataset distillation

Compressing a training set rather than a model: synthesising a tiny set of examples (sometimes one image per class) that trains a network to near-full accuracy. A separate lineage from model distillation, starting with Wang et al. 2018.

Cross-tokenizer distillation

Transferring logit-level knowledge between models with incompatible vocabularies, via optimal-transport losses (ULD), approximate likelihood matching, byte-level interfaces, or learned projection matrices.

Distillation degree

A quantitative estimate of how heavily a model was trained on another model's outputs, measured in the ACL 2025 framework by Response Similarity Evaluation and Identity Consistency Evaluation.

Model collapse

The irreversible loss of distribution tails when generative models are recursively trained on generated data. Established for LLMs, VAEs and Gaussian mixtures by Shumailov et al. in Nature (2024); the structural risk that a distillation-dominated ecosystem runs.

Budget forcing

A test-time control introduced by s1 that either terminates the model's thinking early or extends it by appending 'Wait', letting a distilled student trade compute for accuracy without retraining.

Sources · 48 sources

Every figure on this page comes from one of these primary sources. Compiled 4 September 2026.

1. Distilling the Knowledge in a Neural NetworkarXiv (Hinton, Vinyals, Dean) · 9 March 2015 · paper
2. Model compressionACM KDD (Buciluă, Caruana, Niculescu-Mizil) · 2006 · paper
3. FitNets: Hints for Thin Deep NetsarXiv / ICLR · 27 March 2015 · paper
4. Sequence-Level Knowledge DistillationarXiv / EMNLP · 25 June 2016 · paper
5. DistilBERT, a distilled version of BERTarXiv (Hugging Face) · 2 October 2019 · paper
6. TinyBERT: Distilling BERT for Natural Language UnderstandingarXiv / Findings of EMNLP · 23 September 2019 · paper
7. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesarXiv / ACL · 6 April 2020 · paper
8. MiniLM: Deep Self-Attention DistillationarXiv / NeurIPS · 25 February 2020 · paper
9. On the Efficacy of Knowledge DistillationarXiv / ICCV · 3 October 2019 · paper
10. Knowledge Distillation: A SurveyarXiv / IJCV · 9 June 2020 · paper
11. A Survey on Knowledge Distillation of Large Language ModelsarXiv · 20 February 2024 · paper
12. Distilling Step-by-Step!arXiv / ACL (Google) · 3 May 2023 · paper
13. Orca: Progressive Learning from Complex Explanation Traces of GPT-4arXiv (Microsoft Research) · 5 June 2023 · paper
14. MiniLLM: Knowledge Distillation of Large Language ModelsarXiv / ICLR 2024 · 14 June 2023 · paper
15. On-Policy Distillation of Language Models (GKD)arXiv / ICLR 2024 (Google DeepMind) · 23 June 2023 · paper
16. Zephyr: Direct Distillation of LM AlignmentarXiv (Hugging Face) · 25 October 2023 · paper
17. Compact Language Models via Pruning and Knowledge Distillation (Minitron)arXiv / NeurIPS (NVIDIA) · 19 July 2024 · paper
18. Gemma 2: Improving Open Language Models at a Practical SizearXiv (Google DeepMind) · 31 July 2024 · paper
19. Gemma 3 Technical ReportarXiv (Google DeepMind) · 25 March 2025 · paper
20. Llama 3.2: Revolutionizing edge AI and visionMeta AI · 25 September 2024 · blog
21. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningarXiv (DeepSeek-AI) · 22 January 2025 · paper
22. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learningNature 645(8081) · 17 September 2025 · paper
23. DeepSeek-R1-Distill-Qwen-32B model cardHugging Face (DeepSeek-AI) · 20 January 2025 · docs
24. Sky-T1: Train your own O1 preview model within $450NovaSky, UC Berkeley Sky Computing Lab · 10 January 2025 · blog
25. Bespoke-Stratos: The unreasonable effectiveness of reasoning distillationBespoke Labs · 22 January 2025 · blog
26. Bespoke-Stratos-32B model cardHugging Face (Bespoke Labs) · 22 January 2025 · docs
27. s1: Simple test-time scalingarXiv / EMNLP 2025 (Stanford, UW) · 31 January 2025 · paper
28. LIMO: Less is More for ReasoningarXiv (SJTU / GAIR) · 5 February 2025 · paper
29. Distillation Scaling LawsarXiv / ICML 2025 (Apple) · 12 February 2025 · paper
30. Distillation Scaling Laws (Apple ML Research page)Apple Machine Learning Research · 12 February 2025 · blog
31. Qwen3 Technical ReportarXiv (Qwen Team, Alibaba) · 14 May 2025 · paper
32. OpenThoughts: Data Recipes for Reasoning ModelsarXiv (OpenThoughts consortium) · 4 June 2025 · paper
33. Open-R1: a fully open reproduction of DeepSeek-R1Hugging Face · 28 January 2025 · blog
34. On-Policy DistillationThinking Machines Lab · 27 October 2025 · blog
35. A Survey of On-Policy Distillation for Large Language ModelsarXiv (Song & Zheng) · 1 April 2026 · paper
36. Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical PerspectivearXiv (Kajitsuka, Honda, Takase) · 10 April 2026 · paper
37. DeepSeek-V4: Towards Highly Efficient Million-Token Context IntelligencearXiv (DeepSeek-AI) · 26 April 2026 · paper
38. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningarXiv (NVIDIA) · 12 June 2026 · paper
39. Distillation in 2026 (so far): which frontier models use it and howHugging Face · 8 July 2026 · blog
40. Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained SettingsarXiv (Wani, Page, Dholakia, Ellison) · 28 January 2026 · paper
41. Quantification of Large Language Model DistillationarXiv / ACL 2025 · 22 January 2025 · paper
42. AI models collapse when trained on recursively generated dataNature 631(8022):755-759 · 24 July 2024 · paper
43. Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM ReasoningarXiv (Kim et al., NYU) · 20 May 2025 · paper
44. Alpaca: A Strong, Replicable Instruction-Following ModelStanford CRFM · 13 March 2023 · blog
45. Semantic Scholar Graph API (citation counts)Allen Institute for AI · 4 September 2026 · docs
46. arXiv API user manual (paper-count methodology)arXiv / Cornell University · 4 September 2026 · docs
47. Open-R1 repository (evaluation tables)Hugging Face · 28 January 2025 · docs
48. Contrastive Representation DistillationarXiv (MIT / Google Research) · 23 October 2019 · paper