Developer · compiled 3 September 2026 · 58 sources

Developer view of AI distillation: tooling, platforms and recipes (2026)

By September 2026 a developer can distill a model three ways: with open libraries (Hugging Face TRL now ships four distillation trainers, plus Arcee DistillKit, torchtune, NVIDIA Model Optimizer/NeMo and an Axolotl KD plugin), with managed cloud pipelines (Amazon Bedrock Model Distillation, Azure Foundry stored completions, and OpenAI's now-sunsetting fine-tuning platform), or with per-token fine-tuning APIs (Together, Fireworks, Databricks) fed by teacher-generated data. The field has converged on on-policy distillation where the student generates and the teacher grades every token; Qwen3 reports this needs roughly 1/10 of the GPU hours of RL for a better 8B model. Open reasoning datasets (OpenThoughts3-1.2M, OpenR1-Math-220k, Bespoke-Stratos-17k) plus cheap QLoRA mean a 7B reasoning student can be trained for well under $200 of GPU time, and DeepSeek's R1-Distill family alone has passed 97 million Hugging Face downloads. Managed options are in flux: OpenAI stops accepting new fine-tuning jobs on 2027-01-06, Azure retires stored completions on 2026-10-15, Bedrock currently lists no Anthropic teacher, and Vertex AI documents distillation only for open-model tuning. Every number below carries a source URL; where a vendor no longer publishes a figure it is marked undisclosed.

Key figures · 8 figures

DeepSeek-R1-Distill family, all-time HF downloads

97,824,225 downloads

2.23M in the last 30 days

Sum of the six official R1-Distill repos (1.5B, 7B, 8B, 14B, 32B, 70B) via the Hugging Face API on 2026-09-03

huggingface.co

vLLM GitHub stars

90,892 stars

queried 2026-09-03

Default serving engine for distilled students and EAGLE-3 draft heads

github.com

Unsloth GitHub stars

75,556 stars

vs 74,553 for LLaMA-Factory

Most-starred fine-tuning library; QLoRA 7B student needs ~5 GB VRAM

github.com

Qwen3-8B: distillation vs RL GPU-hours

1,800 GPU-hours

vs 17,920 for RL-only (≈1/10)

Table 21 of the Qwen3 technical report; distilled model also scored higher on AIME'24/'25

arxiv.org

Cheapest managed LoRA SFT (≤16B student)

0.48 USD per 1M tokens

Together AI; Fireworks $0.50

Full-parameter SFT is $1.00-$1.20/M tokens on the same tiers

together.ai

s1-32B reasoning distillation compute

7 H100 GPU-hours

26 min on 16 H100s, 1,000 samples

Traces from Gemini Flash Thinking; beat o1-preview on AIME24 (56.7 vs 44.6)

arxiv.org

OpenThoughts3-1.2M rows

1,200,000 rows

850k math / 250k code / 100k science

Largest open reasoning-distillation set; teacher QwQ-32B; Apache-2.0

huggingface.co

Days left to start a new OpenAI fine-tuning job

125 days

new orgs blocked since 2026-05-07

Last day for new jobs is 2027-01-06, counted from 2026-09-03; inference on existing fine-tunes continues until base models are deprecated

developers.openai.com

Key findings · 8 findings

  1. On-policy distillation is now the default recipe, and TRL ships it out of the box

    TRL v1.12 offers four distillation trainers: GKDTrainer (generalized JSD, lmbda/beta/seq_kd), DistillationTrainer (on-policy, chunked JSD, vLLM-accelerated generation, tool-use and VLM support), GOLDTrainer (cross-tokenizer via Universal Logit Distillation) and MiniLLMTrainer (reverse-KL policy gradient). Hugging Face's July 2026 survey finds Qwen3, DeepSeek-V4, GLM-5, Nemotron 3 Ultra and MiMo-V2-Flash all use some form of on-policy distillation where the teacher grades the student's own rollouts.

    Sources huggingface.co · huggingface.co · huggingface.co

  2. Distillation is roughly 10x cheaper than RL for small reasoning models

    Qwen3's technical report (Table 21) reports Qwen3-8B reaching AIME'24 74.4 / AIME'25 65.5 with 1,800 GPU-hours of on-policy distillation from Qwen3-32B and Qwen3-235B-A22B teachers on top of an off-policy-distilled checkpoint (the same starting point as the RL run; that checkpoint’s cost is excluded from both figures), versus 67.6 / 55.5 with 17,920 GPU-hours of RL from that checkpoint. Independent open runs corroborate the cheapness: s1-32B used 7 H100-hours, Sky-T1-32B about $450 (152 H100-hours).

    Sources arxiv.org · arxiv.org · novasky-ai.github.io

  3. Managed distillation on the big clouds is shrinking, not growing

    OpenAI is winding down its fine-tuning platform in phases (2026-05-07, 2026-07-02, 2027-01-06); Azure retires stored completions on 2026-10-15; Amazon Bedrock's supported-model table states distillation is not currently available for Anthropic models with no restoration timeline; Google's Vertex AI still documents teacher-to-student distillation fine-tuning, but only for open models (Llama 3.1, Qwen) via the GenAI SDK, while Gemini tuning is limited to supervised, RL and preference tuning. The action has moved to per-token fine-tuning APIs (Together, Fireworks, Databricks) and to open libraries.

    Sources developers.openai.com · learn.microsoft.com · docs.aws.amazon.com · docs.cloud.google.com

  4. A 7B reasoning student fits on one consumer GPU with QLoRA

    Unsloth's published minimums are 5 GB VRAM for a 7B QLoRA run and 41 GB for 70B; a 32B student needs 26 GB (QLoRA) or 76 GB (LoRA 16-bit). Long reasoning traces (8k-16k tokens) raise activation memory well above these floors, so an 80 GB A100/H100 at $2.50-$3.95/h (Modal) or $1.99/h preemptible (Together) is the practical single-GPU tier for R1-style distillation.

    Sources unsloth.ai · modal.com · together.ai

  5. Open reasoning datasets have made the teacher's API bill optional

    Bespoke-Stratos-17k cost about $800 of DeepSeek-R1 calls to generate; OpenR1-Math-220k, OpenThoughts3-1.2M (QwQ-32B traces), Mixture-of-Thoughts and NVIDIA's OpenMathReasoning are all Apache-2.0 or similar and together see hundreds of thousands of downloads a month. OpenThinker3-7B, trained on OpenThoughts3 from Qwen2.5-7B-Instruct, reports AIME25 53.3 and LiveCodeBench 51.7, above DeepSeek-R1-Distill-Qwen-32B on the same dataset card.

    Sources huggingface.co · huggingface.co · huggingface.co

  6. Logit access is the dividing line between recipes

    White-box KD (torchtune forward-KL, DistillKit logit/hidden-state losses, NeMo Model Optimizer logit and intermediate-layer losses, Axolotl's KD plugin consuming vLLM top-k logprobs) needs the teacher's weights or logprobs. Black-box distillation from an API (OpenAI stored completions, Bedrock synthetic data, Bespoke Curator or distilabel pipelines) only needs text, which is why it dominates for closed teachers. GOLD/ULD removes the same-tokenizer restriction that torchtune and Axolotl still carry.

    Sources pytorch.org · github.com · github.com · huggingface.co

  7. Speculative-decoding draft heads are a second, cheaper kind of distillation

    EAGLE-3 trains a one-layer draft head on the target model's hidden features; Red Hat/vLLM report an up to 2.5x headline speedup, with measured latency gains of 1.6x-2.1x, and the May 2026 EAGLE 3.1 release reports 2.03x per-user throughput on Kimi-K2.6 at concurrency 1. SGLang's SpecForge (1.1k stars) and TorchSpec train these heads, and they are served with a single --speculative-config flag, so a distilled student can itself be paired with an even smaller draft.

    Sources vllm.ai · developers.redhat.com · github.com

  8. Pruning plus distillation is the enterprise path to a model family

    NVIDIA's Minitron recipe (teacher correction on 94B tokens, depth or width pruning of Llama-3.1-8B, then distillation on 94B (width) or 1.4T (depth) tokens) is reported alongside NVIDIA’s earlier Nemotron prune-and-distill work, from which the up to 40x fewer training tokens per additional model and 1.8x compute savings for a full family are taken, with the depth-pruned 4B running about 2.7x the throughput of the 8B on H100 under TensorRT-LLM. The same pipeline is exposed through NeMo Framework and Model Optimizer (3.7k stars).

    Sources developer.nvidia.com · docs.nvidia.com

Charts · 8 charts

GitHub stars of the distillation toolchain

stars
The values plotted in “GitHub stars of the distillation toolchain”, in stars.
RepositoryStars
llama.cpp126,910
vLLM90,892
Unsloth75,556
LLaMA-Factory74,553
SGLang33,881
Open-R126,447
PEFT21,626
TRL19,210
lm-eval-harness13,879
Axolotl12,436
torchtune5,802
NVIDIA Model Optimizer3,724
distilabel3,385
lighteval2,534
Bespoke Curator1,722
SpecForge1,147
DistillKit1,052
evalchemy610

Queried via api.github.com on 2026-09-03. Serving engines dwarf training libraries; dedicated distillation toolkits remain niche (DistillKit ~1k).

Sources: github.com · github.com · github.com · github.com

Managed fine-tuning price per 1M training tokens by student size

USD
The values plotted in “Managed fine-tuning price per 1M training tokens by student size”, in USD.
Student size bucketTogether LoRA SFT USDTogether full SFT USDFireworks LoRA SFT USDFireworks full SFT USD
<=16B0.481.20.51
17-69B (T) / 16.1-80B (F)1.53.7536
70-100B (T) / 80-300B (F)2.97.25612

The vendors’ size buckets do not line up, so categories 2 and 3 pair different ranges and are labelled with both: Together (T) buckets are <=16B / 17-69B / 70-100B; Fireworks (F) buckets are <=16B / 16.1-80B / 80-300B (plus $10 LoRA / $20 full above 300B). Bars within those categories are therefore adjacent, not equivalent. For comparison OpenAI charges $1.50 (gpt-4.1-nano), $5 (gpt-4.1-mini) and $25 (gpt-4.1) per 1M training tokens while its platform winds down.

Sources: together.ai · fireworks.ai · developers.openai.com

Closed-model student training price (OpenAI fine-tuning)

USD
The values plotted in “Closed-model student training price (OpenAI fine-tuning)”, in USD.
Student modelUSD per 1M training tokens USD
gpt-4.1-nano1.5
gpt-4o-mini3
gpt-4.1-mini5
gpt-4.125
gpt-4o25

o4-mini RFT is priced at $100/hour rather than per token. Batch pricing halves these rates. No GPT-5.x model is fine-tunable.

Sources: developers.openai.com

Minimum VRAM vs student size (Unsloth)

GB
The values plotted in “Minimum VRAM vs student size (Unsloth)”, in GB.
Student parameters (B)QLoRA 4-bit GBLoRA 16-bit GB
33.58
7519
8622
96.524
117.529
148.533
272264
322676
403096
7041164
8148192
9053212
405237950

Unsloth calls these 'absolute minimum' figures; long reasoning traces and on-policy generation need more.

Sources: unsloth.ai

Reported GPU-hours of distillation runs vs student size

GPU-hours
The values plotted in “Reported GPU-hours of distillation runs vs student size”, in GPU-hours.
PointSeriesStudent parameters (B)GPU-hours (H100-class) GPU-hours
s1-32B, 1K samplesDistillation (SFT on traces)327
Sky-T1-32B, 17K samples, 3 epochsDistillation (SFT on traces)32152
Qwen3-8BDistillation (off- + on-policy)81,800
Qwen3-8B RLRL-only baseline817,920

Only runs whose authors published GPU-hours are plotted; Sky-T1 = 8 H100 x 19 h. Qwen3 figures are GPU-hours as stated in the report without hardware detail.

Sources: arxiv.org · novasky-ai.github.io · arxiv.org

DeepSeek-R1-Distill downloads in the last 30 days by student size

downloads
The values plotted in “DeepSeek-R1-Distill downloads in the last 30 days by student size”, in downloads.
StudentDownloads (30d) downloads
Qwen-1.5B453,242
Qwen-7B364,966
Llama-8B377,151
Qwen-14B396,595
Qwen-32B548,072
Llama-70B87,147

Hugging Face API, 2026-09-03. The 32B student is the most downloaded; the 70B is the least, consistent with developers preferring students that fit one GPU.

Sources: huggingface.co

AIME 2024 pass@1 vs student size (R1-Distill family)

%
The values plotted in “AIME 2024 pass@1 vs student size (R1-Distill family)”, in %.
Student parameters (B)AIME 2024 %GPQA Diamond %
1.528.933.8
755.549.1
850.449
1469.759.1
3272.662.1
707065.2

Returns diminish sharply above 14B for AIME; the 8B Llama student trails the 7B Qwen student, showing base-model choice matters as much as size.

Sources: huggingface.co

Open distillation datasets: downloads in the last 30 days

downloads
The values plotted in “Open distillation datasets: downloads in the last 30 days”, in downloads.
DatasetDownloads (30d) downloads
OpenThoughts-114k109,485
OpenR1-Math-220k105,083
OpenMathReasoning69,484
NuminaMath-1.533,515
Bespoke-Stratos-17k19,434
OpenThoughts3-1.2M19,174
Mixture-of-Thoughts10,735
Llama-Nemotron-PT7,531
s1K-1.17,133
Infinity-Instruct3,929
Magpie-Pro-300K-F3,349
AM-R1-Distilled-1.4M1,793

Hugging Face API, 2026-09-03.

Sources: huggingface.co · huggingface.co

Tables · 7 tables

Managed distillation and fine-tuning platforms (September 2026)

10 rows
Managed distillation and fine-tuning platforms (September 2026) — What each hosted platform actually offers a developer who wants to distill, with published prices where they exist.
PlatformDistillation productTeacher optionsStudent optionsPublished training priceStatus
OpenAIStored completions -> Evals -> fine-tune (dashboard 'Distill')Any OpenAI model called with store=truegpt-4.1, gpt-4.1-mini, gpt-4.1-nano (SFT/DPO); o4-mini (RFT)$25 / $5 / $1.50 per 1M training tokens (4.1 / mini / nano); o4-mini RFT $100/hourWinding down: no new orgs since 2026-05-07; no new jobs after 2027-01-06 developers.openai.com
Amazon BedrockModel Distillation (synthetic data + student fine-tune, one job)Nova Pro, Nova Premier, Llama 3.1 405B, Llama 3.1 70B, Llama 3.3 70BNova Micro/Lite/Pro; Llama 3.1 8B/70B, 3.2 1B/3B, 3.3 70BTeacher calls at on-demand rate (synthesis up to 15k pairs) + student at customization rate (per-model rates not on public page excerpt)GA; Anthropic teachers 'not currently available', no timeline docs.aws.amazon.com
Azure / Microsoft FoundryStored completions -> Distill -> Azure OpenAI fine-tune (classic portal)Any Azure OpenAI model with store=trueAzure OpenAI fine-tunable modelsAzure OpenAI fine-tuning rates (not on this page)Stored completions retire 2026-10-15; min 10 completions learn.microsoft.com
Google Vertex AIDistillation fine-tuning for open models (teacher -> student via GenAI SDK); supervised, RL and preference tuning for GeminiAny supported open/Gemini model used as teacher for open-model distillationLlama 3.1, Qwen open models (distillation); Gemini 2.5/3.5 Flash-Lite/Pro (SFT/RL/preference)undisclosed on tuning overview pageGA; legacy text-model distillation page returns 404 docs.cloud.google.com
Together AIPer-token SFT/DPO API (bring teacher-generated data)Any (you generate data)Open models up to 100B+ (DeepSeek-V4 Flash, GLM-5, Qwen 3.5 priced separately)LoRA SFT $0.48/M (<=16B), $1.50 (17-69B), $2.90 (70-100B); full SFT $1.20/$3.75/$7.25; $4 minimumGA together.ai
Fireworks AIPer-token SFT/DPO, RFT per GPU-hourAny (you generate data)Open models up to >300BLoRA SFT $0.50/M (<=16B), $3 (16-80B), $6 (80-300B), $10 (>300B); full 2x; RFT = GPU rate ($8/h H100 from 2026-09-01)GA; dataset 3 to 3M examples (per fine-tuning docs) fireworks.ai
PredibaseSFT/Turbo LoRA/RFT fine-tuningAny (you generate data)Open modelsundisclosed (predibase.com and docs now redirect to Rubrik)Folded into Rubrik Agent Cloud predibase.com
Databricks Mosaic AIFoundation Model Fine-tuning (DBU-priced)Any (you generate data)Llama 3.x family and others$0.65/DBU; e.g. Llama 3.1 8B ~100 DBU (~$65) per 10M words, Llama 3.3 70B ~225 DBU (~$146)GA databricks.com
ModalServerless GPUs (run TRL/Unsloth yourself)AnyAnyH100 $3.95/h, A100-80GB $2.50/h, L40S $1.95/h, per-second billing, $30/month freeGA modal.com
AnyscaleRay-based post-training (LLaMA-Factory, SkyRL, Ray Train)AnyAnyundisclosed on docs pageGA docs.anyscale.com

Bedrock student-training rates for Nova and Llama 3.x are set at 'model customization' rates but the public pricing page excerpt only shows Llama 2 ($1.49/M tokens, 13B) and gpt-oss-20b ($80/training hour). Together and Fireworks size buckets differ slightly (17-69B vs 16.1-80B).

Sources: developers.openai.com · docs.aws.amazon.com · aws.amazon.com · together.ai · fireworks.ai · docs.fireworks.ai · docs.cloud.google.com · modal.com

Open-source distillation library feature matrix

11 rows
Open-source distillation library feature matrix — Which library gives you which knob. Stars and licenses from the GitHub API on 2026-09-03. — Units: GitHub stars in stars.
LibraryGitHub stars starsLicenseLogit / white-box KDOn-policy KDCross-tokenizerPruningLoRA/QLoRANotes
Hugging Face TRL19,210Apache-2.0Yes (GKD, DistillationTrainer, MiniLLM)Yes (vLLM colocate/server)Yes (GOLD/ULD, experimental)NoYesTool-calling and VLM distillation supported huggingface.co
Arcee DistillKit1,052Apache-2.0Yes (KL, JSD, TVD, ranking, hidden-state MSE/cosine)Online teacher inferenceYes (via mergekit-tokensurgeon embedding surgery)NoYesOffline logit capture compressed to ~300 bytes/token github.com
torchtune5,802BSD-3-ClauseYes (forward KL + CE, kd_ratio)NoNoNoYes (LoRA recipes)knowledge_distillation_single_device / _distributed recipes pytorch.org
NVIDIA NeMo + Model Optimizer3,724Apache-2.0Yes (logit_layers, intermediate_layer_pairs cosine)NoNoYes (depth/width, Minitron)Via NeMoNeMo 2.0 GPT checkpoints only; stars are for NVIDIA/Model-Optimizer docs.nvidia.com
Axolotl12,436Apache-2.0Yes (KD plugin; top-k teacher logprobs in dataset)NoNoNoYeskd_ce_alpha / kd_alpha / kd_temperature; not in main docs index github.com
Unsloth75,556Apache-2.0No native KD lossNon/aNoYes (QLoRA focus)Fastest path for SFT on teacher-generated text; 7B QLoRA in 5 GB unsloth.ai
LLaMA-Factory74,553Apache-2.0No native KD lossNon/aNoYes (2-8 bit QLoRA)100+ models, LLaMA Board UI; used to train Sky-T1 github.com
SGLang SpecForge1,147MITDraft-head on target hidden statesOnline/offline modesn/aNon/aEAGLE3, P-EAGLE, DFlash draft training for speculative decoding github.com
Bespoke Curator1,722Apache-2.0No (black-box data synthesis)Non/aNon/aBuilt Bespoke-Stratos-17k from DeepSeek-R1 in ~1.5 h github.com
Argilla distilabel3,385Apache-2.0No (black-box data synthesis)Non/aNon/aPipelines for synthetic instruction/preference data github.com
Hugging Face Open-R126,447Apache-2.0No (SFT on R1 traces + GRPO)Non/aNoVia TRLOpenR1-Distill-7B: AIME24 52.7, MATH-500 89.0 github.com

'No native KD loss' means the library trains on teacher text (sequence-level KD) but has no logit-matching objective. NVIDIA-NeMo/NeMo redirects to a Speech repo on GitHub (18,382 stars); the LLM distillation code now lives in Model Optimizer.

Sources: huggingface.co · github.com · github.com · github.com · github.com · github.com · github.com

Open datasets for distillation

12 rows
Open datasets for distillation — Teacher-generated corpora a developer can train a student on today. Download counts from the Hugging Face API, 2026-09-03. — Units: Downloads (30d) in downloads; Downloads (all-time) in downloads.
DatasetRowsTeacherDomainLicenseDownloads (30d) downloadsDownloads (all-time) downloads
open-thoughts/OpenThoughts3-1.2M1,200,000QwQ-32BMath 850k / code 250k / science 100kApache-2.019,174247,476 huggingface.co
open-thoughts/OpenThoughts-114k114kDeepSeek-R1Math, code, science, puzzlesApache-2.0109,4851,477,621 huggingface.co
bespokelabs/Bespoke-Stratos-17k16,710DeepSeek-R1 (via Curator)5k code, 10k math, 1k science/puzzleApache-2.019,434321,082 huggingface.co
open-r1/OpenR1-Math-220k220k problems (94k 'default' split), 2-4 traces eachDeepSeek-R1Math (NuminaMath-1.5 problems)Apache-2.0105,083572,723 huggingface.co
open-r1/Mixture-of-Thoughts350k verified tracesDeepSeek-R1Math, code, sciencenot declared on card (science split derives from CC-BY-4.0 Llama-Nemotron)10,735131,591 huggingface.co
AI-MO/NuminaMath-1.5~900k problemsHuman/CoT (base for R1 traces)Competition mathsee card33,515106,494 huggingface.co
nvidia/OpenMathReasoningsee cardDeepSeek-R1 / QwQMath (CoT + tool-integrated)CC-BY-4.069,484335,233 huggingface.co
nvidia/Llama-Nemotron-Post-Training-Datasetsee cardDeepSeek-R1, Qwen, LlamaReasoning + chat + safetyCC-BY-4.07,531106,953 huggingface.co
simplescaling/s1K-1.11,000DeepSeek-R1 (1.1); Gemini Flash Thinking (1.0)Hard math/science questionsMIT7,13378,174 huggingface.co
a-m-team/AM-DeepSeek-R1-Distilled-1.4M1.4MDeepSeek-R1General reasoningCC-BY-NC-4.0 (non-commercial)1,79351,544 huggingface.co
Magpie-Align/Magpie-Pro-300K-Filtered300kLlama-3-70B-Instruct (self-synthesized prompts)General instruction/chatLlama 33,34927,342 huggingface.co
BAAI/Infinity-Instruct7M+ foundational + chat subsetsCompiled + synthesizedGeneral instructionCC-BY-SA-4.03,929119,827 huggingface.co

Row counts and licenses for NuminaMath-1.5, OpenMathReasoning, Nemotron and AM-1.4M are taken from dataset names/cards as listed; verify the card before commercial use. R1-derived traces come from a model released under DeepSeek's MIT licence, but each dataset card sets its own terms: AM-DeepSeek-R1-Distilled-1.4M is CC-BY-NC-4.0 (non-commercial) and Mixture-of-Thoughts declares no licence at all.

Sources: huggingface.co · huggingface.co · huggingface.co · arxiv.org · arxiv.org

Minimum GPU memory by student size (SFT on teacher outputs)

13 rows
Minimum GPU memory by student size (SFT on teacher outputs) — Unsloth's published 'absolute minimum' VRAM for QLoRA (4-bit) and LoRA (16-bit) at default sequence length, mapped to the cheapest serverless GPU that fits. — Units: QLoRA 4-bit VRAM in GB; LoRA 16-bit VRAM in GB; Price in USD/hour.
Student paramsQLoRA 4-bit VRAM GBLoRA 16-bit VRAM GBCheapest fitting GPU (Modal)Price USD/hour
3B3.58T4 16GB0.59 unsloth.ai
7B519T4 (QLoRA) / L4 24GB (LoRA)0.8 unsloth.ai
8B622L4 24GB0.8 unsloth.ai
9B6.524L4 24GB (QLoRA) / L40S 48GB (LoRA)1.95 unsloth.ai
11B7.529L40S 48GB1.95 unsloth.ai
14B8.533L40S 48GB1.95 unsloth.ai
27B2264L40S (QLoRA) / A100-80GB (LoRA)2.5 unsloth.ai
32B2676L40S (QLoRA) / A100-80GB (LoRA)2.5 unsloth.ai
40B3096L40S (QLoRA) / 2x A100-80GB (LoRA)5 unsloth.ai
70B41164L40S/A100-80GB (QLoRA) / 3x H100 (LoRA)11.8 unsloth.ai
81B48192A100-80GB (QLoRA) / 3x H100 (LoRA)11.8 unsloth.ai
90B53212A100-80GB (QLoRA) / 3x H100 (LoRA)11.8 unsloth.ai
405B2379504x H100 (QLoRA) / 12x H100 (LoRA)47.4 unsloth.ai

GPU mapping and hourly price use Modal's per-second rates (T4 $0.59, L4 $0.80, L40S $1.95, A100-80GB $2.50, H100 $3.95) and are the author's mapping of Unsloth's VRAM floors, priced for the LoRA column where it needs the larger GPU. Long reasoning traces (8k-16k tokens) and on-policy generation raise memory well above these floors; on-policy trainers also need memory for the teacher and a vLLM engine.

Sources: unsloth.ai · modal.com

Published distillation runs with disclosed compute or cost

10 rows
Published distillation runs with disclosed compute or cost — What it actually took to produce well-known distilled students, as reported by their authors.
StudentTeacher / dataTraining samplesComputeReported costHeadline result
s1-32B (Qwen2.5-32B-Instruct)Gemini Flash Thinking traces1,000 (s1K)26 min on 16 H100 (7 GPU-hours)undisclosedAIME24 56.7 vs o1-preview 44.6; MATH500 93.0 arxiv.org
Sky-T1-32B-Preview (Qwen2.5-32B-Instruct)QwQ-32B-Preview + GPT-4o-mini reformatting17k19 h on 8 H100 (DeepSpeed ZeRO-3), LLaMA-Factory~$450 (Lambda pricing)AIME24 43.3 vs o1-preview 40.0 novasky-ai.github.io
Bespoke-Stratos-32BDeepSeek-R1 via Bespoke Curator16,710undisclosed (data generation ~1.5 h)~$800 for data generationAIME24 63.3; MATH500 93.0; GPQA-D 58.1 huggingface.co
OpenThinker3-7B (Qwen2.5-7B-Instruct)QwQ-32B (OpenThoughts3-1.2M)1.2Mundisclosed on cardundisclosedAIME25 53.3; HMMT 42.7; LCB 51.7; GPQA-D 53.7 huggingface.co
OpenR1-Distill-7B (Qwen2.5-Math-7B-RoPE-300k)DeepSeek-R1 (Mixture-of-Thoughts)350k8x H100 80GB node; duration undisclosedundisclosedAIME24 52.7; MATH-500 89.0; GPQA-D 52.8; LCB v5 39.4 github.com
DeepSeek-R1-Distill-Qwen-7BDeepSeek-R1800k curatedundisclosedundisclosedAIME24 55.5; MATH-500 92.8; GPQA-D 49.1; LCB 37.6 huggingface.co
Qwen3-8B (strong-to-weak distillation)Qwen3-32B and Qwen3-235B-A22B, off- then on-policyundisclosed1,800 GPU-hours (vs 17,920 RL-only)undisclosedAIME24 74.4 / AIME25 65.5 (RL-only: 67.6 / 55.5) arxiv.org
Llama-3.1-Minitron-4B (pruned from 8B)Llama-3.1-8B (teacher-corrected on 94B tokens)94B tokens (width-pruned) / 1.4T tokens (depth-pruned)undisclosed GPU-hours; 40x fewer tokens than from scratchundisclosed (1.8x family savings claimed)MMLU 60.53 (width); ~2.7x 8B throughput (depth) on H100 developer.nvidia.com
Llama-3.2-1B KD (torchtune case study)LoRA-tuned Llama-3.1-8Balpaca_cleaned1x A100 80GB; duration undisclosedundisclosedcommonsense_qa 0.5717 (kd_ratio 1.0) vs 0.5536 base pytorch.org
gpt-4o-mini distilled from gpt-4o (OpenAI cookbook)gpt-4o stored completions500 wine reviewsOpenAI managedundisclosed (gpt-4o-mini training was $3/M tokens)Validation accuracy 79.33% vs 64.67% base, 79.67% teacher developers.openai.com

Where authors did not publish GPU-hours or dollars the cell says undisclosed rather than an estimate.

Sources: arxiv.org · novasky-ai.github.io · arxiv.org · pytorch.org

DeepSeek-R1-Distill students: size vs score vs adoption

6 rows
DeepSeek-R1-Distill students: size vs score vs adoption — The canonical 2025 reasoning-distillation family, with Hugging Face download counts (API, 2026-09-03). — Units: Params in B; AIME 2024 pass@1 in %; MATH-500 in %; GPQA Diamond in %; LiveCodeBench in %; Downloads (30d) in downloads; Downloads (all-time) in downloads.
ModelParams BAIME 2024 pass@1 %MATH-500 %GPQA Diamond %LiveCodeBench %Downloads (30d) downloadsDownloads (all-time) downloads
DeepSeek-R1-Distill-Qwen-1.5B1.528.983.933.816.9453,24220,845,014 huggingface.co
DeepSeek-R1-Distill-Qwen-7B755.592.849.137.6364,96615,138,582 huggingface.co
DeepSeek-R1-Distill-Llama-8B850.489.14939.6377,15119,567,548 huggingface.co
DeepSeek-R1-Distill-Qwen-14B1469.793.959.153.1396,5959,133,965 huggingface.co
DeepSeek-R1-Distill-Qwen-32B3272.694.362.157.2548,07227,257,094 huggingface.co
DeepSeek-R1-Distill-Llama-70B707094.565.257.587,1475,882,022 huggingface.co

All six are MIT-licensed and were fine-tuned on 800k R1-curated samples. Downloads exclude community GGUF/MLX re-uploads (e.g. unsloth's 7B GGUF adds 27k/month).

Sources: huggingface.co

Evaluation harnesses used to score distilled students

3 rows
Evaluation harnesses used to score distilled students — The three frameworks that appear in nearly every open distillation paper, and what each is good for. — Units: GitHub stars in stars.
HarnessGitHub stars starsLicenseStrengthUsed by
EleutherAI lm-evaluation-harness13,879MITIndustry-standard task library (MMLU, GSM8K, HellaSwag, TruthfulQA); vLLM/HF backendstorchtune KD case study, most model cards github.com
Hugging Face lighteval2,534MIT1000+ tasks incl. AIME24/25, MATH500, GPQA, LCB; sample-level result dumps; serves via HF endpointsOpen-R1, Open-RS github.com
mlfoundations evalchemy610not declared on GitHubReasoning benchmarks (AIME24/25, AMC23, MATH500, LiveCodeBench, GPQADiamond) with multi-GPU data-parallel sharding and completion caching; API models via CuratorOpenThinker, Bespoke-Stratos github.com

Stars from the GitHub API on 2026-09-03. Evalchemy builds on lm-evaluation-harness.

Sources: github.com · github.com · github.com

Timeline · 20 events

  1. product

    Arcee releases DistillKit v0.1

    Open-source logit- and hidden-state distillation toolkit; later powers Virtuoso, SuperNova-Medius and Arcee-Blitz.

    Source: blog.arcee.ai

  2. research

    NVIDIA publishes the Llama-3.1-Minitron prune-and-distill recipe

    Teacher correction on 94B tokens, 50% depth or width pruning, distillation on 94B tokens (width-pruned) or 1.4T tokens (depth-pruned); 40x fewer tokens per extra model.

    Source: developer.nvidia.com

  3. product

    OpenAI launches Model Distillation in the API

    Stored completions, Evals and fine-tuning integrated so gpt-4o outputs can train gpt-4o-mini from the dashboard.

    Source: openai.com

  4. product

    Amazon Bedrock Model Distillation announced (preview)

    One job generates teacher synthetic data (up to 15k pairs) and fine-tunes a smaller student; teacher calls billed at on-demand rates.

    Source: docs.aws.amazon.com

  5. research

    Sky-T1-32B-Preview: an o1-preview-class model for ~$450

    17k QwQ-32B traces, 19 h on 8 H100 with LLaMA-Factory; AIME24 43.3 vs o1-preview 40.0.

    Source: novasky-ai.github.io

  6. product

    DeepSeek-R1 ships six distilled students (1.5B-70B, MIT)

    Fine-tuned on 800k R1-curated samples; the family now exceeds 97M Hugging Face downloads.

    Source: huggingface.co

  7. research

    Bespoke-Stratos-17k released

    Sky-T1 pipeline re-run with DeepSeek-R1 as teacher through Bespoke Curator: ~1.5 h and ~$800 of API calls; Stratos-32B reaches AIME24 63.3.

    Source: huggingface.co

  8. research

    s1: 1,000 samples and 26 minutes on 16 H100s beat o1-preview on AIME24

    Gemini Flash Thinking traces distilled into Qwen2.5-32B-Instruct; introduces budget forcing.

    Source: arxiv.org

  9. research

    Hugging Face Open-R1 publishes OpenR1-Math-220k

    220k NuminaMath-1.5 problems with 2-4 DeepSeek-R1 traces each; the 94k 'default' split is the recommended SFT set.

    Source: huggingface.co

  10. research

    torchtune KD case study: Llama-3.1-8B into Llama-3.2-1B on one A100

    Forward-KL + CE recipe; a LoRA-tuned teacher and kd_ratio 1.0 gave the best commonsense_qa (0.5717 vs 0.5536).

    Source: pytorch.org

  11. research

    Qwen3 report: distillation beats RL at 1/10 the GPU-hours

    Qwen3-8B distilled from 32B/235B teachers in 1,800 GPU-hours vs 17,920 for RL, with higher AIME scores.

    Source: arxiv.org

  12. research

    OpenThoughts3-1.2M and OpenThinker3-7B

    1.2M QwQ-32B traces after 1000+ curation ablations; OpenThinker3-7B reports AIME25 53.3.

    Source: huggingface.co

  13. product

    EAGLE-3 speculative decoding lands in vLLM

    Draft heads trained on target hidden states; up to 2.5x speedups reported.

    Source: developers.redhat.com

  14. product

    Bedrock adds reinforcement fine-tuning for gpt-oss-20b and Qwen3-32B

    Open-weight students trained from small prompt sets with rule-based or AI-judge graders, served via OpenAI-compatible APIs.

    Source: aws.amazon.com

  15. product

    OpenAI begins winding down the fine-tuning platform

    Job creation closed to organizations that never fine-tuned; tightened on 2026-07-02 to orgs without recent fine-tuned inference.

    Source: developers.openai.com

  16. product

    EAGLE 3.1 released with vLLM and TorchSpec

    Up to 2x longer acceptance length than EAGLE-3 in long context; 2.03x per-user throughput on Kimi-K2.6 at concurrency 1.

    Source: vllm.ai

  17. research

    Hugging Face surveys distillation in 2026 frontier models

    Gemma 3/4, DeepSeek-V4, Qwen3, GLM-5, Nemotron 3 Ultra, MiMo-V2-Flash and Cursor Composer 2.5 all use (mostly on-policy) distillation; TRL implements the techniques.

    Source: huggingface.co

  18. market

    Fireworks raises on-demand GPU prices

    H100/H200 from $7 to $8 per hour and B200 from $10 to $13; RFT is billed at these rates.

    Source: fireworks.ai

  19. product

    Azure OpenAI stored completions retire

    Microsoft directs users to the Responses API and Agent Traces; the classic 'Distill' button depends on stored completions.

    Source: learn.microsoft.com

  20. product

    OpenAI stops all new fine-tuning jobs

    Inference on existing fine-tuned models continues until each base model is deprecated.

    Source: developers.openai.com

Glossary · 14 terms

Sequence-level KD (SeqKD)
Fine-tuning the student on text generated by the teacher. Needs no logits, so it works with API teachers; this is what 'distilling R1 traces' means in practice.
Logit (white-box) KD
Matching the student's next-token distribution to the teacher's full distribution with KL/JSD; requires teacher weights or logprobs (torchtune, DistillKit, Axolotl KD plugin, NeMo).
On-policy distillation
The student generates its own completions and the teacher scores each token; fixes train/inference mismatch. Implemented by TRL's GKDTrainer (lmbda), DistillationTrainer, MiniLLM and used by Qwen3, DeepSeek-V4 and others.
Generalized JSD (beta)
Loss interpolating forward KL (beta=0) and reverse KL (beta=1); TRL exposes it as the beta parameter in GKD/Distillation/GOLD configs.
Cross-tokenizer distillation (ULD / GOLD)
Aligning teacher and student tokens by visible text spans and merging probabilities so a Qwen teacher can distill into a Llama student despite different vocabularies.
Stored completions
OpenAI/Azure feature (store=true) that logs prompt-response pairs from a teacher model into a dataset that can be evaluated or used to fine-tune a smaller model.
Teacher correction
Fine-tuning the teacher on the distillation corpus first so its distribution matches the transfer set; NVIDIA used 94B tokens for Minitron and torchtune found a LoRA-tuned teacher lowers KD loss.
Prune-and-distill (Minitron)
Remove layers (depth) or hidden/MLP width from a larger model, then recover accuracy by distilling from the original; yields model families with far fewer training tokens.
Rejection sampling
Keeping only teacher traces whose final answers verify (exact match for math, unit tests for code) before SFT; used by Sky-T1, Bespoke-Stratos and Open-R1.
QLoRA
LoRA adapters on a 4-bit-quantized base model; the reason a 7B student can be trained in ~5 GB of VRAM.
Draft model / EAGLE-3
A tiny model or head trained to predict the target model's tokens for speculative decoding; a form of distillation whose success metric is acceptance length, not benchmark score.
Budget forcing
s1's test-time trick of appending 'Wait' or forcing an end-of-thinking token to control reasoning length in a distilled model.
Data synthesis (Bedrock)
Bedrock's proprietary prompt augmentation that can expand a distillation set to at most 15k prompt-response pairs, billed at the teacher's on-demand rate.
Evaluation harness
A framework (lm-eval-harness, lighteval, evalchemy) that runs standardized benchmarks so a distilled student can be compared with its teacher and with published baselines.

Sources · 58 sources

Every figure on this page comes from one of these primary sources. Compiled 3 September 2026.

  1. Generalized Knowledge Distillation Trainer (TRL docs)Hugging Face · 2026 · docs
  2. Distillation Trainer (TRL docs)Hugging Face · 2026 · docs
  3. GOLD Trainer (TRL docs)Hugging Face · 2026 · docs
  4. MiniLLM Trainer (TRL docs)Hugging Face · 2026 · docs
  5. Distillation in 2026 (so far): which frontier models use it and howHugging Face · 8 July 2026 · blog
  6. arcee-ai/DistillKitArcee AI · 2026 · docs
  7. DistillKit v0.1 technical paperArcee AI · August 2024 · blog
  8. Distilling Llama3.1 8B into 1B in torchtunePyTorch · February 2025 · blog
  9. meta-pytorch/torchtuneMeta · 2026 · docs
  10. How to Prune and Distill Llama-3.1 8B to an NVIDIA Llama-3.1-Minitron 4B ModelNVIDIA · August 2024 · blog
  11. NeMo Framework: Knowledge DistillationNVIDIA · 2026 · docs
  12. NVIDIA/Model-OptimizerNVIDIA · 2026 · docs
  13. Axolotl KD plugin READMEAxolotl AI · 2026 · docs
  14. Unsloth requirements (VRAM table)Unsloth · 2026 · docs
  15. hiyouga/LlamaFactoryLLaMA-Factory · 2026 · docs
  16. OpenAI API deprecations (fine-tuning platform wind-down)OpenAI · 2026 · docs
  17. OpenAI model optimization guideOpenAI · 2026 · docs
  18. OpenAI API pricing (fine-tuning section)OpenAI · 2026 · pricing
  19. Model Distillation in the APIOpenAI · 1 October 2024 · blog
  20. Leveraging model distillation to fine-tune a model (cookbook)OpenAI · October 2024 · docs
  21. Customize a model with distillation in Amazon BedrockAWS · 2026 · docs
  22. Prerequisites for model distillation (supported teacher/student pairs)AWS · 2026 · docs
  23. Amazon Bedrock pricingAWS · 2026 · pricing
  24. Amazon Bedrock reinforcement fine-tuning adds open-weight modelsAWS · 17 February 2026 · news
  25. Stored completions and distillation (Foundry classic)Microsoft · 6 July 2026 · docs
  26. Distillation in Azure AI Foundry (blog)Microsoft · December 2024 · blog
  27. Vertex AI: tune Gemini models (overview)Google Cloud · 2026 · docs
  28. Supervised and distillation fine-tuning for open modelsGoogle Cloud · 3 September 2026 · docs
  29. Together AI pricingTogether AI · 2026 · pricing
  30. Fireworks AI pricingFireworks AI · September 2026 · pricing
  31. Fireworks fine-tuning docsFireworks AI · 2026 · docs
  32. Databricks Foundation Model Fine-tuning pricingDatabricks · 2026 · pricing
  33. Modal pricingModal · 2026 · pricing
  34. Post-training for LLMs on AnyscaleAnyscale · 2026 · docs
  35. Predibase pricing (redirects to Rubrik)Predibase / Rubrik · 2026 · pricing
  36. DeepSeek API models and pricingDeepSeek · 2026 · pricing
  37. DeepSeek-R1-Distill-Qwen-7B model cardDeepSeek · 20 January 2025 · docs
  38. OpenThoughts3-1.2M dataset cardOpen Thoughts · June 2025 · docs
  39. OpenThoughts: Data Recipes for Reasoning ModelsarXiv · June 2025 · paper
  40. Bespoke-Stratos-17k dataset cardBespoke Labs · January 2025 · docs
  41. OpenR1-Math-220k dataset cardHugging Face · February 2025 · docs
  42. huggingface/open-r1Hugging Face · 2026 · docs
  43. Magpie: Alignment Data Synthesis from ScratcharXiv / ICLR 2025 · June 2024 · paper
  44. Infinity Instruct: Scaling Instruction Selection and SynthesisarXiv / BAAI · June 2025 · paper
  45. Sky-T1: Train your own O1 preview model within $450NovaSky (UC Berkeley) · 10 January 2025 · blog
  46. s1: Simple test-time scalingarXiv · 31 January 2025 · paper
  47. Qwen3 Technical ReportarXiv / Alibaba · May 2025 · paper
  48. EleutherAI/lm-evaluation-harnessEleutherAI · 2026 · docs
  49. huggingface/lightevalHugging Face · 2026 · docs
  50. mlfoundations/evalchemyDataComp / Bespoke Labs · 2026 · docs
  51. vllm-project/vllmvLLM · 2026 · docs
  52. EAGLE 3.1: Advancing Speculative Decoding Through CollaborationvLLM · 26 May 2026 · blog
  53. Fly Eagle(3) fly: faster inference with vLLM and speculative decodingRed Hat · 1 July 2025 · blog
  54. sgl-project/SpecForgeSGLang · 2026 · docs
  55. sgl-project/sglangSGLang · 2026 · docs
  56. ggml-org/llama.cppggml · 2026 · docs
  57. bespokelabsai/curatorBespoke Labs · 2026 · docs
  58. argilla-io/distilabelArgilla / Hugging Face · 2026 · docs