Track 10: Training LLMs, Evaluate, Compress and Ship

Distillation: Teach a Small Model From a Big One

Distil a task from a large model to a small one.

Status: Coming soon AdvancedVideo lessonAbout 13 min Free tools

Part of Train and Fine-Tune Models (43 of 45)

Get notified on YouTube

Course

Part of the free syllabus, not yet in a certificate course.

Udemy: coming soon

Tools

Watch first

In this module

  1. Benchmarks Explained, and lm-evaluation-harness
  2. Evaluate Your Fine-Tune Properly
  3. Quantization: GGUF, GPTQ, AWQ, bitsandbytes
  4. Merge, Export, Run Your Model (and Model Merging)
  5. Distillation: Teach a Small Model From a Big One
  6. Capstone: Fine-Tune for a Real Work Task
  7. Model Merging With mergekit: SLERP, TIES, DARE