Quantization: GGUF, GPTQ, AWQ, bitsandbytes

Course

Part of the free syllabus, not yet in a certificate course.

Udemy: coming soon

Tools

• GGUF / llama.cpp quantize (open-source, free)
• AWQ / GPTQ tools (open-source, free)
• bitsandbytes (open-source, free)

Watch first

• Your First Fine-Tune on Colab (Instruction Tuning)

Then try

• Merge, Export, Run Your Model (and Model Merging)

In this module

1. Benchmarks Explained, and lm-evaluation-harness
2. Evaluate Your Fine-Tune Properly
3. Quantization: GGUF, GPTQ, AWQ, bitsandbytes
4. Merge, Export, Run Your Model (and Model Merging)
5. Distillation: Teach a Small Model From a Big One
6. Capstone: Fine-Tune for a Real Work Task
7. Model Merging With mergekit: SLERP, TIES, DARE