Track 9: LLM Engineering and Production
Agent Testing & Evaluation v2 (update of the live DeepEval course)
35 lessons
Track 9: LLM Engineering and Production, Evals in Depth
Golden Datasets and Test Sets That Last
Build a versioned golden set with dev/test splits and edge cases.
Get notified on YouTube Udemy certificate: coming soon
Course
Agent Testing & Evaluation v2 (update of the live DeepEval course)
Udemy: coming soon
Tools
- Hugging Face Datasets (open-source, free)
- DeepEval (open-source, freemium)
Watch first
- Build an Eval Dataset (and Compare Prompts With promptfoo)
- Error Analysis First: Read the Traces, Then Pick Metrics
Then try
- Metrics for LLM Apps: From Code Checks to Rubrics
- Human Review Loops: Annotation Queues and Rubrics
- LangSmith Evaluations: Datasets and Experiments
- Langfuse Evals: Datasets, Experiments, LLM Judges
- Braintrust: Experiments, Scorers and autoevals
In this module
- Eval-Driven Development: Write the Test Before the Prompt
- Error Analysis First: Read the Traces, Then Pick Metrics
- Golden Datasets and Test Sets That Last
- Metrics for LLM Apps: From Code Checks to Rubrics
- Tool-Call and MCP Contract Tests
- Safety Evals: Jailbreaks, Refusals and Over-Refusal
- Human Review Loops: Annotation Queues and Rubrics
- Calibrate Your LLM Judge: Human Agreement and Bias
- Agent Task-Success Evals: End-State Checks and pass^k
- Test Multi-Agent Systems: Handoffs, Loops and Who Broke It
- Flaky Evals in CI: Repeats, Thresholds, Baselines
- Online Evals and A/B Tests on Live Traffic
- Benchmarks and Their Limits: Contamination, Saturation
- Capstone: The Agent Quality Platform
The course this lesson belongs to