Track 9: LLM Engineering and Production
Agent Testing & Evaluation v2 (update of the live DeepEval course)
35 lessons
Track 9: LLM Engineering and Production, Evals in Depth
Metrics for LLM Apps: From Code Checks to Rubrics
Choose the cheapest metric that catches each failure mode.
Get notified on YouTube Udemy certificate: coming soon
Course
Agent Testing & Evaluation v2 (update of the live DeepEval course)
Udemy: coming soon
Tools
Watch first
Then try
- Component RAG Evals: Retriever, Reranker, Generator
- Tool-Call and MCP Contract Tests
- Safety Evals: Jailbreaks, Refusals and Over-Refusal
- Benchmarks and Their Limits: Contamination, Saturation
- Inspect (UK AISI): Tasks, Solvers and Scorers
- OpenAI Evals: Open-Source Framework and Paid Evals API
In this module
- Eval-Driven Development: Write the Test Before the Prompt
- Error Analysis First: Read the Traces, Then Pick Metrics
- Golden Datasets and Test Sets That Last
- Metrics for LLM Apps: From Code Checks to Rubrics
- Tool-Call and MCP Contract Tests
- Safety Evals: Jailbreaks, Refusals and Over-Refusal
- Human Review Loops: Annotation Queues and Rubrics
- Calibrate Your LLM Judge: Human Agreement and Bias
- Agent Task-Success Evals: End-State Checks and pass^k
- Test Multi-Agent Systems: Handoffs, Loops and Who Broke It
- Flaky Evals in CI: Repeats, Thresholds, Baselines
- Online Evals and A/B Tests on Live Traffic
- Benchmarks and Their Limits: Contamination, Saturation
- Capstone: The Agent Quality Platform
The course this lesson belongs to