Track 9: LLM Engineering and Production
Agent Testing & Evaluation v2 (update of the live DeepEval course)
35 lessons
Track 9: LLM Engineering and Production, Evals in Depth
Safety Evals: Jailbreaks, Refusals and Over-Refusal
Measure unsafe-answer and over-refusal rates on a fixed test set.
Get notified on YouTube Udemy certificate: coming soon
Course
Agent Testing & Evaluation v2 (update of the live DeepEval course)
Udemy: coming soon
Tools
- promptfoo (open-source, free)
- Inspect (UK AI Security Institute) (open-source, free)
- Llama Guard (open-source)
Watch first
In this module
- Eval-Driven Development: Write the Test Before the Prompt
- Error Analysis First: Read the Traces, Then Pick Metrics
- Golden Datasets and Test Sets That Last
- Metrics for LLM Apps: From Code Checks to Rubrics
- Tool-Call and MCP Contract Tests
- Safety Evals: Jailbreaks, Refusals and Over-Refusal
- Human Review Loops: Annotation Queues and Rubrics
- Calibrate Your LLM Judge: Human Agreement and Bias
- Agent Task-Success Evals: End-State Checks and pass^k
- Test Multi-Agent Systems: Handoffs, Loops and Who Broke It
- Flaky Evals in CI: Repeats, Thresholds, Baselines
- Online Evals and A/B Tests on Live Traffic
- Benchmarks and Their Limits: Contamination, Saturation
- Capstone: The Agent Quality Platform
The course this lesson belongs to