Flaky Evals in CI: Repeats, Thresholds, Baselines

Course

Agent Testing & Evaluation v2 (update of the live DeepEval course)

Udemy: coming soon

Tools

• GitHub Actions (freemium)
• DeepEval (open-source, freemium)
• promptfoo (open-source, free)

Watch first

• Regression Gates in CI

Then try

• Online Evals and A/B Tests on Live Traffic

In this module

1. Eval-Driven Development: Write the Test Before the Prompt
2. Error Analysis First: Read the Traces, Then Pick Metrics
3. Golden Datasets and Test Sets That Last
4. Metrics for LLM Apps: From Code Checks to Rubrics
5. Tool-Call and MCP Contract Tests
6. Safety Evals: Jailbreaks, Refusals and Over-Refusal
7. Human Review Loops: Annotation Queues and Rubrics
8. Calibrate Your LLM Judge: Human Agreement and Bias
9. Agent Task-Success Evals: End-State Checks and pass^k
10. Test Multi-Agent Systems: Handoffs, Loops and Who Broke It
11. Flaky Evals in CI: Repeats, Thresholds, Baselines
12. Online Evals and A/B Tests on Live Traffic
13. Benchmarks and Their Limits: Contamination, Saturation
14. Capstone: The Agent Quality Platform