Agent Testing & Evaluation v2 (update of the live DeepEval course)
In production, premiere planned 23 January 2027 BeginnerIntermediateAdvanced
Get notified on YouTube Udemy certificate: coming soon
- Udemy
- Coming soon
- Lessons
- 35
- Planned length
- About 8 h 48 min
- Projects
- 1
- Practicals
- Planned: a hands-on practical and a Udemy assignment in every section, plus a capstone
What you will be able to do
- Build a test set and score a RAG app
- Find which RAG stage causes each wrong answer
- Score an agent's steps, not just its answer
- Write your first five eval cases
- Collect cases and run config-driven comparisons
- Write a judge prompt and check it against humans
Syllabus
RAG: Retrieval-Augmented Generation: RAG Evaluation
- RAGAS Tutorial, With a Synthetic Test Set Build a test set and score a RAG app
- Component RAG Evals: Retriever, Reranker, Generator Find which RAG stage causes each wrong answer
AI Agents: Human-in-the-Loop and Guardrails
- Evaluate Agents: Trajectories and Tool Use Score an agent's steps, not just its answer
LLM Engineering and Production: Testing and Evals
- Why Evals Are the Unit Tests of AI Write your first five eval cases
- Build an Eval Dataset (and Compare Prompts With promptfoo) Collect cases and run config-driven comparisons
- LLM-as-Judge, Done Right Write a judge prompt and check it against humans
- Regression Gates in CI Block a release when quality drops
- Evaluating Agents End to End Score tool use, trajectories and outcomes
LLM Engineering and Production: Security
- Red-Team Your AI App: garak and PyRIT Run an automated attack scan and fix findings
LLM Engineering and Production: Evals in Depth
- Eval-Driven Development: Write the Test Before the Prompt Turn a feature idea into pass/fail checks before changing a prompt
- Error Analysis First: Read the Traces, Then Pick Metrics Name your app's top failure modes from 50 real outputs
- Golden Datasets and Test Sets That Last Build a versioned golden set with dev/test splits and edge cases
- Metrics for LLM Apps: From Code Checks to Rubrics Choose the cheapest metric that catches each failure mode
- Tool-Call and MCP Contract Tests Assert the right tool, the right arguments and safe handling of tool errors
- Safety Evals: Jailbreaks, Refusals and Over-Refusal Measure unsafe-answer and over-refusal rates on a fixed test set
- Human Review Loops: Annotation Queues and Rubrics Run a weekly review queue that grows the golden set
- Calibrate Your LLM Judge: Human Agreement and Bias Measure judge-human agreement and fix the biases that skew scores
- Agent Task-Success Evals: End-State Checks and pass^k Score agents on finished tasks, not nice-sounding answers
- Test Multi-Agent Systems: Handoffs, Loops and Who Broke It Catch bad handoffs and runaway loops between agents
- Flaky Evals in CI: Repeats, Thresholds, Baselines Gate releases on real regressions, not random noise
- Online Evals and A/B Tests on Live Traffic Score sampled production traces and run a safe A/B rollout
- Benchmarks and Their Limits: Contamination, Saturation Read a leaderboard critically and build a private benchmark instead
- Capstone: The Agent Quality Platform Ship datasets, metrics, red-team tests, CI gates and live monitoring for one agent
LLM Engineering and Production: Eval Tools (One Lesson Each)
- promptfoo: Your First Eval Suite in YAML Compare prompts and models side by side from one config file
- Inspect (UK AISI): Tasks, Solvers and Scorers Write an Inspect task and read its log viewer
- OpenAI Evals: Open-Source Framework and Paid Evals API Run a graded eval on the framework and on the API (verify price)
- LangSmith Evaluations: Datasets and Experiments Run and compare experiments in LangSmith (paid plans; free tier limits: verify price)
- Langfuse Evals: Datasets, Experiments, LLM Judges Score traces and dataset runs in self-hosted Langfuse
- Braintrust: Experiments, Scorers and autoevals Run experiments with autoevals scorers (paid platform; free tier: verify price)
- Arize Phoenix Evals: Trace First, Then Evaluate Evaluate spans from a traced agent locally
- TruLens: Feedback Functions and the RAG Triad Score context relevance, groundedness and answer relevance
- DeepEval Advanced: Custom Metrics and Agent Tracing Write custom metrics and trace-level agent evals in DeepEval
- RAGAS Advanced: Agent, Tool-Call and Multi-Turn Metrics Evaluate agent workflows and conversations with RAGAS
- One Eval, Six Tools: Pick Your Eval Stack (Free vs Paid) Choose a stack by cost, hosting, CI fit and team size
Frontier and Advanced: Agent Security and Red Teaming
- Red-Team Agents With PyRIT, garak and promptfoo Run an automated red-team campaign against a tool-using agent
Take before
Lesson list from the ADITEJUS curriculum. Planned lengths are estimates per lesson; Udemy shows the final length.