Track 9: LLM Engineering and Production, Eval Tools (One Lesson Each)

LangSmith Evaluations: Datasets and Experiments

Run and compare experiments in LangSmith (paid plans; free tier limits: verify price).

Status: Coming soon IntermediateVideo lessonAbout 13 min Free tools
Get notified on YouTube Udemy certificate: coming soon

Course

Agent Testing & Evaluation v2 (update of the live DeepEval course)

Udemy: coming soon

Tools

Watch first

In this module

  1. promptfoo: Your First Eval Suite in YAML
  2. Inspect (UK AISI): Tasks, Solvers and Scorers
  3. OpenAI Evals: Open-Source Framework and Paid Evals API
  4. LangSmith Evaluations: Datasets and Experiments
  5. Langfuse Evals: Datasets, Experiments, LLM Judges
  6. Braintrust: Experiments, Scorers and autoevals
  7. Arize Phoenix Evals: Trace First, Then Evaluate
  8. TruLens: Feedback Functions and the RAG Triad
  9. DeepEval Advanced: Custom Metrics and Agent Tracing
  10. RAGAS Advanced: Agent, Tool-Call and Multi-Turn Metrics
  11. One Eval, Six Tools: Pick Your Eval Stack (Free vs Paid)

The course this lesson belongs to