← All courses
Advanced · 7 lessons
AI evals engineering: testing systems that never repeat themselves
Why “looks right” is not a testing strategy: golden datasets from production traces, LLM-as-judge with bias control, regression CI gates for prompts, pass@k for coding agents, and statistical significance when switching models.
- 01“Looks right” is not a strategyWhy vibe checks do not scale: LLM non-determinism, an eval taxonomy, error analysis over real traces, and a first 20-case eval in one evening.40 min
- 02Golden datasets from production tracesProduction traces as the source of truth: case selection and stratification, a JSONL dataset schema, labeling with inter-annotator agreement, and leak-free versioning.50 min
- 03LLM-as-judge: when it works and when it liesDirect scoring vs pairwise, position/verbosity/self-enhancement bias, calibrating the judge against human labels, and binary rubrics you can trust.55 min
- 04Regression pipelines and CI gatesThe prompt is code: an eval on every PR, promptfoo in GitHub Actions, absolute thresholds and non-regression, a smoke set on PRs and a full nightly run.50 min
- 05Coding-agent evals: pass@k and diff qualityThe pass@k formula, SWE-bench and task completion, diff quality beyond green tests, and a harness built from tasks in your own repository.55 min
- 06A/B tests and statistical significanceOne run is a point, not a distribution: SEM and confidence intervals, paired design, McNemar and bootstrap, sample size, and a canary when switching models.50 min
- 07Capstone: catch the planted regressionAssemble the eval harness end to end — dataset, calibrated judge, statistical gate, CI — and verify it with a mutation test of planted regressions.60 min