← All courses
Intermediate · 7 lessons
LLMOps and agent observability: debugging the black box
Traces and spans instead of printf: instrumenting agents with OpenTelemetry, LangSmith, and Langfuse, token and latency dashboards, hunting failure patterns in production traces, alerting on quality drift, and the traces → eval sets → fixes loop.
- 01Traces and spans: why printf debugging diesAn agent is a non-deterministic distributed process: why log lines cannot answer “what went wrong”, the anatomy of trace/span/generation, and the OTel GenAI semantic conventions.40 min
- 02Instrumentation: OpenTelemetry, LangSmith, LangfuseThree instrumentation paths — auto-instrumentation, SDK decorators, and the manual OTel API: the SDK → OTLP → backend pipeline, @observe in Langfuse, @traceable in LangSmith, and picking a stack for your project.55 min
- 03Token, cost, and latency dashboardsTTFT, tokens/s, and p95 instead of the mean: computing cost per request, the panels of a working dashboard, segmentation by model and feature, and aggregate traps that hide regressions.50 min
- 04Failure patterns in production tracesA taxonomy of agent failures — loops, context truncation, swallowed tool errors: each pattern’s signature in traces, search and clustering, and the path from pattern to root cause.50 min
- 05Alerting on quality drift, not just uptimeWhy 200 OK says nothing about answer usefulness: online quality signals, LLM-as-judge on a traffic sample, thresholds and burn rate against alert fatigue, and input-distribution drift detection.50 min
- 06The feedback loop: traces → eval sets → fixesThe LLMOps data flywheel: picking cases from production traces into datasets, a regression loop with a CI gate, linking prompt versions to traces, and a weekly review process.50 min
- 07Capstone: trace detectiveDiagnosing a failing production agent from raw traces: aggregates, reading a sample, root-cause evidence, and an RCA report with eval cases against recurrence.60 min