The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

This lecture provides an actionable, experience-based playbook for shifting from AI experimentation to reliable, enterprise-scale production. Sandipan Bhaumik highlights that initial AI projects often follow a pattern of over-promising and under-delivering due to a lack of proper observability, evaluation, and governance. He argues that 'the AI is the easy part' and that success relies on establishing five core pillars: Evaluation, Observability, Data Foundation, Orchestration, and Governance.

The talk meticulously breaks down each pillar, providing concrete examples. He emphasizes that evaluation must be defined quantitatively before writing code and describes the three necessary layers of evaluation: deterministic, semantic (using LLM-as-a-judge), and behavioral (monitoring tool calls and agent loops). The data foundation, covering both prompt/RAG context and tracing data, is presented as the critical infrastructure for enabling these evaluations.

Finally, the lecture examines the complexities of multi-agent orchestration and the necessity of incorporating human-in-the-loop governance for high-stakes decisions. The speaker draws from a real-world case study at a retail bank, showing how following this phased, test-driven approach allowed them to reach production success in eight weeks after an initial failed attempt.