Build Smarter Voice Agents

The panel, featuring Adam Schuld (CTO, Super), Zhongren Shao (Sr. Software Engineer, Retell), and Ryan Seams (VP, Customer Solutions, AssemblyAI), explores the intricacies of building voice AI agents. They begin by discussing the prevalence of cascading architectures, driven by their ease of configuration and the ability to optimize individual components like ASR, LLM, and TTS layers for latency and accuracy. For speech-to-speech applications, they note the more experimental nature and less established tooling.

A key question revolves around how to evaluate and prioritize factors like latency versus accuracy, with the consensus leaning towards prioritizing low latency (1-1.5 seconds) for a good user experience, even if it means some trade-offs in accuracy. The discussion touches on the challenges of real-world deployments, where factors like network issues and background noise can degrade performance.

They also explore the importance of context and prompt engineering, explaining how their internal "scratchpad" system allows them to store and retrieve conversational context, enabling more natural and less repetitive AI responses. The panelists discuss the potential future of voice AI, with the idea of highly personalized agents that leverage deep understanding of user history and preferences to provide tailored assistance.

Finally, they touch upon the cost implications of large models and the ongoing search for more efficient architectures, acknowledging the challenges but also the rapid advancements in the field. The conversation highlights the blend of technical considerations, user experience design, and practical engineering required to build effective voice AI agents.