Published Paper Details:

This paper discusses the challenges associated with measuring customer-perceived reliability for conversational AI assistants and proposes a framework to measure the reliability of such assistants. We first provide an overview of conversational AI assistants and propose a dimensional approach to defining reliability that covers the broad range of use-cases served by these assistants. We analyze existing mechanisms used to measure reliability of assistants and discuss challenges presented by the new wave of assistants powered by Large Language Models (LLMs). The proposed solution framework leverages end-to-end telemetry, feedback collection through real-time sentiment analysis, and offline analysis of interaction logs by generative models to provide a comprehensive view of reliability. This research's framework includes further analysis of data to identify the source of defects and feed information back to the assistant, enabling the assistant to adjust its responses as needed. By building on recent progress in LLMs, this framework tackles issues such as context-awareness, external tool integration and scalability. Finally, this work offers tangible ideas for implementation while considering real world limitations around cost, privacy, and compliance with regulations.