A major telecommunication company recently faced challenges with their interactive voice response (IVR) system. They encountered difficulties with establishing an efficient method to identify software defects early and often in their Software Delivery Lifecycle (SDLC). This was largely due to a lack of effective dependency management in two priority areas - test data and test environments. As a consulting services partner of 20 years, Kenway wanted to equip the team responsible for IVR Quality Assurance (QA) with the tools and processes needed to support a healthier QA experience. This, in turn, would allow for easier and more accurate identification of any defects earlier in the development lifecycle.
The QA transformation involved facilitating the early and recurrent identification of legitimate software defects using automation while also proactively managing dependencies like test data and environment stability. Implementing this strategy correctly would enable the team to test more code, in less time, with fewer hours devoted to manual test calls and data manipulation, thereby reducing the overall cost of the certification process, and increasing the validation volume that can be successfully performed.
Kenway’s two decades of experience working with this client gave the team a unique perspective into their processes, strengths, and weaknesses. Through Kenway's long-term partnership with Cyara, the industry leader in AI-led CX assurance, Kenway identified Cyara Botium as the perfect solution to support its client's QA transformation.
During this project, their primary challenge was improving the efficiency of automation within the QA framework, which had been hindered by inadequate management of core QA dependencies like valid test data and environment stability. Previous attempts to scale automation for the team were unsuccessful due to unstable test data and environments that did not meet compliance standards. Kenway’s challenge was to effectively integrate and consistently apply Botium’s AI-driven testing capabilities across the SDLC while finding a way to stabilize the testing environments and data.
Amidst overcoming historical difficulties, the client is also currently engaged in a significant modernization initiative across its Contact Center ecosystem, moving from a legacy IVR and Chat experience to Google’s Contact Center AI (CCAI) platform. At the time of the Botium implementation, the team was dramatically expanding the number of self-service opportunities it offered to customers, providing faster issue resolution, and reducing the number of calls to their contact centers. This growth in self-service capabilities introduced more opportunities for defects and increased demand on the QA team to test a higher volume of intents.
Many of the existing testing procedures to accommodate Google CCAI were manual and the client started its voice migration with more than twenty QA resources supporting the program. Those resources performed roughly 80% of tests manually by picking up a phone and dialing the new experience to validate the requirements. Given the complexity of the application, relying on manual testing made it difficult for the testers to cover all the important scenarios, and very rarely did they have time to undertake the low priority or regression tests needed to fully certify the release. This resulted in defects making it to production, exhaustive backlogs, and decreased usability in the application. Extended delivery timelines caused by the need to re-validate functionality that should never have made it to production also led to budget impacts due to higher overall costs for the migration project.
To get the full benefit of the AI transformation with Google CCAI, it became necessary for the telecommunications company to deliver AI-driven experiences quickly, with quality, and at scale. This meant certifying many state-of-the-art, AI-driven capabilities – all of which were delivering rather complex, self-service functions across multiple environments – simultaneously, seamlessly, and consistently. The client identified a critical need to expedite the return on investment (ROI) and the business benefits they would realize from their new platform. Leadership also needed a way to test at a higher volume and limit defects in production without increasing the headcount needed to manage QA.
To overcome those challenges, Kenway worked with the client’s IVR team to implement Cyara Botium, a state-of-the-art solution for automated chatbot testing. Botium plays a pivotal role in ensuring that customer interactions across various channels like chat, web, and voice are of the highest quality, leveraging AI-driven capabilities to help facilitate automated testing against AI-based solutions.
Cyara Botium was hand-picked to address the limitations in scaling automation as well as having an effective AI certification model. By implementing Botium, the IVR team has achieved the following benefits for their AI transformation:
Botium has Session Initiation Protocol (SIP) testing functionality integrated into its platform, which is essential for an IVR system. SIP-based testing invokes an IVR via a toll-free number terminating through the telephony cloud onto the platform that manages the IVR applications. The Kenway and Cyara teams coordinated to design a unique test plan that connected with the client’s systems, retained the Dialogflow agent interaction, and successfully communicated with applications on the IVR framework. This was achieved directly through API interactions, without using SIP.
Kenway leveraged Botium's default API connector to directly interface with Google's Dialogflow Agent, allowing access to key library assets such as entities and intents. By customizing Botium's API connector configuration and unique case-by-case payload configurations, Botium’s interactions mirror SIP test call interactions across the IVR framework. This innovative approach enabled a seamless and effective validation process.
However, automation and AI can only take testing so far. This major telecommunications provider has a complex business and even more complex customer profiles, making test data management difficult. To mitigate any issues with test data and false failures in Botium, Kenway created a personified test data pool that mimicked real-life customer profiles and could be reused across different testing scenarios. This test data pool is rigorously governed to make sure that data is always up-to-date and accurate.
The collaboration between Kenway, Cyara, the client, and Google resulted in a uniquely tailored Botium solution. This approach made it possible for testers to thoroughly examine complete conversational scenarios, including interactions with connected IVR systems and their dependencies, without having to leave the Botium graphical user interface. Kenway’s implementation of Botium showcases a proactive and innovative approach to enhancing customer experience through technology. This work was necessitated by the need for delivery teams to not only deploy new, cutting-edge, AI technologies faster, but also to ensure their rapid, high-quality, and scalable delivery.
This telecommunications provider is only beginning their AI-driven QA journey. Google’s CCAI platform has opened the door to new opportunities and the rapid growth of Generative AI capabilities has added nearly infinite new possibilities for customers and permutations of test cases for the QA team.
The implementation originally began with a small subset of intents and a few test data personas, all of which were focused on English language calls. The client’s leadership team has already asked if Botium can expand into Spanish. And if Botium can validate the flows that are backed by Generative AI? And, thankfully, Cyara’s answer was “yes, absolutely”.
