Is your AI bot responding correctly? From manual Generative AI testing on WhatsApp to automated CI/CD

Testing software has always been a discipline and engineering challenge. However, when building conversational assistants on WhatsApp powered by Generative AI, where up to 90% of the logic relies on a Large Language Model (LLM), traditional rigid assertion techniques no longer work. How do you know if a non-deterministic response is correct before deploying it to production?

In this success story, we show you how we transformed the end-to-end workflow of a leading conversational solutions company: moving from a 100% manual validation setup on WhatsApp to a Continuous Integration and Continuous Delivery (CI/CD) pipeline with an automated testing architecture using the LLM-as-a-judge technique and the unification of business rules with the system prompt.

The Challenge: 50+ Bots, Manual Testing, and Generative AI Uncertainty

Our client's team develops and maintains WhatsApp bots on an internal platform hosting over 50 assistants in a single repository. As solution complexity grew, several technical and process bottlenecks emerged:

• Lack of automation: Unit tests and integration tests were non-existent. Every validation required deploying locally and interacting manually from a cell phone on WhatsApp.

• Abandoned testing frameworks: A previous attempt based on Gherkin and Cucumber existed, but the glue code required to connect the steps with the internal platform was so complex that the team abandoned it.

• Disorganized delivery: Branch management featured long-lived branches and GitHub Actions pipelines that guaranteed neither healthy builds nor traceability regarding which version was running in each environment.

• The non-deterministic challenge of AI: When incorporating LLMs, exact text matching became impossible. Each execution could return a different yet equally valid response, making traditional assertions obsolete.

  
  

Under this context, the client needed to speed up deliveries, but doing so without a clear quality strategy meant multiplying risk in production.

The Solution: The Esfera Way and Methodology for Generative AI Testing

True to the Esfera Way, we worked under a fundamental premise: quality is not a speed bump, it is its main engine. Delivering fast does not mean typing code at full speed, but reaching production with confidence, traceability, and no surprises.

Through a Technical Coaching framework and hands-on co-development (1 Architect + 2 Full Stack Developers), we implemented a progressive transformation based on three pillars:

1. Evolutionary Testing and Specification by Example

• Modular isolation: We separated the bot's logic from the client's proprietary library into independent modules to cover them with unit tests.

• DSL over Gherkin without WhatsApp: We simplified the existing test framework by creating a domain-specific language (DSL) that translates bot responses (text, buttons, forms) into simple comparisons. This allowed simulating entire conversations without depending on the WhatsApp interface or writing complex glue code.

• Visibility with Allure Reports: We integrated visual reporting that presents conversations in a user-friendly format identical to actual chat interfaces.

2. Continuous Integration and Continuous Delivery (CI/CD)

• Trunk-Based Development: We eliminated endless branches to adopt a true continuous integration flow.

• Dedicated Pipeline per Bot: We designed pipelines in GitHub Actions that compile and test only the bot affected by a change.

• Traceable deployments: Every push is automatically deployed to Release Candidate, requiring only a 15-minute traceable approval to move into production.

3. Generative AI Testing: LLM-as-a-Judge Methodology

• Unification of Rules and System Prompt: We designed an architecture where the same business rules expressed in tests directly generate the bot's system prompt, ensuring a single source of truth.

• Evaluation with a Second LLM: To validate non-deterministic responses, we applied the LLM-as-a-judge pattern. We translated business expectations into assertions evaluated by a supervisor model within the same CI pipeline, storing evidence of prompts, responses, and verdicts.

• Evolutionary Architecture: We evolved the solution toward multi-agent systems with persistent memory and Retrieval-Augmented Generation (RAG) architecture over proprietary knowledge bases.

Results: From Manual Uncertainty to Automated Confidence

The technical and methodological transformation drastically raised the team's standards:

Dimension	Previous State	Result with Grupo Esfera
Test Coverage	Zero automation; 100% manual testing on phones.	Full suite (unit, exploratory, and integration) executed on every change.
Delivery Flow	Manual deployments from local machines without traceability.	CI/CD on Trunk-Based Development with deployments to RC and production release in 15 mins.
AI Validation	Empirical, manual, and unpredictable validation.	LLM-as-a-judge evaluation integrated into the pipeline with complete evidence.
Business Rules	Documentation disconnected from the final prompt.	Business rules and system prompt generated from a single source of truth.
Bot Capabilities	Rigid, deterministic flows.	Multi-agent architecture, RAG, and persistent memory.

Take Your Team's Engineering to the Next Level

To make this transformation possible, we combined our Technical Coaching service (evolution of practices and teams) with our Generative AI Solution Development service.

Want to transform how your team builds and tests software products powered by Generative AI?

👉 Get in touch with our team to discuss how we can partner with you.