What did Grupo Esfera’s coaching with this healthcare team consist of?

Grupo Esfera designed a two-tiered intervention: strategic consulting for the Quality Certification team, and direct mentoring for two pilot teams, featuring training sessions, group implementation workshops, and bi-weekly follow-ups.

What types of tests were implemented?

Seven distinct test types were incorporated: unit and integration tests, staging tests for external services (featuring an inheritance hierarchy to standardize shifts across multiple providers), and the reuse of monitoring as smoke tests during deployments.

How long did the engagement last and what concrete results did it deliver?

The joint engagement spanned 6 months, during which 13 training sessions were delivered and 14 concrete improvements were driven, with automated workflows already integrated into the CI/CD pipeline.

<- Back

>>>

Conversational solutions

Evolution of conversational bot development and testing with Generative AI

From manual testing via WhatsApp to automated testing and LLM-as-a-judge to validate generative responses.

Grupo Esfera transformed end-to-end how the client's team builds and tests its conversational bots, moving from 100% manual validation to a continuous integration pipeline with a proprietary methodology for testing Generative AI responses.

I WANT TO SCHEDULE A MEETING

SEE MORE CASES

Summary

From deterministic bots to conversational assistants with Generative AI

Grupo Esfera transformed, end-to-end, how one of the client's teams builds and tests conversational WhatsApp bots. Starting from a setup with zero automated testing—where everything was manually validated by interacting with WhatsApp—they progressively built a proprietary testing framework based on Gherkin, continuous integration, and continuous delivery. When the bots began relying increasingly on Generative AI (accounting for up to 90% of the logic in some cases), Esfera designed a testing methodology to automatically validate non-deterministic responses: the LLM-as-a-judge technique, coupled with an architecture where the same business rules powering the tests directly generate the bot's prompt.

Discovery Phase

INDUSTRY

Conversational solutions

SERVICE

Technical Coaching (team evolution, practices, and processes) + Generative AI Solution Development

TEAM

3 professionals (1 architect + 2 full-stack developers)

METHODOLOGY

Scrum + eXtreme Programming (XP), Trunk-Based Development, Feature Toggles, Continuous Integration & Continuous Delivery (CI/CD), Slicing, Visual Story Maps, Evolutionary Discovery, Emergent Architecture

QUALITY

Test-First (Gherkin/Cucumber), unit testing, exploratory testing, Allure Reports, LLM-as-a-Judge, mocking (Mockoon)

STACK

JavaScript, .NET Core, C#, GitHub Actions, Vector Store, OpenAI API (Responses API / Chat Completions API)

Challenge

The client's team builds WhatsApp bots (connected to Meta) for multiple end clients, each with its own business logic, on an internal platform with over 50 bots coexisting within a single repository.

1

Absence of automated testing

There were no unit or integration tests for the bot's behavior. All validation was done manually by deploying the bot to a test environment from the developer's local machine and manually interacting with it via WhatsApp.

2

Abandoned testing framework

The team providing the bot's internal library had previously built a functional testing framework based on Gherkin and Cucumber, but no team was using it: writing the glue code to connect each test step to an actual bot action required such a deep knowledge of the platform that it was impractical.

3

Disorganized delivery lacking traceability

Branch management exhibited a long-lived branch anti-pattern, and the CI pipeline consisted of GitHub Actions that failed to guarantee a healthy build. In many cases, there was no way to know with certainty which version of a bot was running in the release candidate environment.

4

Rigid specification disconnected from actual usage

A bot's flow was agreed upon with the client as a closed flowchart, delivered, and never touched again. Without continuous refinement, it was common for a client to decommission a bot after 6 months because no one was using it.

5

Generative AI multiplied testing complexity

With the integration of generative modules—in some bots, up to 90% of the behavior relies on a language model—comparing exact responses was no longer sufficient. Each execution could yield a different output, valid in different ways, making it impossible to validate behavior using traditional assertions.

Solution

Evolutionary testing and Generative AI

Applying the Esfera Way, Grupo Esfera formed a team that operated on the principle that quality is the driver of speed, not a barrier: delivering fast did not mean writing code quickly, but rather reaching the production environment fast and with confidence. The transformation was progressive and relied on concrete tactics across three fronts: testing, delivery, and Generative AI.

Learn more about the Esfera Way

Testing — Modules and unit tests

Bot logic, isolated and testable

We separated the bot code from the client's proprietary library into independent modules, covered by unit tests.

Testing — Specification by example

Live Gherkin, independent of WhatsApp

Integration with OpenAI, Vector Stores (RAG), and multi-agent architectures, with cost control per conversation.

Delivery — Continuous integration

From long-lived branches to a per-bot pipeline

We migrated to Trunk-Based Development and built from scratch a pipeline that compiles and tests every bot affected by a change.

Delivery — Continuous deployment

From local machine to production with a single click

Every change is deployed only to a release candidate, and moves to production with a 15-minute traceable approval.

Generative AI — Prompt Engineering

Fewer rules, greater control

We discovered that over-engineering the prompt made it less predictable: we started with the absolute minimum set of rules and added constraints only as needed.

Generative AI — Nondeterministic Testing

An AI judge for AI responses

We translated business expectations into generative assertions evaluated by a second model (LLM-as-a-judge), preserving full evidence across every run.

Generative AI — Rules and prompting

The rules that are tested are the rules that are enforced

Each business rule directly generates the bot's system prompt, eliminating the drift between what is tested and what is instructed to the model.

Resultas