Use Cases

How Garanti BBVA Validated UGI Chatbot's LLMs with Co-one

Garanti BBVA aimed to enhance digital customer services and improve satisfaction by transitioning its chatbot system, UGI, from a natural language processing (NLP)-based architecture to one powered by large language models (LLMs).

Challenge: Limitations of Traditional Testing Methods

This technical transformation also meant redefining the accuracy and reliability of customer communication. However, traditional testing approaches proved inadequate for the dynamic and multilayered nature of LLM-based systems. The bank needed a scalable, human-centered validation process that reflected real user behavior.

Solution: Co-one's Three-Layered Testing and Evaluation Approach

Co-one deployed a custom-designed, three-layered testing framework to meet this need:

1. Intent Testing

Each new intent was tested by Co-one’s expert team and its powerful crowdsource community. A wide range of user profiles simulated interactions, and chatbot responses were evaluated for clarity, accuracy, and satisfaction.

2. Model Validation

Once the chatbot was deployed, continuous monitoring ensured accuracy and identified hallucinations or biases. Co-one categorized these issues by version and delivered detailed error cluster reports to the bank for early resolution.

3. Customer-Centered Learning

Real-time analysis of customer queries and chatbot responses helped detect model errors, missing intents, and unmet user needs. This insight extended beyond model improvement to inform Garanti BBVA’s overall service strategy.

Results: Measurable Performance Gains

  • Analysis of up to 300,000 customer interactions and an average of 20,000 test data points per month.
  • LLM responses are evaluated on six criteria: factual accuracy, clarity, guidance relevance, tolerance to typos, ethical compliance, and risk content detection.

Outcome

UGI’s accuracy and user experience improved significantly. Internal support load was reduced, testing processes became faster, and operational efficiency increased. We delivered infrastructure and helped build an AI solution that prioritizes customer experience.

References

More in use cases

MANK and Co-one case study cover on evaluating a banking voice assistant.

Use Cases ·

How MANK evaluated its banking voice assistant

MANK worked with Co-one to automate call-flow testing, assess how its voice assistant handled adversarial conversations, and improve speech recognition and synthesized speech.

Talk to Co-one

Bring a workflow. We’ll start there.

Show us where your agent makes decisions or takes action. We’ll discuss what to simulate, what to validate, and where runtime controls belong.

Book a demo

Choose whether to allow optional analytics and marketing tags.