Use Cases
How Garanti BBVA Validated UGI Chatbot's LLMs with Co-one
Garanti BBVA aimed to enhance digital customer services and improve satisfaction by transitioning its chatbot system, UGI, from a natural language processing (NLP)-based architecture to one powered by large language models (LLMs).
Challenge: Limitations of Traditional Testing Methods
This technical transformation also meant redefining the accuracy and reliability of customer communication. However, traditional testing approaches proved inadequate for the dynamic and multilayered nature of LLM-based systems. The bank needed a scalable, human-centered validation process that reflected real user behavior.
Solution: Co-one's Three-Layered Testing and Evaluation Approach
Co-one deployed a custom-designed, three-layered testing framework to meet this need:
1. Intent Testing
Each new intent was tested by Co-one’s expert team and its powerful crowdsource community. A wide range of user profiles simulated interactions, and chatbot responses were evaluated for clarity, accuracy, and satisfaction.
2. Model Validation
Once the chatbot was deployed, continuous monitoring ensured accuracy and identified hallucinations or biases. Co-one categorized these issues by version and delivered detailed error cluster reports to the bank for early resolution.
3. Customer-Centered Learning
Real-time analysis of customer queries and chatbot responses helped detect model errors, missing intents, and unmet user needs. This insight extended beyond model improvement to inform Garanti BBVA’s overall service strategy.
Results: Measurable Performance Gains
- Analysis of up to 300,000 customer interactions and an average of 20,000 test data points per month.
- LLM responses are evaluated on six criteria: factual accuracy, clarity, guidance relevance, tolerance to typos, ethical compliance, and risk content detection.
Outcome
UGI’s accuracy and user experience improved significantly. Internal support load was reduced, testing processes became faster, and operational efficiency increased. We delivered infrastructure and helped build an AI solution that prioritizes customer experience.

