Overview
This project involves a study conducted by Co-one for Türkiye İş Bank's chatbot, Maxi, focusing on text annotation and intent classification. Co-one collaborated with Türkiye İş Bank to enhance the performance and accuracy of Maxi, the bank’s AI-powered chatbot, in understanding and responding to customer queries. Through comprehensive text annotation and intent classification, Co-one aimed to optimize Maxi’s natural language processing capabilities, ensuring a seamless customer experience and efficient query resolution.
The Problem
Before the project, Türkiye İş Bank’s chatbot, Maxi, faced challenges in accurately understanding and classifying customer intents. The existing system lacked the precision required to correctly interpret and respond to a wide range of customer queries. This led to inefficiencies in providing timely and accurate information. To overcome these limitations, Türkiye İş Bank sought a solution that would improve Maxi’s ability to comprehend and categorize user intents effectively.
Customer’s Data Labeling History
As a solution to the data labeling need for the company’s Chatbot project, which is Maxi, they tried to annotate data in-house. However, since a data labeling team has not been established (which can be costly for the company) and data labeling is not the main job of the employees, this process has taken a long time.
Customer’s Meeting & Agreement Process with Co-one
Co-one caught the attention of the customer in an acceleration program, Workup, which it participated in. Then İş Bank decided to do a demo. The demo was held in December 2021. After the demo was successful, in April 2022, it was decided to deliver data in the text classification area with a monthly volume of 200,000 with the agreement on the annual SAAS model. Data could be delivered on a daily/weekly/monthly basis as requested by our customers.
Understanding Customer Needs and Creating a Guideline
Upon receiving the raw data from our customers, which is made possible through the Co-one Data Annotation Dashboard or any other preferred delivery method of their choice, our team of proficient Project Engineers meticulously assess and comprehend the client’s requisites for labeling the data. Following a detailed analysis of the customer’s needs, a project guideline is developed. This approach allows our customers to maintain full control over the data labeling stages of Co-one. With the guideline, we offer a preliminary solution to mitigate the risk of any unforeseen setbacks during the project.
The Data Labeling Process and the Power of The Crowdsource
Our customer forwards us a monthly list of inquiries that the chatbot system is unable to address or has misconstrued, in an Excel format. Upon receipt of this data, Co-one’s proficient Crowdsource Team is responsible for annotating data based on the data types, data industry, expertise, and Crowdsource member accuracy scores.
To ensure an effective solution to edge cases, we meticulously scrutinize each piece of data. Before working with Co-one, the chatbot, Maxi, encountered perplexing scenarios in the past.
“I need money, what should I do” as “my money is stuck in the ATM.”
Subsequently, our Crowdsource Team has annotated that are specifically tailored to the format of the data and aligned with the annotation guidelines. Each data point undergoes manual labeling by three annotators to enhance accuracy.
Cross Validation
AI Cross-Validates the data to see matches and detect mismatched data. Flagged data is sent to Co-one Quality Team for further reviewing & correcting by expert annotators.
As a result, we get unbiased text labeling by distributing the data to our multiple users. In other words, we can reach a result in different ways by using the power of crowdsourcing. This ensures that the resulting chatbot product is unbiased. And the quality of data annotated with cross-validation is under our control.
Increase/Decrease in the Customer’s Data
By our annual agreement with our customer, a monthly data volume of 200,000 was stipulated. However, due to the company’s ever-evolving needs, the data specifications have undergone periodic alterations, with some months featuring the provision of chatbot or IVR (interactive voice response) data.
Despite the fluctuating data volumes, we have seamlessly managed to deliver a proficient service. As a crowdsourced platform, we have the advantage of swift training and assigning additional labelers, enabling us to cater to the increasing labeling requirements and provide weekly delivery.
Data Delivery, Satisfaction, and Reporting
By consistently reviewing and responding to customer feedback, we have developed a dynamic mechanism that facilitates continuous improvement of our data labeling process. This approach has led to enhanced data quality with each passing month.
Furthermore, Co-one provides data reports that offer valuable insights into your data. These reports include comprehensive information on data characterization, RGB analysis, sector-specific insights, data distribution, edge cases, and unexpected results/anomalies within the dataset. By availing yourself of these reports, you can gain a deeper understanding of your data, while simultaneously increasing the value of using Co-one as your trusted data partner.
Data Security
Co-one offers data security through all stages of data annotation. The data is secured by precautions, authorized access methods, and preventive methods.
Each data that is transferred or processed through Co-one subsystems is only accessible by authenticated clients with expiring access tokens for each data. The data communication protocols are ensured to be safe by complying with HTTPS and Google authentication systems.
The data points that are requested by mobile clients can not share device screens and can not take screenshots to further block data leakage.
For preventive methods, the annotator clients who are authenticated to view a single data point in a given time are obliged to comply with data protection, and failing to comply with data protection responsibilities is subject to sanctions.
Are you looking to transform your AI-powered chatbot’s performance and enhance customer interactions? Co-one is here to improve your chatbot. Whether you’re facing challenges in intent classification or text annotation, we have the expertise to streamline your chatbot’s performance. Don’t let inefficiencies hold back your customer service. Reach out to us today, and let’s unlock the full potential of your AI chatbot.
Customer Review
Co-one’s intent generation service for our chatbot, Maxi, has accelerated our data-feeding process by analyzing Maxi chatbot dialogs every week, saving us valuable time and resources. Maxi’s dialog accuracy rate has reached up to %98 with valuable contributions of Co-one. We are delighted with Co-one’s commitment to a high data accuracy rate and look forward to continued collaboration.
Gamze Ortakaya - Innovation and Digital Strategy Sub Manager
Commentaires