Machine Learning - Toss

tossAug 3, 20264 min readCurated summary

How DS and MLE Work Together

The post explains how Toss Bank improved collaboration between Data Scientists (DS) and ML Engineers (MLE) by progressively formalizing their responsibilities. What began as manually transferring notebooks evolved into standardized Python files and finally into installable model packages built around explicit interfaces. The result was faster deployments, consistent observability, and clearer ownership, while AI-generated code introduced a new need to standardize coding style as well. ## Problems with Notebook-Based Handoffs Initially, DS built models and inference code in Jupyter notebooks, then handed them to MLE. - MLE had to recreate the serving code from scratch. - Dependencies, configuration files, and source code were often missing or difficult to reproduce. - Preprocessing logic could be interpreted differently by DS and MLE. - “It works in the notebook” did not guarantee that it would work in production. - As the number of models increased, communication and rework grew rapidly. This approach separated people, not code, so the division of responsibility remained unclear. ## Phase 1: Separating Logic into `.py` Files The team next moved the collaboration boundary from people to files. - DS kept notebooks for experimentation and training. - Core inference logic was extracted into `.py` files. - MLE reviewed these files and validated them through CI. - DS’s intended model behavior was preserved more reliably. - Communication costs decreased. However, the files lacked a standardized structure. - Models used inconsistent function names such as `predict()`, `run()`, and `inference()`. - Code still required modifications when moved into the serving environment. - Global configuration changes in one model could affect other models sharing the same process. - Logging, metrics, and error handling could not be applied consistently across models. ## Phase 2: Defining an Interface Contract The team ultimately standardized the boundary through the `commons-ml-model` package. - A base abstraction defines a common model structure. - DS implements three methods: - `pre_process` - `inference` - `post_process` - The base class handles shared concerns such as: - Logging - Metrics - Tracing - Timing and request tracking - DS packages the implementation as a reusable library. - MLE installs the package with `pip install` and deploys it without rewriting the model. This turns the division of work into a code-level contract. DS focuses on model behavior, while MLE owns serving infrastructure and operational concerns. Updating the base abstraction can also add observability features to every model at once. ## Monorepo Collaboration The team manages the abstraction package and individual model packages in a single monorepo using `uv` workspaces. - Changes to the abstraction and affected models can be reviewed in one pull request. - DS and MLE review the same code in the same repository. - CI, release, and versioning policies are centralized. - Switching from Poetry to `uv` improved build speed by three to five times. The tradeoff is that changes to shared packages can affect many models, and the repository becomes heavier as more packages accumulate. ## Standardizing AI-Generated Code AI-assisted development created a separate collaboration problem: consistent structure did not guarantee consistent coding style. The team introduced `pfmls-stylepack` to encode team conventions for AI tools. - Naming conventions are standardized. - Exception-handling patterns are prescribed. - Rules determine when to use enums instead of hard-coded strings. - Hooks apply conventions while code is being generated. - AI-generated code can explain when a particular rule influenced its implementation. The team therefore distinguishes between: - **Structural consistency:** interfaces define what each role implements. - **Style consistency:** shared rules define how code should be written. Both are necessary for smooth reviews. ## Lessons from the Evolution - The hardest decision is choosing the right collaboration boundary: excessive structure limits flexibility, while insufficient structure recreates inconsistency. - Documentation and early DS–MLE pairing reduce the learning curve for the package-based workflow. - Shared libraries are a double-edged sword: one change can cause broad impact, but one fix can also benefit every model. - In the age of AI-generated code, teams must standardize not only responsibilities and interfaces but also implementation style. The practical recommendation is to make collaboration contracts executable: define stable interfaces, package model code for reuse, centralize shared serving behavior, and enforce coding conventions automatically.

Read original

Machine Learning CI/CD Python API Design+4

tossJun 16, 20264 min readCurated summary

User Segmentation for Understanding 28 Million MAU, TUES

Toss developed TUES (Toss User Engagement Segment) to analyze its 28 million monthly active users from a platform-wide perspective. It groups users by their service-use patterns, enabling Toss to understand user motivations, design segment-specific strategies, and explain changes in company-wide metrics. TUES V2 improves on the original by capturing usage depth, multi-service behavior, and engagement with individual service categories. ## Platform-Wide User Segmentation - Service-specific segments such as “users of Service A” are not mutually exclusive or collectively exhaustive because users may use multiple services. - TUES groups users with similar patterns across Toss’s entire service ecosystem. - It helps identify: - Which services users primarily use - How engaged they are with the app - Which user groups may be suitable for particular growth or marketing strategies ## How TUES V1 Worked - Toss calculated each user’s service-use rate per app open. - For example, a user who opened the app 60 times and used Toss Pay during 20 of those sessions had a 33% usage rate. - Users with similar service-usage distributions were grouped using K-Means clustering. - The raw clusters were interpreted and renamed to make them more useful for product and strategy teams. - V1 included: - **Highly engaged users:** Users who regularly use several services - **Service-oriented users:** Users primarily focused on Toss Bank, Toss Securities, inquiry services, benefits, transfers, or other services - **Simple visitors:** Users who open the app but rarely use its services ## How Toss Uses TUES - **Transition strategy:** Teams can plan how to move users from simple visits to service-oriented engagement and eventually to highly engaged usage. - **Product growth:** Product teams can quickly identify which user segments use their service most and combine that insight with transition strategies. - **Behavior analysis:** TUES reveals when users change segments, begin churning, or return after inactivity. - **Top-line metric analysis:** When MAU changes, Toss can identify which user segments moved and which services likely caused the change. - **Targeted marketing:** Marketers use TUES segments for campaigns such as push notifications. The segments are also available in Toss’s internal marketing tool, TUBA. ## Limitations of TUES V1 After roughly two years of use, Toss identified several weaknesses: - V1 measured only the probability of using a service during an app open, not the number of times it was used. - Users who engaged with a service once and users who used it ten times could appear equivalent. - It could not show engagement with secondary service categories. - K-Means is a hard-clustering method, so each user belonged to only one segment despite often using multiple services. - New major services, including Toss Shopping, App in Toss, and Toss Pay, were grouped into a generic “ETC” category. ## TUES V2 Improvements - **Usage-depth measurement:** V2 uses the number of service interactions per app open as a feature, capturing the intensity of engagement. - **Soft clustering:** Instead of assigning each user to one segment, V2 calculates each user’s degree of association with multiple segments and selectively uses those results. - **Three-layer structure:** Users are described through: 1. Overall app engagement 2. Primary service orientation 3. Engagement with each individual service category - The layers are built sequentially, making it clearer why a user belongs to a segment and what action may be appropriate next. ## New Strategic Capabilities in V2 - Teams can identify which service-category engagement should increase first to move users from a semi-engaged segment to a highly engaged one. - Individual service teams, or silos, can quantitatively connect actions that increase service engagement with company-wide segment and performance changes. - Products can more clearly compare the engagement profiles of users who do and do not use their services. - Cross-activation strategies now have a more precise starting point based on service-level engagement. ## Future Development Toss plans to combine TUES with additional analytical frameworks to: - Create faster and more detailed transition strategies using concepts such as service similarity. - Build strategic user maps based on user profiles and service-use patterns. - Quantify segment-specific value by combining TUES with frameworks such as MTVi. TUES demonstrates how platform-level segmentation can make a growing MAU base easier to understand and act upon. By combining overall engagement, primary service use, and service-level depth, TUES helps Toss develop more targeted growth strategies and connect individual product actions to broader company outcomes.

Read original

Machine Learning Data Analytics Soft Clustering User Segmentation+4

tossApr 23, 20265 min readCurated summary

Introducing Toss Place's Data Bot 'PANDA': How every team member works like a data expert

PANDA, short for Place Analytics & Data, is Toss Place’s AI data-analysis assistant, designed to let employees retrieve and interpret approved data without waiting for analysts. It was created after the team found that 70% of data requests involved simple metric lookups rather than complex analysis. The project’s main conclusion is that reliable AI analytics depends less on prompting alone and more on standardized data, business definitions, controlled table selection, and iterative validation. ## Why Toss Place Built PANDA - Employees previously relied on analysts to search dashboards, write SQL, or manually investigate data requests. - PANDA provides self-service access within each employee’s security permissions. - It reduces routine extraction work for analysts, allowing them to focus on deeper analysis. - The goal is to establish a stronger culture of “data democracy,” where employees can access and use data immediately. ## Challenges with a Simple AI Chatbot Early experiments showed that asking an AI model to search all company data produced unreliable and expensive results: - Referencing thousands of tables and internal documents consumed excessive tokens. - The model sometimes selected different tables for identical questions, producing inconsistent answers. - It often misunderstood business definitions. For example, “active stores” could mean stores with completed installations or stores that had processed payments. - Inefficient SQL caused unnecessary Snowflake data scans and higher warehouse costs. ## Standardized Data Marts as a Single Source of Truth Toss Place collaborated across its Data Analysis and Data Platform teams to establish reliable standard data marts. - Core concepts, such as store information, were consolidated into standardized tables. - Naming conventions made table and column purposes easier for both people and AI to understand: - Tables follow `{mart_type}_{domain}_{subject}`, such as `fact_device_error_log`. - Columns follow `{prefix}_{entity}_{attribute}_{suffix}`, such as `is_merchant_active`. - Table and column descriptions were documented comprehensively. - The standardization effort reduced ambiguity by ensuring the same business concepts were represented consistently. ## Connecting Business Language to Data Data structures alone could not answer questions about terms such as “installed store” or “store category.” - Domain-specific terms and metric definitions were documented. - These business definitions were linked to the relevant standard data marts. - Data analysts helped reconcile differing interpretations and establish shared organizational definitions. - This gave PANDA the context needed to apply the correct business logic. ## Scoring and Ranking for Reliable Table Selection PANDA limits its search to well-managed tables and uses dbt tags to import selected metadata into a Manifest file. - Tables are ranked using: - **Similarity score:** Based on relationships between the question and table, including table-name matches and description relevance. - **Hierarchy weight:** Reflecting the reliability of the data layer. - The final score is calculated as: `similarity score × hierarchy weight` - Weights are assigned as follows: - Company-wide SSOT metrics: ×4 - Validated standard marts: ×3 - Domain analysis marts: ×2 - Raw bronze data and logs: ×1 - This improves accuracy, consistency, and trustworthiness while reducing unnecessary warehouse exploration. ## Agentic Loop for Querying and Validation Rather than expecting a correct answer in one attempt, PANDA uses an agentic loop. - It selects appropriate tools based on the question. - It explores tables, generates and executes queries, and reviews the results. - If the result appears inaccurate, it can inspect the schema again, modify the query, and retry. - If necessary, it asks the user for clarification. - This approach allows PANDA to handle exceptions dynamically instead of relying only on predefined rules. ## Answers Designed for Practical Use PANDA structures responses so users can understand and apply the results: - **Result:** The requested data or metric. - **Query criteria:** The period, filters, and aggregation method used. - **Insight:** An interpretation that can support practical decisions. This makes PANDA more than a number-retrieval chatbot; it also exposes part of the reasoning process normally provided by a data analyst. ## Adoption and User Response PANDA quickly became part of everyday work at Toss Place. - One-third of employees used it on its first day. - Half of the organization had tried it within a week. - More than 4,000 messages were exchanged during that period. - Current adoption is approximately 70%. - Employees reported feeling more comfortable asking small questions and using data while away from their desks. - Users particularly valued receiving insights alongside raw figures. - Unexpectedly, developers and even data professionals used PANDA actively, suggesting that its answers achieved a meaningful level of trust. ## Future Development PANDA was developed and launched in just one month, but the team plans further improvements. - Increase data coverage to more than 90%. - Raise answer accuracy above 97%. - Use real user questions, follow-up behavior, and abandonment patterns to identify unmet needs. - Expand beyond basic data retrieval to reduce more of the data team’s workload. PANDA’s central lesson is that effective enterprise AI does not require the most complicated technology. It requires solving a real business pain point with trustworthy data foundations, clear definitions, and a workflow that users can rely on.

Read original

Machine Learning AI Database Design Sql+4

tossDec 22, 2025Original article

Toss's AI Technology Recognized (opens in new tab)

Toss ML Engineer Jin-woo Lee presents FedLPA, a novel Federated Learning algorithm accepted at NeurIPS 2025 that addresses the critical challenges of data sovereignty and non-uniform data distributions. By allowing AI models to learn from localized data without transferring sensitive information across borders, this research provides a technical foundation for expanding services like Toss Face Pay into international markets with strict privacy regulations. ### The Challenge of Data Sovereignty in Global AI * Traditional AI development requires centralizing data on a single server, which is often impossible due to international privacy laws and data sovereignty regulations. * Federated Learning offers a solution by sending the model to the user’s device (client) rather than moving the data, ensuring raw biometric information never leaves the local environment. * Standard Federated Learning fails in real-world scenarios where data is non-IID (Independent and Identically Distributed), meaning user patterns in different countries or regions vary significantly. ### Overcoming Limitations in Category Discovery * Existing models assume all users share similar data distributions and that all data classes are known beforehand, which leads to performance degradation when encountering new demographics. * FedLPA incorporates Generalized Category Discovery (GCD) to identify both known classes and entirely "novel classes" (e.g., new fraud patterns or ethnic features) that were not present in the initial training set. * This approach prevents the model from becoming obsolete as it encounters new environments, allowing it to adapt to local characteristics autonomously. ### The FedLPA Three-Step Learning Pipeline * **Confidence-guided Local Structure Discovery (CLSD):** The system builds a similarity graph by comparing feature vectors of local data. It refines these connections using "high-confidence" samples—data points the model is certain about—to strengthen the quality of the relational map. * **InfoMap Clustering:** Instead of requiring a human to pre-define the number of categories, the algorithm uses the InfoMap community detection method. This allows the client to automatically estimate the number of unique categories within its own local data through random walks on the similarity graph. * **Local Prior Alignment (LPA):** The model uses self-distillation to ensure consistent predictions across different views of the same data. Most importantly, an LPA regularizer forces the model’s prediction distribution to align with the "Empirical Prior" discovered in the clustering phase, preventing the model from becoming biased toward over-represented classes. ### Business Implications and Strategic Value * **Regulatory Compliance:** FedLPA removes technical barriers to entry for markets like the EU or Southeast Asia by maintaining high model performance while strictly adhering to local data residency requirements. * **Hyper-personalization:** Financial services such as Fraud Detection Systems (FDS) and Credit Scoring Systems (CSS) can be trained on local patterns, allowing for more accurate detection of region-specific scams or credit behaviors. * **Operational Efficiency:** By enabling models to self-detect and learn from new patterns without manual labeling or central intervention, the system significantly reduces the cost and time required for global maintenance. Implementing localized Federated Learning architectures like FedLPA is a recommended strategy for tech organizations seeking to scale AI services internationally while navigating the complex landscape of global privacy regulations and diverse data distributions.

Machine Learning AI Computer Vision Data Privacy+5

tossDec 1, 2025Original article

Toss Next ML Challenge (opens in new tab)

Toss recently hosted the "Toss Next ML Challenge," a large-scale competition focused on predicting advertisement Click-Through Rates (CTR) using real-world, anonymized data from the Toss app. By tasking over 2,600 participants with developing high-performance models under real-time serving constraints, the event successfully identified innovative technical approaches to feature engineering and model ensembling. ### Designing a Real-World CTR Prediction Task * The competition required participants to predict the probability of a user clicking a display ad based on a dataset of 10.7 million training samples. * Data included anonymized features such as age, gender, ad inventory IDs, and historical user behavior. * A primary technical requirement was "real-time navigability," meaning models had to be optimized for fast inference to function within a live service environment. ### Overcoming Anonymization with Sequence Engineering * To maintain data privacy while allowing external access, Toss provided anonymized features in a single flattened table, which limited the ability of participants to perform traditional data joins. * A complex, raw "Sequence" feature was intentionally left unprocessed to serve as a differentiator for high-performing teams. * Top-tier participants demonstrated extreme persistence by deriving up to 37 unique variables from this single sequence, including transition probabilities, unique token counts, and sequence lengths. ### Winning Strategies and Technical Trends * All of the top 30 teams utilized Boosting Tree-based models (such as XGBoost or LightGBM), while Deep Learning was used only by a subset of participants. * One standout solution utilized a massive ensemble of 260 different models, providing a fresh perspective on the limits of ensemble learning for predictive accuracy. * Performance was largely driven by the ability to extract meaningful signals from anonymized data through rigorous cross-validation and creative feature interactions. The results of the Toss Next ML Challenge suggest that even in the absence of domain-specific context due to anonymization, meticulous feature engineering and robust tree-based architectures remains the gold standard for tabular data. For ML engineers, the competition underscores that the key to production-ready models lies in balancing complex ensembling with the strict latency requirements of real-time serving.

Machine Learning AI Feature Engineering Xgboost+4