How DS and MLE Work Together
The post explains how Toss Bank improved collaboration between Data Scientists (DS) and ML Engineers (MLE) by progressively formalizing their responsibilities. What began as manually transferring notebooks evolved into standardized Python files and finally into installable model packages built around explicit interfaces. The result was faster deployments, consistent observability, and clearer ownership, while AI-generated code introduced a new need to standardize coding style as well. ## Problems with Notebook-Based Handoffs Initially, DS built models and inference code in Jupyter notebooks, then handed them to MLE. - MLE had to recreate the serving code from scratch. - Dependencies, configuration files, and source code were often missing or difficult to reproduce. - Preprocessing logic could be interpreted differently by DS and MLE. - “It works in the notebook” did not guarantee that it would work in production. - As the number of models increased, communication and rework grew rapidly. This approach separated people, not code, so the division of responsibility remained unclear. ## Phase 1: Separating Logic into `.py` Files The team next moved the collaboration boundary from people to files. - DS kept notebooks for experimentation and training. - Core inference logic was extracted into `.py` files. - MLE reviewed these files and validated them through CI. - DS’s intended model behavior was preserved more reliably. - Communication costs decreased. However, the files lacked a standardized structure. - Models used inconsistent function names such as `predict()`, `run()`, and `inference()`. - Code still required modifications when moved into the serving environment. - Global configuration changes in one model could affect other models sharing the same process. - Logging, metrics, and error handling could not be applied consistently across models. ## Phase 2: Defining an Interface Contract The team ultimately standardized the boundary through the `commons-ml-model` package. - A base abstraction defines a common model structure. - DS implements three methods: - `pre_process` - `inference` - `post_process` - The base class handles shared concerns such as: - Logging - Metrics - Tracing - Timing and request tracking - DS packages the implementation as a reusable library. - MLE installs the package with `pip install` and deploys it without rewriting the model. This turns the division of work into a code-level contract. DS focuses on model behavior, while MLE owns serving infrastructure and operational concerns. Updating the base abstraction can also add observability features to every model at once. ## Monorepo Collaboration The team manages the abstraction package and individual model packages in a single monorepo using `uv` workspaces. - Changes to the abstraction and affected models can be reviewed in one pull request. - DS and MLE review the same code in the same repository. - CI, release, and versioning policies are centralized. - Switching from Poetry to `uv` improved build speed by three to five times. The tradeoff is that changes to shared packages can affect many models, and the repository becomes heavier as more packages accumulate. ## Standardizing AI-Generated Code AI-assisted development created a separate collaboration problem: consistent structure did not guarantee consistent coding style. The team introduced `pfmls-stylepack` to encode team conventions for AI tools. - Naming conventions are standardized. - Exception-handling patterns are prescribed. - Rules determine when to use enums instead of hard-coded strings. - Hooks apply conventions while code is being generated. - AI-generated code can explain when a particular rule influenced its implementation. The team therefore distinguishes between: - **Structural consistency:** interfaces define what each role implements. - **Style consistency:** shared rules define how code should be written. Both are necessary for smooth reviews. ## Lessons from the Evolution - The hardest decision is choosing the right collaboration boundary: excessive structure limits flexibility, while insufficient structure recreates inconsistency. - Documentation and early DS–MLE pairing reduce the learning curve for the package-based workflow. - Shared libraries are a double-edged sword: one change can cause broad impact, but one fix can also benefit every model. - In the age of AI-generated code, teams must standardize not only responsibilities and interfaces but also implementation style. The practical recommendation is to make collaboration contracts executable: define stable interfaces, package model code for reuse, centralize shared serving behavior, and enforce coding conventions automatically.
Read original(opens in new tab)
