Monorepo

13 posts

toss4 min readCurated summary

The Monorepo Hope Edition: One Year to Bring a Despairing Repo Back to Hope

Toss argues that a monorepo alone does not guarantee a consistent or efficient frontend development experience. The real problem was dependency-version fragmentation across services, which made installations slow, platform changes risky, and upgrades difficult. Toss addressed this by introducing shared dependency “catalogs,” standardizing core libraries while preserving controlled, gradual upgrades. ## Toss’s Frontend Development Environment - More than 100 frontend engineers maintain products inside and outside the Toss app. - Despite the large number of products, services use nearly identical versions of React 19, Next.js 15, TypeScript, bundlers, and linters. - A shared monorepo makes it easier to: - Maintain a consistent development environment. - Share code across services. - Propagate platform-wide changes. - Give users access to features such as React Concurrent Mode and Server Components. ## Problems with an Unmanaged Monorepo - Services used widely different dependency versions, including outdated React and supporting libraries. - This created fragmented developer experiences: - Some services had fast development servers and modern APIs. - Older services were slower and harder to develop. - Dependency installation could take more than a minute even with caching. - Platform teams struggled to test shared libraries across many React and library versions. - Service developers avoided upgrades because compatibility risks and migration costs were high. - Older services consequently became locked into outdated dependencies. ## Why Toss Rejected a Polyrepo Strategy - Splitting the monorepo into separate repositories could reduce the size of each individual project and improve installation times. - However, polyrepos would not solve the main issues: - Development environments would remain fragmented. - Shared-code development and updates would become more expensive. - Differences between services could become even more pronounced. - Toss concluded that improving dependency management within the monorepo was preferable to abandoning it. ## Simplifying the Dependency Tree - The central issue was that services selected different versions of the same core libraries. - Toss identified roughly 10–20 commonly used libraries, including: - React - Component libraries such as TDS - Jotai - TypeScript - ESLint - Standardizing these dependencies could: - Reduce installation time. - Provide a consistent developer experience. - Make platform-library testing more predictable. - Enable automated migration scripts and compatibility layers. - Lower the cost of adopting breaking changes. - In practice, developers usually chose libraries rather than requiring specific versions, making centralized versions practical. ## Dependency Catalogs - Toss defined recommended versions as a shared **Catalog** using pnpm or Yarn workspace configuration. - Services reference catalog-managed dependencies with the `catalog:` protocol instead of specifying independent versions. - Named catalogs can support different release channels, such as: - `stable` - `beta` - Toss initially included essential dependencies such as React, Next.js, TypeScript, TDS, and the Toss App SDK. - Catalog packages had to be tested in representative service environments before release. - New services automatically referenced the latest catalog. - CI detected cases where developers accidentally bypassed catalog versions. - Existing services were migrated collaboratively with their code owners. - Catalog changes were released as new versions and rolled out gradually rather than modifying a shared version in place. - Upgrade scripts and AI Skills reduced the effort required to migrate services. ## Results After Full Adoption - Dependency duplication fell substantially: - `.pnp.cjs` shrank from 96 MB to 15 MB, an approximately 84% reduction. - Development-server startup improved from 26.7 to 20.3 seconds, about 23% faster. - Full dependency installation decreased from 528.4 to 249.9 seconds, about 52% faster. - Developers gained greater confidence that catalog packages had already been tested in real services. - Centralized version control reduced incompatible transitive dependencies, such as one package requiring version 1 while a service used version 2. - Better dependency visibility made large architectural improvements safer, including work involving RSC, TypeScript 7, Rspack, and end-to-end testing. - Services could adopt improved platform packages more consistently and with less upgrade friction. The practical recommendation is to retain the monorepo, but enforce a curated set of shared dependency versions through catalogs, CI checks, staged releases, and automated migration tooling. This combines the sharing benefits of a monorepo with a more predictable and maintainable development environment.

Read original(opens in new tab)
toss4 min readCurated summary

How DS and MLE Work Together

The post explains how Toss Bank improved collaboration between Data Scientists (DS) and ML Engineers (MLE) by progressively formalizing their responsibilities. What began as manually transferring notebooks evolved into standardized Python files and finally into installable model packages built around explicit interfaces. The result was faster deployments, consistent observability, and clearer ownership, while AI-generated code introduced a new need to standardize coding style as well. ## Problems with Notebook-Based Handoffs Initially, DS built models and inference code in Jupyter notebooks, then handed them to MLE. - MLE had to recreate the serving code from scratch. - Dependencies, configuration files, and source code were often missing or difficult to reproduce. - Preprocessing logic could be interpreted differently by DS and MLE. - “It works in the notebook” did not guarantee that it would work in production. - As the number of models increased, communication and rework grew rapidly. This approach separated people, not code, so the division of responsibility remained unclear. ## Phase 1: Separating Logic into `.py` Files The team next moved the collaboration boundary from people to files. - DS kept notebooks for experimentation and training. - Core inference logic was extracted into `.py` files. - MLE reviewed these files and validated them through CI. - DS’s intended model behavior was preserved more reliably. - Communication costs decreased. However, the files lacked a standardized structure. - Models used inconsistent function names such as `predict()`, `run()`, and `inference()`. - Code still required modifications when moved into the serving environment. - Global configuration changes in one model could affect other models sharing the same process. - Logging, metrics, and error handling could not be applied consistently across models. ## Phase 2: Defining an Interface Contract The team ultimately standardized the boundary through the `commons-ml-model` package. - A base abstraction defines a common model structure. - DS implements three methods: - `pre_process` - `inference` - `post_process` - The base class handles shared concerns such as: - Logging - Metrics - Tracing - Timing and request tracking - DS packages the implementation as a reusable library. - MLE installs the package with `pip install` and deploys it without rewriting the model. This turns the division of work into a code-level contract. DS focuses on model behavior, while MLE owns serving infrastructure and operational concerns. Updating the base abstraction can also add observability features to every model at once. ## Monorepo Collaboration The team manages the abstraction package and individual model packages in a single monorepo using `uv` workspaces. - Changes to the abstraction and affected models can be reviewed in one pull request. - DS and MLE review the same code in the same repository. - CI, release, and versioning policies are centralized. - Switching from Poetry to `uv` improved build speed by three to five times. The tradeoff is that changes to shared packages can affect many models, and the repository becomes heavier as more packages accumulate. ## Standardizing AI-Generated Code AI-assisted development created a separate collaboration problem: consistent structure did not guarantee consistent coding style. The team introduced `pfmls-stylepack` to encode team conventions for AI tools. - Naming conventions are standardized. - Exception-handling patterns are prescribed. - Rules determine when to use enums instead of hard-coded strings. - Hooks apply conventions while code is being generated. - AI-generated code can explain when a particular rule influenced its implementation. The team therefore distinguishes between: - **Structural consistency:** interfaces define what each role implements. - **Style consistency:** shared rules define how code should be written. Both are necessary for smooth reviews. ## Lessons from the Evolution - The hardest decision is choosing the right collaboration boundary: excessive structure limits flexibility, while insufficient structure recreates inconsistency. - Documentation and early DS–MLE pairing reduce the learning curve for the package-based workflow. - Shared libraries are a double-edged sword: one change can cause broad impact, but one fix can also benefit every model. - In the age of AI-generated code, teams must standardize not only responsibilities and interfaces but also implementation style. The practical recommendation is to make collaboration contracts executable: define stable interfaces, package model code for reuse, centralize shared serving behavior, and enforce coding conventions automatically.

Read original(opens in new tab)
figma3 min readCurated summary

How Figma Stays Ahead of Vulnerabilities With Agents | Figma Blog

Figma uses AI agents to prevent, detect, and fix vulnerabilities during code generation, pull-request review, and historical code audits. Its central strategy is a shared security policy combined with continuous measurement of precision and recall. The company began with PR review because developer feedback and replaying known bugs created the fastest improvement loop. ## A Shared Policy Across the Development Lifecycle - The same policy guides agents during: - Code generation - Pull-request review - Full-repository auditing - The policy records: - Trust boundaries - Accepted risks - Security precedents - Improving PR review first helped Figma refine the policy before applying it to other stages. ## Measuring Precision and Recall - **Precision** measures how many reported findings are genuine vulnerabilities; low precision creates false positives. - **Recall** measures how many real vulnerabilities the system detects; low recall creates false negatives. - Figma measures: - Precision through author thumbs-up or thumbs-down feedback on findings. - Recall by replaying the reviewer against commits containing known bugs. - These separate signals allow the team to improve both dimensions rather than optimizing for only one. ## AI-Assisted Pull-Request Review - Every pull request receives an automated review. - Findings are posted directly to the PR, allowing developers to respond and fix issues in context. - Figma currently runs: - Claude Code with Opus 4.8 at extra-high effort - Codex with GPT-5.6 Sol at high effort - The models identify different classes of bugs, so Figma reports a finding if either model detects one. - Reviews cost approximately $0.50 per pull request at the median. The agents have identified both sophisticated and conventional vulnerabilities, including: - A multi-step desktop-client exploit in which an injected sandbox object exposed the host realm’s `Function` constructor and enabled possible code execution. - An insecure direct object reference where an authenticated user could retrieve another organization’s invoice by supplying its ID. ## Building Trust Before Exposing Findings - Figma launched Anthropic’s Claude Code Security Reviewer in August 2025 in shadow mode. - Initial results were strong at reproducing known vulnerability root causes, but only 4 of 27 findings—about 15%—were valid. - The team prioritized precision first because developers quickly lose trust in tools that generate excessive false positives. - Figma set a practical target of at least 70% precision. - Developer-facing comments were withheld until precision exceeded that level over a two-week period without severe false positives. - Security engineers replayed the reviewer across eight weeks of historical pull requests and manually labeled incorrect findings. - Those examples were used to create and refine the shared security policy. ## Continuous Improvement Through Precedents - A **precedent** documents why a finding is valid or invalid in a particular context. - Human feedback and historical vulnerability replays feed back into the policy. - This process lets Figma improve the agent automatically rather than relying only on model changes or one-time prompt tuning. Figma’s approach treats agentic security as an engineering and measurement problem, not simply a matter of asking an AI to scan code. Organizations adopting similar systems should establish feedback loops, measure precision and recall independently, and build developer trust before making automated findings part of everyday development.

Read original(opens in new tab)
gitlab3 min readCurated summary

GitLab: Built for the agentic engineering era

GitLab argues that AI coding only becomes truly “agentic engineering” when paired with infrastructure built for machine-scale concurrency, lifecycle context, and enterprise governance. At GitLab Transcend, it announced new source control, context, security, orchestration, and purchasing capabilities designed to let agents work faster without sacrificing control. The overall goal is to convert rapid AI-generated development into reliable business value rather than unmanaged complexity. ## The challenge: speed without control - Research across more than 1,500 developers and technology leaders found: - 91% of organizations use at least two AI coding tools. - 54% use three or more. - Some customer codebases are growing by as much as five times per year. - Fragmented development lifecycles create several problems: - Human-scale source control systems struggle with thousands of concurrent agents. - Agents lack context about dependencies, deployments, and production behavior. - Rapidly changing code is difficult to govern. - Fixed contracts make AI adoption difficult to forecast. - 73% of respondents worry about maintaining AI-generated code, while only 21% see productivity improvements across the full SDLC. ## GitLab’s agentic infrastructure model GitLab presents its platform as four coordinated systems: - **Motor system:** Source control, pipelines, and deployments that execute work. - **Nervous system:** Context that helps agents and humans make informed decisions. - **Immune system:** Security, governance, identity, policy, audit, and approvals. - **Orchestration system:** GitLab Duo Agent Platform, which coordinates work across the lifecycle. The company says these systems operate consistently whether work is performed by a developer or an agent. ## Next-generation source control for agent concurrency Git’s traditional workflow creates bottlenecks when every developer runs hundreds of agents: - Agents repeatedly clone repositories even when they need only one file. - Thousands of simultaneous sessions can overwhelm a human-oriented backend. - Shared accounts and branches make it difficult to isolate, audit, or discard agent work. GitLab’s next-generation SCM, currently in private beta, retains Git protocol compatibility while redesigning the backend and interfaces for agents. It is intended to support thousands of parallel agents working safely across repositories. Early internal tests reported: - Up to 2× fewer tokens - Up to 50× faster wall-clock execution - Up to 1,000× less network traffic ## GitLab Orbit: lifecycle context for agents Agents often understand the code they modify but not the broader software lifecycle, causing wasted iterations, hallucinations, and incorrect work across large or multiple repositories. GitLab Orbit, in public beta, provides a continuously updated context graph connecting: - Code - Work items - Pipelines - Deployments - Production signals This gives agents and engineers a shared source of truth. GitLab reports that Orbit-grounded agents achieved up to 11× faster responses, 4.5× better cost efficiency, and 45× fewer hallucinations in early testing. Compare the Market’s testing on 79 merge requests found that graph-grounded agents placed inline review comments correctly 69.6% of the time, compared with 57.7% for a conventional RAG approach. ## Governance, orchestration, and purchasing GitLab also announced: - **Agents for security and governance for agents**, covering identity, policy, auditing, and approval of agent actions, in private beta. - **GitLab Duo Agent Platform**, generally available since January, allowing agents to pick up issues, review code, and fix pipelines. - **GitLab Flex**, a purchasing model intended to accommodate the unpredictable pace of AI adoption. - A Transcend hackathon inviting developers to build agents and workflows using Orbit. GitLab’s practical recommendation is not to slow down AI-assisted development, but to pair it with dedicated infrastructure for concurrency, full-lifecycle context, and enforceable governance.

Read original(opens in new tab)
dropbox3 min readCurated summary

Introducing Nova, our internal platform for coding agents

Nova is Dropbox’s internal cloud platform for running coding agents across the software development lifecycle. Rather than building separate tools for coding, CI debugging, migrations, and operational tasks, Dropbox created a shared platform that supports interactive sessions and autonomous workflows within its monorepo and infrastructure. The platform grounds agent changes in real builds and tests, making AI assistance more reliable and easier to integrate into engineering workflows. ## The Case for a Shared Platform - Engineering work includes repetitive but important tasks such as: - Debugging CI failures - Updating dependencies - Improving test coverage - Fixing flaky tests - Managing migrations and operational work - Different tasks require different interaction models: - Interactive chat for developer-driven work - Asynchronous workflows for long-running remediation and automation - Dropbox’s environment has specialized requirements: - A large monorepo - Bazel for builds and tests - Caching and remote execution - On-premises infrastructure - Dropbox-specific validation workflows - Off-the-shelf coding agents were designed primarily for local development and did not naturally fit this environment. ## How Nova Runs Coding Sessions - Each session runs in an isolated environment using a specific snapshot of the codebase. - Callers provide: - The repository commit - A task description - Optional validation commands - Iteration limits and branch settings - Nova can run builds and tests after an agent proposes a change. - If validation fails, the results are sent back to the agent so it can continue troubleshooting. - This creates a feedback loop of: - Propose a change - Validate it in the real environment - Correct failures - Repeat as needed - Nova supports multiple coding agents behind a common interface. - Engineers can access it through: - A web interface - A command-line client - An API - Internal scripts and services - The platform also provides prompt evaluation, observability, feedback collection, skills, plugins, and MCP integrations for accessing systems such as logs and monitoring tools. ## Deterministic Code Publication - Nova keeps code publication outside the agent. - Each session is limited to a single branch. - This makes active work and publication status predictable. - It avoids the complexity of agents creating and managing multiple branches. - The deterministic model simplifies automation such as: - Running tests - Rebasing onto the main branch - Tracking which changes belong to each session ## Engineering Workflows Using Nova ### Developer-Driven Sessions - Engineers use Nova’s web interface for quick fixes and prototypes without disrupting local work. - Validation commands can use Bazel selectivity tools to target the relevant compile and test dependencies. - Slack discussions can be carried into Nova sessions, preserving context and reducing manual setup. ### Flaky Test Remediation - Dropbox built Deflaker, a durable workflow connected to Athena, its flaky-test detection system. - Deflaker gathers examples of a test passing and failing. - It sends the associated logs to Nova. - The agent analyzes the evidence, identifies a likely cause, and proposes a fix. - This demonstrates how Nova can combine investigation, context gathering, and code changes in a longer-running automated process. ## Practical Takeaway Dropbox’s experience suggests that coding agents are most useful when embedded in existing engineering systems rather than treated as isolated code-generation tools. A shared platform like Nova can support many workflows while preserving consistent execution, validation, context, and observability.

Read original(opens in new tab)
meta3 min readCurated summary

Escaping the Fork: How Meta Modernized WebRTC Across 50+ Use Cases

Meta escaped the “forking trap” by replacing its divergent WebRTC fork with a modular architecture based on the latest upstream release. The system builds legacy and current WebRTC versions side by side, enabling runtime A/B testing across more than 50 use cases before rollout. This improved performance, binary size, and security while establishing a repeatable process for continuous upstream upgrades. ## Why the WebRTC Fork Became a Problem - Meta’s RTC stack supports Messenger, Instagram video calls, Cloud Gaming, and Meta Quest casting. - Internal optimizations and bug fixes gradually caused its WebRTC fork to diverge from upstream. - As the fork accumulated custom changes, merging community improvements became increasingly expensive and risky. - A one-time upgrade was impractical because WebRTC serves billions of users across diverse devices and environments. ## Requirements for a Sustainable Upgrade Strategy - Meta needed to: - Run legacy and upstream-based WebRTC implementations simultaneously. - Dynamically assign users to either version for safe A/B testing. - Statically link both versions into the same application. - Maintain custom patches in a monorepo without repeatedly rebuilding the migration process. - Standard patch-file workflows were considered difficult to scale for Meta’s large codebase. ## Shim Layer and Dual-Stack Architecture - A shim library was placed between application code and WebRTC. - Applications call a unified, version-neutral API rather than calling either WebRTC implementation directly. - A runtime “flavor” configuration routes each call to either the legacy or latest implementation. - Shimming at the lowest practical layer avoided duplicating the higher-level call orchestration library: - Full duplication would have added about 38 MB uncompressed. - The shim-based design added roughly 5 MB, an 87% reduction. ## Resolving C++ Symbol Collisions - Linking two WebRTC copies normally violates the C++ One Definition Rule and creates thousands of duplicate symbols. - Meta automated namespace rewriting: - `webrtc::` in the current version became `webrtc_latest::`. - The legacy version became `webrtc_legacy::`. - Global functions, variables, and classes outside namespaces were moved into namespaces where possible or assigned flavor-specific names. - Macro conflicts, including `RTC_CHECK` and `RTC_LOG`, were addressed by: - Removing unnecessary includes. - Renaming infrequently used macros. - Sharing modules such as `rtc_base` between versions to reduce duplication and shimming work. ## Preserving Backward Compatibility - Renaming symbols could have broken existing call sites, especially code built for only one WebRTC flavor. - An initial solution forward-declared every required symbol, but this created a large and fragile maintenance burden. - The improved approach used C++ `using` declarations to bulk-import a flavor namespace into the familiar `webrtc::` namespace. - This preserved existing source-level APIs without adding binary overhead, while allowing Meta to migrate selected call sites incrementally. ## Runtime Flavor Dispatch - Shim adapters and converters must instantiate objects from either the legacy or current namespace. - A template-based helper library keeps shared adapter logic in one place. - Template specializations handle version-specific behavior. - A global flavor enum, initialized during application startup, determines which WebRTC implementation is used. - The design also supports single-flavor builds during the transition. Meta’s approach demonstrates that large internal modifications do not have to require a permanent fork. A low-level shim, automated renamespacing, compatibility imports, and template-based dispatch provide a practical foundation for continuously rebasing custom functionality onto upstream WebRTC while safely validating each release through A/B testing.

Read original(opens in new tab)
gitlab2 min readCurated summary

5 ways GitLab pipeline logic solves engineering problems

GitLab’s pipeline model addresses complex CI/CD needs by combining composable features rather than relying on a single linear workflow. Parent-child pipelines, DAG execution, and multi-project triggers help teams scale monorepos and coordinate services across repositories while preserving clear ownership and failure visibility. The article argues that these patterns make pipelines both faster and easier to maintain. ## Monorepos: Parent-child pipelines and DAG execution - A monorepo containing frontend, backend, and documentation projects should not rebuild everything for every change. - Parent pipelines can trigger child pipelines for individual services using `trigger: include`. - Multiple included files are merged into one child pipeline, allowing jobs across files to share context and reference one another with `needs:`. - `strategy: depend` makes the parent wait for child pipelines and report one overall success or failure while retaining detailed drill-down. - Each service can own its pipeline configuration, reducing the risk that changes in one service break another. - DAG execution with `needs:` allows dependent jobs to start as soon as their prerequisites finish instead of waiting for an entire stage. - For example, API tests can begin immediately after the API build completes, without waiting for unrelated jobs. ## Microservices: Cross-repository pipelines - When frontend and backend services live in separate repositories, independent pipelines may miss integration failures. - GitLab multi-project pipelines allow one repository to trigger and await a pipeline in another project. - The frontend can generate an API contract artifact, publish it, and trigger the backend pipeline with `strategy: depend`. - The backend downloads the artifact through the GitLab Jobs API using `CI_JOB_TOKEN`. - An integration test can reject breaking API changes and propagate the failure back to the frontend pipeline. - The backend job uses `CI_PIPELINE_SOURCE == "pipeline"` so the contract validation runs only when initiated by the frontend, not during ordinary backend pushes. - The frontend project identifier is supplied through a CI/CD variable such as `FRONTEND_PROJECT_ID`. These patterns let teams reduce unnecessary work, preserve service-level ownership, and make cross-service compatibility checks part of the delivery process.

Read original(opens in new tab)
dropbox3 min readCurated summary

Reducing our monorepo size to improve developer velocity

Dropbox’s server monorepo grew to 87GB, making full clones take over an hour and threatening GitHub’s 100GB limit. The root cause was inefficient Git delta compression of internationalization files, not unusually large source files. By changing how the repository was repacked, Dropbox reduced it to about 20GB and cut clone times to under 15 minutes. ## Repository Size and Developer Velocity - The monorepo contains backend services and libraries used across Dropbox. - AI feature development often requires coordinated changes across ranking, retrieval, evaluation, and UI systems. - A full clone exceeded one hour at 87GB, slowing onboarding and affecting CI jobs that start from fresh clones. - Internal synchronization systems also processed more data, increasing timeout and reliability risks. - The repository grew by roughly 20–60MB per day, with occasional increases above 150MB. - At that rate, Dropbox expected to hit GitHub Enterprise Cloud’s 100GB hard limit within months. ## How Git Compression Caused the Growth - Git normally reduces storage by representing similar file versions as deltas rather than complete copies. - Its default file-matching heuristic considers only the final 16 characters of a path. - Dropbox’s i18n files used paths such as: - `i18n/metaserver/[language]/LC_MESSAGES/[filename].po` - Because the language component appears early in the path, Git often compared files from different languages instead of related versions of the same language. - Translation updates consequently produced oversized deltas and disproportionately large pack files. ## Testing `--path-walk` - Dropbox tested Git’s experimental `--path-walk` option during a local repack. - The option considers the full directory structure when selecting delta candidates. - A local repack reduced the repository from the low-80GB range to the low-20GB range, confirming that packing—not data volume—was the main issue. - GitHub could not use this approach because it conflicted with server-side optimizations such as bitmaps and delta islands. ## Why Server-Side Repacking Was Necessary - Local optimization cannot permanently change the packs GitHub generates for clones and fetches. - GitHub dynamically constructs transfer packs based on what each client needs. - Dropbox’s mirror experiment showed that an aggressive repack could reduce the repository from 84GB to 20GB: - `git repack -adf --depth=250 --window=250` - The repack took approximately nine hours. - Dropbox worked with GitHub Support to apply a compatible server-side solution. - Larger `window` and `depth` values make Git search more thoroughly for compression opportunities, trading increased repack time for smaller storage and transfer sizes. ## Results - Repository size fell from 87GB to approximately 20GB—a 77% reduction. - Clone time dropped from more than an hour to under 15 minutes. - The work reduced pressure on GitHub’s repository size limit and improved the performance of developer and CI workflows. Dropbox’s experience shows that monorepo growth can result from repository layout interacting poorly with Git’s compression heuristics. When large repositories exhibit abnormal growth, teams should inspect pack-file behavior and consider server-side repacking rather than focusing only on removing large files.

Read original(opens in new tab)
dropbox2 min readCurated summary

Insights from our executive roundtable on AI and engineering productivity

Dropbox argues that AI improves engineering productivity only when tied to measurable business outcomes rather than adopted for its own sake. The company has expanded AI use across the software development lifecycle, while recognizing trade-offs involving quality, maintenance, and organizational change. Its executive roundtable concluded that leadership, formal AI competency, and stronger outcome measurement will be central to realizing AI’s potential. ## Dropbox’s AI Adoption Strategy - Dropbox made AI adoption a company-wide priority with leadership sponsorship, enabling teams to experiment more easily and reducing delays in approving new tools. - Engineers use AI across code review, documentation, debugging, testing, and other stages of development. - Because Dropbox operates a large, multilingual monorepo, it combines commercial tools such as Claude Code and Cursor with internally built systems. - One internal tool detects failed pull-request builds and uses Dropbox’s AI platform to suggest fixes. - Most developers now use at least one AI tool. - Dropbox tracks monthly pull-request throughput per engineer and has observed higher output among developers who use AI coding tools more actively. - The company also monitors engineer sentiment, reporting increased positive sentiment and reduced negative sentiment as adoption improves. ## Focus of the Executive Roundtable Leaders from multiple companies discussed engineering productivity and AI in rotating peer groups organized around three themes: - **Measuring impact** - Identifying ways to measure AI-driven productivity gains. - Connecting engineering improvements to broader business results. - **Leadership alignment** - Establishing how executives should communicate AI deployment progress. - Determining the appropriate pace and scope of adoption. - **The human element** - Recruiting, evaluating, and developing AI-capable employees. - Applying lessons from developer productivity to help non-engineering teams work more effectively. ## Lessons About AI and Productivity - **Balance is essential:** Faster development must not come at the expense of software quality or increased long-term maintenance costs. - **Leadership sets standards:** Technical managers play a key role in defining responsible and effective AI usage norms. - **AI skills should be formalized:** Including AI competency in career frameworks demonstrates that it is a lasting strategic capability rather than a temporary trend. - **Extra capacity needs direction:** Dropbox is currently using productivity gains to address technical debt, complete migrations, and improve reliability. ## Priorities for 2026 Dropbox’s main unresolved challenge is linking engineering productivity metrics to tangible business outcomes. Its next phase will focus on mapping AI-driven gains to specific results, extending operational discipline beyond engineering, and improving end-to-end product velocity.

Read original(opens in new tab)
daangnOriginal article

The Journey of Karrot Pay’ (opens in new tab)

Daangn Pay’s backend evolution demonstrates how software architecture must shift from a focus on development speed to a focus on long-term sustainability as a service grows. Over four years, the platform transitioned from a simple layered structure to a complex monorepo powered by Hexagonal and Clean Architecture principles to manage increasing domain complexity. This journey highlights that technical debt is often the price of early success, but structural refactoring is essential to support organizational scaling and maintain code quality. ## Early Speed with Layered Architecture * The initial system was built using a standard Controller-Service-Repository pattern to meet the urgent deadline for obtaining an electronic financial business license. * This simple structure allowed for rapid development and the successful launch of core remittance and wallet features. * As the service expanded to include promotions, billing, and points, the "Service" layer became overloaded with cross-cutting concerns like validation and permissions. * The lack of strict boundaries led to circular dependencies and "spaghetti code," making the system fragile and difficult to test or refactor. ## Decoupling Logic via Hexagonal Architecture * To address the tight coupling between business logic and infrastructure, the team adopted a Hexagonal (Ports and Adapters) approach. * The system was divided into three distinct modules: `domain` (pure POJO rules), `usecase` (orchestration of scenarios), and `adapter` (external implementations like DBs and APIs). * This separation ensured that core business logic remained independent of the Spring Framework or specific database technologies. * While this solved dependency issues and improved reusability across REST APIs and batch jobs, it introduced significant boilerplate code and the complexity of mapping between different data models (e.g., domain entities vs. persistence entities). ## Scaling to a Monorepo and Clean Architecture * As Daangn Pay grew from a single project into dozens of services handled by multiple teams, a Monorepo structure was implemented using Gradle multi-projects. * The architecture evolved to separate "Domain" modules (pure business logic) from "Service" modules (the actual runnable applications like API servers or workers). * An "Internal-First" policy was adopted, where modules are private by default and can only be accessed through explicitly defined public APIs to prevent accidental cross-domain contamination. * This setup currently manages over 30 services, providing a balance between code sharing and strict boundary enforcement between domains like Money, Billing, and Points. The evolution of Daangn Pay’s architecture serves as a practical reminder that there is no "perfect" architecture from the start; rather, the best design is one that adapts to the current size of the organization and the complexity of the business. Engineers should prioritize flexibility and structural constraints that guide developers toward correct patterns, ensuring the codebase remains manageable even as the team and service scale.

airbnb4 min readCurated summary

Migrating Airbnb’s JVM Monorepo to Bazel

Airbnb migrated its tens-of-millions-of-lines JVM monorepo from Gradle to Bazel over 4.5 years, achieving faster builds, testing, IntelliJ syncs, and development deployments. The move was driven by Bazel’s scalable remote execution, hermetic builds, and ability to provide shared infrastructure across Airbnb’s language-specific repositories. A gradual rollout, extensive automation, and close collaboration with service teams were central to making the migration successful. ## Results of the Migration - Build CSAT increased from 38% to 68%. - Local build and test times became 3–5 times faster. - IntelliJ syncs became 2–3 times faster. - Development-environment deployments became 2–3 times faster. ## Why Airbnb Chose Bazel ### Faster Builds Through Remote Execution - Large Gradle builds frequently took more than 20 minutes locally, while pre-merge CI builds had a p90 of 35 minutes. - Gradle had already been optimized with powerful machines and build sharding, but sharding caused underutilization and duplicated shared work. - Bazel’s cacheable actions and remote build execution enabled thousands of actions to run in parallel on short-lived workers. - “Build without the Bytes” reduced the amount of build output developers needed to download. - Bazel analysis runs in parallel, unlike the often single-threaded configuration phase of large Gradle projects. - Remote execution also improved local build performance, not just CI performance. ### More Reliable and Reproducible Builds - Gradle tasks could access the entire filesystem, creating accidental dependencies and race conditions. - Bazel sandboxes expose only declared inputs to each action, preventing undeclared files from affecting builds. - Bazel’s remote execution runs actions in identical containers with strict resource limits. - Using remote execution for both local and CI builds reduced differences between developer and CI environments. ### A Shared Build Infrastructure Layer Because Airbnb’s web, iOS, Python, Go, and JVM repositories all use Bazel, the company could standardize infrastructure for: - Remote caching - Remote build execution - Affected-target calculation - Build Event Protocol instrumentation and logging ## Starting with a Proof of Concept - Airbnb first migrated Viaduct, a large GraphQL monolith platform. - Viaduct was selected because it was complex, had slow builds, affected roughly 300 product engineers monthly, and had an infrastructure team willing to collaborate. - Bazel and Gradle initially coexisted, allowing developers to choose either system. - The team ported Viaduct’s build logic and created an automated Bazel build-file generator because the Gradle dependency graph continued to change. - Although Bazel was initially 2–4 times faster locally, developers did not adopt it immediately. - The team spent several additional months fixing missing integrations and bugs before Viaduct engineers voluntarily switched. ## Scaling Across the JVM Monorepo - Airbnb expanded breadth-first, aiming to make the entire repository compile and test under Bazel. - Gradle and Bazel continued to coexist during the migration. - This allowed developers to use Bazel locally while deployments still relied on Gradle. - Gradle provided a fallback when Bazel infrastructure, such as remote caching or execution, experienced incidents. - Maintaining two build graphs was costly, so Airbnb invested heavily in automation rather than requiring developers to maintain Bazel files manually. ## Automated Build-File Generation - The generator was inspired by Gazelle but was built internally to meet stricter performance requirements and handle dependency cycles. - It parses Java, Kotlin, and Scala source files to identify packages, imports, and symbol declarations. - These relationships are used to construct a file-level dependency graph. - Since generation ran on every commit before merging, Airbnb added external caching to keep it fast. - CI publishes a cached repository index for each mainline commit, allowing the generator to rescan only directories changed since that commit. Airbnb’s experience suggests that a large build-system migration is most effective when introduced incrementally: prove the benefits on a representative service, automate maintenance, preserve a fallback during rollout, and address developer workflow issues before expanding across the organization.

Read original(opens in new tab)
slack3 min readCurated summary

Optimizing Our E2E Pipeline

Slack optimized its monorepo E2E pipeline by avoiding frontend rebuilds when a pull request contains no frontend changes. Using `git diff` to detect relevant changes and serving recent frontend artifacts from S3 through an internal CDN, the team reduced build frequency by 60% and cut end-to-end pipeline time from roughly 10 minutes to 2 minutes. The changes also lowered storage and compute costs and improved test reliability. ## The Cost of Unnecessary Frontend Builds - Slack’s E2E pipeline validates frontend, backend, database, and service changes before merging into `main`. - Previously, every run rebuilt the frontend, even when a pull request changed only backend or unrelated files. - A typical pipeline included: - About 5 minutes for the frontend build - Deployment to QA - More than 200 E2E tests taking another 5 minutes - With hundreds of pull requests merged daily, redundant builds caused: - Thousands of unnecessary builds each week - Nearly a gigabyte of S3 data per build - Terabytes of duplicate stored artifacts - Significant developer and cloud-compute costs ## Conditional Frontend Builds - Slack used `git diff` with three-dot notation to compare the checked-out branch against `main`. - If frontend files had changed, the pipeline ran a new frontend build. - If no frontend changes were detected, the build step was skipped. - Git analyzed the repository’s more than 100,000 tracked files in only a few seconds. ## Reusing Prebuilt Assets - When a new build was unnecessary, the pipeline located a recent frontend build already stored in AWS S3. - The selected artifact was still in production, ensuring the E2E tests used sufficiently current frontend assets. - An internal CDN served those assets to the QA environment. - S3 naming and asset-management conventions made it possible to find an appropriate artifact in under three seconds on average. ## Results and Additional Benefits - Frontend build frequency fell by 60%. - Average E2E pipeline time dropped from about 10 minutes to 2 minutes. - Monthly savings included hundreds of hours of compute and developer waiting time. - S3 usage decreased by several terabytes per month. - Test flakiness reached its lowest measured level, partly because asset delivery became more consistent. - The work also exposed legacy systems and generated a backlog of future maintenance improvements. Slack’s experience demonstrates that pipelines should not automatically repeat expensive steps when their inputs have not changed. Detecting affected files and reusing trustworthy build artifacts can substantially improve speed, reliability, and cost without requiring a wholesale rewrite of the CI/CD system.

Read original(opens in new tab)
figma2 min readCurated summary

Keeping Figma Fast | Figma Blog

Figma’s original performance-testing system relied on a single MacBook running a few repeated scenarios, which worked when the product and team were smaller. As Figma expanded into plugins, FigJam, Dev Mode, and many other features, that approach became too limited and fragile—the laptop eventually overheated while the company was remote. Figma concluded it needed a scalable, proactive testing framework capable of catching regressions across its growing codebase. ## From One Laptop to a Growing Product - In 2018, one MacBook repeatedly ran performance scenarios and reported timing changes to a shared dashboard. - A major renderer restructuring and WebAssembly fixes made Figma three times faster. - Over five years, the product grew substantially, adding: - Plugins and Community features - FigJam - Dev Mode - Numerous ongoing product updates - The existing test files could no longer represent the product’s expanding feature set and edge cases. ## Limits of the Original Testing Process - Granular performance tests measure specific behavior at scale, such as panning through a file while 100 users edit layers and type simultaneously. - Creating feature-specific tests became impractical as Figma grew beyond 400 engineers and managers. - Increasing release velocity made it difficult for any individual or single machine to track every performance-affecting change. - During the shift to remote work in 2020, the office MacBook remained running unattended and eventually overheated. - Attempts to reproduce the setup on another laptop were unsuccessful, demonstrating that the system was not operationally scalable. ## Requirements for a New System Figma used the overhaul to define an ideal performance-testing framework: - **Test every proposed code change:** Performance checks should run against changes in the main monorepo, allowing regressions to be found during development rather than after users encounter them. - **Support proactive performance work:** Small delays can significantly disrupt users who spend hours working in Figma. - **Run tests in parallel:** Dozens of stress scenarios would need to execute simultaneously, similar to Figma’s existing cloud-based CI testing. - **Finish quickly:** Performance guardrail checks were required to complete in under 10 minutes. - **Scale across real hardware:** Running every pull request on physical machines could require roughly 100 identical runners at peak capacity. Figma’s experience shows that performance testing must evolve alongside product complexity. A lightweight single-machine setup can be effective initially, but larger teams and faster release cycles require automated, parallel, hardware-aware testing integrated directly into CI.

Read original(opens in new tab)