Optimizing Our E2E Pipeline (opens in new tab)
Slack optimized its monorepo E2E pipeline by avoiding frontend rebuilds when a pull request contains no frontend changes. Using `git diff` to detect relevant changes and serving recent frontend artifacts from S3 through an internal CDN, the team reduced build frequency by 60% and cut end-to-end pipeline time from roughly 10 minutes to 2 minutes. The changes also lowered storage and compute costs and improved test reliability. ## The Cost of Unnecessary Frontend Builds - Slack’s E2E pipeline validates frontend, backend, database, and service changes before merging into `main`. - Previously, every run rebuilt the frontend, even when a pull request changed only backend or unrelated files. - A typical pipeline included: - About 5 minutes for the frontend build - Deployment to QA - More than 200 E2E tests taking another 5 minutes - With hundreds of pull requests merged daily, redundant builds caused: - Thousands of unnecessary builds each week - Nearly a gigabyte of S3 data per build - Terabytes of duplicate stored artifacts - Significant developer and cloud-compute costs ## Conditional Frontend Builds - Slack used `git diff` with three-dot notation to compare the checked-out branch against `main`. - If frontend files had changed, the pipeline ran a new frontend build. - If no frontend changes were detected, the build step was skipped. - Git analyzed the repository’s more than 100,000 tracked files in only a few seconds. ## Reusing Prebuilt Assets - When a new build was unnecessary, the pipeline located a recent frontend build already stored in AWS S3. - The selected artifact was still in production, ensuring the E2E tests used sufficiently current frontend assets. - An internal CDN served those assets to the QA environment. - S3 naming and asset-management conventions made it possible to find an appropriate artifact in under three seconds on average. ## Results and Additional Benefits - Frontend build frequency fell by 60%. - Average E2E pipeline time dropped from about 10 minutes to 2 minutes. - Monthly savings included hundreds of hours of compute and developer waiting time. - S3 usage decreased by several terabytes per month. - Test flakiness reached its lowest measured level, partly because asset delivery became more consistent. - The work also exposed legacy systems and generated a backlog of future maintenance improvements. Slack’s experience demonstrates that pipelines should not automatically repeat expensive steps when their inputs have not changed. Detecting affected files and reusing trustworthy build artifacts can substantially improve speed, reliability, and cost without requiring a wholesale rewrite of the CI/CD system.
