Shipyard: How We Built Slack’s Next-Generation EC2 Platform
Slack’s Shipyard is a next-generation EC2 platform that replaces continuously modified, long-lived instances with immutable, deployable infrastructure artifacts. It combines layered machine images, service-level deployments, progressive rollouts, automated rollback, and short-lived instances to make EC2 operations more predictable and secure. The platform preserves EC2’s flexibility for workloads that cannot easily move to containers while adopting modern application-delivery practices. ## Why Slack Built Shipyard - Slack previously improved its Chef infrastructure with: - Multiple resilient Chef stacks - Versioned cookbook deployments - Safer promotion workflows - Split production environments and signal-based Chef runs - Despite these improvements, the traditional model of continuously updating instances still caused: - Infrastructure drift - Difficult service-level deployments - Coordination problems across infrastructure layers - Increasing operational complexity - Shipyard shifts infrastructure management from mutable instances and constant configuration enforcement to build pipelines, deployable artifacts, and automated safety mechanisms. ## Shipyard’s Core Capabilities - Supports AMD64 and ARM-based Graviton processors. - Supports Ubuntu, RHEL, and Amazon Linux. - Targets workloads that cannot easily migrate to containers, including: - Infrastructure services - Kubernetes worker nodes - Egress network stacks - Integrates with Slack’s Gondola deployment orchestrator for progressive rollouts. - Uses service health metrics to automatically pause deployments or roll back to a known-good version. - Uses layered images so instances perform less work during startup and provision quickly and consistently across regions. ## Configuration Management Without Continuous Mutation - Under the previous model, scheduled Chef jobs repeatedly checked and reapplied configuration. - Shipyard applies configuration during defined lifecycle stages, primarily: - Image baking - Initial instance provisioning - Service deployment - Configuration management tools no longer continuously modify the entire running system. - This reduces background workload, prevents unexpected overwrites, and makes instance behavior easier to understand. ## Fleet Inventory with Peekaboo - Shipyard introduces Peekaboo, a near-real-time inventory system for EC2. - Peekaboo uses AWS EventBridge, OpenSearch, and Lambda to collect cloud events and instance metadata. - It provides: - A fleet-exploration UI - An API for integrations - A command-line interface - Unlike Chef Server, Peekaboo is not limited to Shipyard-managed instances and can provide visibility across the entire EC2 fleet. ## Short-Lived Immutable Instances - Shipyard regularly rotates instances instead of relying on in-place updates. - Limited instance lifespans: - Reduce the window in which vulnerabilities can persist - Prevent long-term configuration drift - Encourage teams to replace instances rather than repair them manually - This makes infrastructure more closely resemble immutable application artifacts. ## The `slack-zero` Golden Image - `slack-zero` is Slack’s shared foundational AMI, maintained by the Compute Platform Team with security and monitoring teams. - It includes: - Operating-system hardening - Networking and service discovery - Monitoring and security agents - Common tools and foundational configuration - Service-specific images are built on top of `slack-zero`, similar to layering application images on a Docker base image. - When foundational components change, Slack produces a new immutable base image and rebuilds downstream service images to inherit updates. ## AWS Image Builder - Slack uses AWS Image Builder instead of Packer to create `slack-zero`. - Image Builder provides lifecycle policies that automatically remove old AMIs and reduce storage costs. - Each new image publishes its latest AMI identifier through an AWS Systems Manager parameter, helping downstream systems discover the current base image. Shipyard’s central recommendation is to treat EC2 infrastructure as versioned, replaceable artifacts rather than mutable machines. Combining trusted base images, automated provisioning, metric-driven deployment controls, fleet-wide inventory, and scheduled instance replacement gives Slack safer and more predictable EC2 operations at scale.
Read original(opens in new tab)