Kubernetes - Slack | Techlist.io

slackMay 5, 20263 min readCurated summary

From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

Slack had more than 700 SSH-based operators running critical EMR workloads, creating security risks, operational failures, and barriers to infrastructure modernization. The company replaced these connections with REST-based job submission across eight data regions without downtime. YARN Distributed Shell was the key enabler for migrating arbitrary command-line jobs that lacked dedicated REST APIs. ## How Slack’s SSH Architecture Developed - Airflow originally connected directly to EMR master nodes using `SSHOperator`. - Over time, teams created more than 700 SSH-based jobs for: - Spark and MapReduce workloads - AWS CLI commands - Custom Python scripts - Data-transfer operations such as `hadoop distcp` - The approach was simple but tightly coupled orchestration workers to production clusters. ## Security and Operational Costs of SSH - Direct SSH access expanded the attack surface. - SSH keys had to be distributed and rotated across orchestration workers. - Auditing required correlating activity across multiple systems. - Permissions became complicated, often involving custom security groups and configurations. - Jobs ran on EMR master nodes, causing resource contention. - Restarted Kubernetes pods could break SSH connections. - Long-running processes could become orphaned “zombie” jobs. - Connection failures made job success or failure difficult to determine. - SSH dependencies blocked Spark-on-Kubernetes, EMR on EKS, AWS child-account migration, and better observability. - Slack’s search-indexing pipeline was especially sensitive because it processed terabytes of data daily and supported search for millions of users. ## REST-Based Job Submission - SSH creates a stateful connection whose failure can leave job status ambiguous. - REST APIs provide a durable, server-managed lifecycle: - `POST` submits a job and returns an ID. - `GET` retrieves its status. - `DELETE` cancels it cleanly. - Clients can crash or restart without terminating the underlying job. - Existing systems such as YARN, Trino, and Snowflake use this model. - YARN provides REST submission for Hadoop, Spark, Hive, and MapReduce workloads, but not arbitrary shell commands. ## YARN Distributed Shell - Spark and Hive already had REST-compatible options through Livy and HiveServer2. - The difficult cases were MapReduce and more than 300 CLI-based jobs. - Slack considered custom wrapper services, Ansible or Salt, and creating a new YARN job type. - These alternatives added complexity, security work, or long-term maintenance. - YARN Distributed Shell—implemented through `ApplicationMaster`—could execute arbitrary scripts inside YARN containers. - It used existing YARN APIs and authentication mechanisms, avoiding a custom security layer. ## The Distributed Shell Workflow - Upload a command script to S3, such as an `aws s3 sync` operation. - Submit a YARN application specifying: - The Distributed Shell application master - The S3 script location - Script metadata such as length and timestamp - YARN then: - Allocates a resource-managed container - Downloads and executes the script - Enforces memory and vCore limits - Provides isolation, retries, cancellation, and centralized logging By using REST submission and YARN Distributed Shell, Slack could remove SSH from its EMR data pipelines while preserving support for both standard data-processing jobs and arbitrary command-line workloads.

Read original

Kubernetes Rest Api Apache Spark Apache Airflow+3

slackOct 23, 20253 min readCurated summary

Advancing Our Chef Infrastructure: Safety Without Disruption

Slack chose to improve its existing Chef and EC2 infrastructure rather than migrate to Policyfiles, avoiding disruptive cookbook and role changes. The central strategy is to divide production into six Availability Zone–based Chef environments, limiting deployment blast radius while preserving existing workflows. A canary environment and staggered release train provide earlier detection of configuration problems and safer fleet-wide rollouts. ## Why Slack Avoided Policyfiles - Policyfiles could have improved long-term safety by replacing roles and environments. - Migrating dozens of teams and their cookbooks would have required substantial effort. - Slack concluded that the short-term disruption and migration risk outweighed the benefits. - Instead, the team extended its existing EC2 framework without requiring cookbook or role changes. ## Splitting Production Chef Environments - Previously, all production nodes used one shared Chef environment. - Cron-triggered Chef runs were staggered across Availability Zones to prevent simultaneous fleet-wide changes. - This reduced the impact of bad changes on existing nodes, but newly provisioned instances immediately consumed the latest version from the shared environment. - During large scale-out events, a broken configuration could therefore spread rapidly to many new nodes. - Slack split production into six environments: `prod-1` through `prod-6`. - Service teams still launch instances as `prod`; internally, nodes are assigned to a numbered environment based on their Availability Zone. - Updates to one environment now affect only the nodes mapped to that environment. ## Extending Poptart Bootstrap - Slack’s base AMIs include `Poptart Bootstrap`, which runs through `cloud-init` during instance startup. - It creates the node’s Chef object, configures DNS, and posts success or failure notifications to Slack. - Slack extended it to inspect the node’s AZ ID and select the appropriate numbered production environment. - This automatically distributes new nodes across isolated Chef environments without requiring service teams to change their provisioning process. ## Canary Deployments and the Release Train - Cookbook changes are promoted: - To sandbox at the top of the hour - To development environments through a Kubernetes cron job - To production beginning at 30 minutes past the hour - `prod-1` acts as the canary production environment. - It receives the latest changes hourly when new cookbook artifacts exist. - This tests changes in real production conditions soon after they are created. - `prod-2` through `prod-6` follow a release train. - A version advances gradually through the production environments. - The next rollout begins only after the previous version has reached `prod-6`. - This sequencing limits the number of affected nodes and makes regressions easier to identify. ## Why `prod-1` Updates Frequently - If the canary waited until a version had passed through every production environment, it would test artifacts containing larger batches of accumulated changes. - Updating `prod-1` frequently keeps the feedback loop close to the originating change. - The remaining production environments provide progressively broader validation after the canary stage. - For example, a new artifact can move from sandbox and dev to `prod-1`, then advance through `prod-2` to `prod-6` while newer artifacts continue entering the canary path. Slack’s approach preserves its existing Chef ecosystem while adding isolation, automated environment assignment, and staged promotion. The result is a safer deployment process that reduces blast radius and catches production issues earlier without forcing widespread application changes.

Read original

Kubernetes Aws Ec2 Chef Canary Deployment+3

slackOct 7, 20254 min readCurated summary

Deploy Safety: Reducing customer impact from change

Slack’s Deploy Safety Program reduced customer-impact hours by 90% from its peak by focusing on safer change across all deployment systems, rather than optimizing individual services in isolation. The program combines measurable reliability goals, automated detection and rollback, blast-radius reduction, and cultural change. Its core lesson is to invest broadly, measure results, and expand approaches that demonstrably reduce customer impact without slowing development. ## Defining the Reliability Problem - Slack became increasingly mission-critical, raising customer expectations for reliability. - In analysis of customer-facing incidents, 73% were triggered by Slack-induced change, especially code deployments. - Incidents occurred across hundreds of services and multiple deployment systems, producing inconsistent levels of customer impact. - Customers reported that interruptions became significantly more disruptive after roughly 10 minutes. - Earlier reliability efforts often focused on individual deployment systems or services, leading to manual processes that slowed innovation and reduced engineering morale. ## North Star Goals and the Deploy Safety Manifesto The initial program goals applied to Slack’s highest-importance services: - Detect and automatically remediate deployment problems within 10 minutes. - Detect and manually remediate problems within 20 minutes. - Identify problematic deployments before they reach 10% of the fleet. - Preserve Slack’s engineering and development velocity. These goals later evolved into a Deploy Safety Manifesto covering all deployment systems and processes, including: - Automated safety improvements. - Deployment guardrails. - Changes to engineering practices and safety culture. ## Measuring Customer Impact Slack defined its primary program metric as: - **Hours of customer impact from high-severity and selected medium-severity change-triggered incidents.** The metric is an imperfect proxy for customer sentiment because: - Incident severity reflects current or anticipated impact, not always the final customer experience. - Medium-severity incidents require additional filtering to determine whether their actual impact is relevant. - It can be difficult to connect an individual engineering project directly to changes in customer sentiment. Slack evaluates the metric using four principles: - Measure outcomes rather than activity. - Distinguish real measurements from proxy metrics. - Apply subjective criteria consistently. - Regularly validate the metric against feedback from leaders who speak directly with customers. ## Choosing Where to Invest At the beginning of the program, Slack did not know which projects would produce the greatest benefit or when results would appear. Incident data is inherently delayed, while customers are experiencing reliability problems immediately. The investment strategy therefore emphasized: - Broad initial investment and a bias toward action. - Addressing known customer pain first. - Expanding successful projects and repeatable patterns. - Reducing investment in areas with limited impact. - Maintaining a flexible roadmap that could change as results emerged. Projects were prioritized according to whether they could: - Detect deployment problems earlier. - Improve automatic remediation time. - Improve manual rollback and remediation time. - Reduce severity by limiting deployment blast radius. ## Improving Webapp Backend Deployments Slack identified Webapp backend deployments as the largest source of change-triggered incidents and iteratively improved their safety: - Built automated metric monitoring. - Added automatic alerts and manual rollback procedures to validate alignment with customer impact. - Introduced automatic deployments and rollback. - Demonstrated that repeated automatic rollbacks could keep customer impact below 10 minutes. - Expanded monitoring to additional metrics. - Optimized manual rollback processes. - Added manual rollback capability for the frontend. - Began consolidating deployment practices through a centralized orchestration system inspired by ReleaseBot and AWS Pipelines. - Extended metrics-based deployment and automatic remediation beyond Bedrock and Kubernetes. These improvements made Webapp backend, frontend, and some infrastructure deployments significantly safer, with continued quarter-over-quarter improvement. ## Iterative Expansion Slack applied the same pattern across other areas: - Try an intervention. - Measure whether customer impact improves. - Invest further when the approach succeeds. - Reuse successful patterns in other systems. - Reduce or redirect investment when results are limited. The article notes that some efforts, such as faster mobile-app issue detection, were successful, while others produced less noticeable improvements. Slack’s experience suggests that deployment safety works best as an ongoing program: establish measurable customer-focused goals, automate detection and recovery, control blast radius, and continuously replicate proven practices without sacrificing delivery speed.

Read original

Kubernetes Incident Management Aws Pipelines Deployment Orchestration+2