Safeguarding Dynamic Configuration Changes at Scale
Airbnb’s Sitar platform is designed to make runtime configuration changes as safe and reliable as code deployments. It combines Git-based reviews, automated validation, staged rollouts, observability, and fast rollback with a highly available distribution system. Separating decision-making from config delivery, while using local caches, lets teams change behavior quickly without unnecessarily increasing outage risk. ## Requirements for a Modern Configuration Platform - Provides an end-to-end workflow for defining, reviewing, testing, and deploying configuration. - Treats configuration like code: - Versioned and reviewable - Auditable - Governed by ownership and access controls - Supports isolated local and canary testing before production rollout. - Accommodates multiple tenants with different: - Deployment triggers - Guardrails - Rollout strategies - Enables incident responders to make emergency changes while preserving auditability and visibility into who changed what, when, and which users or services were affected. ## Sitar’s Architecture Sitar consists of four major layers: - **Developer-facing layer:** Configs are usually managed through GitHub pull requests. The Sitar portal supports exceptions and administrative operations, including emergency deployments. - **Control plane:** Validates schemas, enforces ownership and authorization, selects rollout targets, manages progressive deployment, and supports rollback and targeted testing. - **Data plane:** Stores config values and versions as the source of truth, then distributes updates reliably and efficiently. - **Agents and client libraries:** An agent sidecar fetches subscribed configs and maintains a local cache. In-process client libraries read from that cache and expose values to application code, with optional fallbacks. A typical change moves from a Git workflow through validation and rollout decisions, into the data plane, and finally to sidecars and application clients. ## Git-Based Configuration Management - GitHub is the default interface because it integrates with Airbnb’s existing CI/CD systems and review practices. - Teams can use pull requests, mandatory reviewers, approval flows, and complete change history. - Related configs are grouped into tenants with defined owners, custom tests, and dedicated continuous-delivery pipelines. - The Sitar portal remains available for teams that need a UI or for urgent changes that must bypass the standard CI/CD process. ## Progressive Rollouts and Rollbacks - CI first checks schema correctness, expected structure, types, and other automated requirements. - Config changes require review and approval before deployment. - After merging, changes roll out gradually: - Start with a limited environment, AWS zone, or percentage of Kubernetes pods. - Evaluate the change at each stage. - Expand only when results are healthy. - Authors and stakeholders are notified when regressions are detected, and bad changes can be rolled back quickly. - Limiting the initial scope reduces the blast radius of configuration errors. ## Separating Control and Data Planes - The control plane decides whether and how a change should be deployed. - The data plane stores and distributes the resulting configuration. - This separation allows rollout policies and authorization logic to evolve independently from storage and delivery infrastructure. - Changes to one layer are less likely to disrupt the other. ## Local Caching and Resilient Clients - Each service runs an agent sidecar alongside its application container. - The sidecar periodically retrieves subscribed configs and persists them locally. - Client libraries read configuration from the local cache for fast, in-process access. - If the configuration backend becomes unavailable or degraded, services can continue using the last known good values. ## Practical Takeaway A reliable dynamic configuration system should combine code-like governance with runtime flexibility. Git reviews, validation, staged deployment, strong observability, plane separation, and local caching allow teams to respond quickly while keeping configuration failures contained and reversible.
Read original(opens in new tab)