Blue Green Deployment

cloudflareJun 24, 20263 min readCurated summary

Unlocking the Cloudflare app ecosystem with OAuth for all

Cloudflare opened self-managed OAuth to all customers so developers can build SaaS integrations, internal platforms, CI/CD workflows, and agentic tools without relying on difficult-to-manage API tokens. The expansion required improvements to permissions, consent, revocation, and phishing protections, as well as a major upgrade to the Hydra-based OAuth engine. Cloudflare used staged migrations, custom database changes, token-handling safeguards, and queued revocations to minimize disruption and preserve users’ security controls. ## Why Cloudflare Expanded OAuth Access - Previously, third-party OAuth integrations were limited to manually approved partners. - Other developers had to use API tokens, which are less convenient and poorly suited to delegated access. - Self-managed OAuth lets customers: - Request narrowly scoped permissions. - Give users clearer consent controls. - Revoke application access from the dashboard. - Build integrations and agentic tools using standard OAuth flows. - Cloudflare improved consent screens to identify the requesting application and its permissions, while making application ownership more visible to reduce phishing risks. ## Planning the Hydra Upgrade - Cloudflare used Hydra, an open-source OAuth engine, but its older deployment could not support the platform’s growing scale and new use cases. - The upgrade was split into two stages: - First, move to the latest 1.X release. - Then, perform the larger 2.X migration. - The 1.X database migrations created operational risks: - Standard index creation could take exclusive locks on critical tables. - Schema changes added columns and moved data between tables. - Hydra’s SDK used `SELECT *`, creating deserialization problems after schema changes. - Cloudflare rewrote migrations to use `CREATE INDEX CONCURRENTLY` and built a custom Hydra version that selected explicit columns. ## Designing a Blue-Green Migration - An in-place 2.X upgrade was rejected because of the volume of schema changes. - A blue-green deployment was chosen, but the migration would take several hours. - Disabling writes would prevent new authorizations and revocations, leaving users unable to manage application access during the upgrade. - Instead, Cloudflare kept writes enabled while reducing the amount of data that could be lost during the cutover: - Token expiry times were temporarily extended to multiple hours, reducing refresh-token writes. - Revocation events were written to Cloudflare Queues. - After switching to the green database, queued revocations could be replayed. - Preserving revocations was essential to prevent applications that users had disabled from regaining access. ## Lessons from the 1.X Upgrade - The custom migrations completed faster than expected without user impact. - A hard cutover was necessary because the old Hydra version could not read tokens created by the new version. - The new version introduced stricter refresh-token invalidation: - Reusing a refresh token invalidated the entire access and refresh-token chain. - This caused problems for high-volume clients such as Wrangler and MCP clients. - Cloudflare added refresh-token coalescing in the Worker routing layer: - Briefly caching requests allowed retries to be served without triggering invalidation. - Hydra 2.X provides a configurable refresh-token grace period, offering a more direct solution for safe retries. ## Executing the 2.X Upgrade - Cloudflare prepared a blue-green migration to avoid several hours of customer-facing downtime. - The strategy depended on reducing token writes, recording all revocations externally, switching databases, and replaying queued events afterward. - The provided article ends while beginning the detailed discussion of the 2.X execution. Cloudflare’s approach demonstrates that opening a security-sensitive platform to broad OAuth usage requires more than exposing an authorization endpoint. Safe adoption depends on explicit permissions, transparent consent, reliable revocation, backward-compatible token behavior, and migration plans that protect users even during infrastructure cutovers.

Read original

Blue Green Deployment Cloudflare Workers Oauth Wrangler+3

airbnbSep 24, 20253 min readCurated summary

Building a Next-Generation Key-Value Store at Airbnb

Airbnb rebuilt Mussel, its key-value store for derived data, from a complex EC2-based system into a cloud-native NewSQL platform. Mussel v2 combines bulk ingestion, streaming writes, low-latency reads, flexible consistency, and automated operations while supporting more than 100 existing use cases. A gradual, reversible blue/green migration moved production workloads without data loss or customer-visible downtime. ## Why Airbnb Rebuilt Mussel - New use cases—including real-time fraud detection, personalization, and dynamic pricing—required both streaming updates and large-scale bulk ingestion. - Mussel v1 had become difficult to operate and scale: - Node changes required multi-step Chef scripts on EC2. - Static hash partitioning created hotspots and latency spikes. - Consistency options were limited. - Resource consumption and costs were difficult to track. - Mussel v2 provides Kubernetes-based automation, dynamic range sharding, configurable consistency, namespace tenancy, quotas, and usage dashboards. ## Mussel v2 Architecture ### Stateless Dispatcher - A horizontally scalable Kubernetes service translates client requests into backend queries and mutations. - It supports: - Dual writes and shadow reads during migration - Retries, rate limiting, and dynamic throttling - Service-mesh security and discovery - Point lookups, range queries, prefix queries, and low-latency stale reads - Each dataname maps to a logical table, simplifying access patterns. ### Kafka-Based Write Pipeline - Writes are first persisted to Kafka for durability. - The Replayer and Write Dispatcher apply them to the backend in order. - Kafka absorbs traffic bursts and supports consistency, migrations, bootstrapping, and upgrades. - Airbnb plans to eventually rely more directly on the distributed database for ingestion and replication to reduce latency and operational complexity. ### Bulk Loading - Mussel retains support for both: - **Merge** jobs, which add data to existing tables - **Replace** jobs, which swap in a new dataset - Existing Airflow onboarding workflows transform warehouse data into a standard format and upload it to S3. - A stateless controller coordinates ingestion, while Kubernetes StatefulSet workers load data in parallel. - Deduplication, delta merges, and insert-on-duplicate-key-ignore improve throughput and reduce unnecessary writes. ## Scalable Data Expiration - Mussel v1 depended on storage-engine compaction for TTL expiration, which became inefficient at scale. - V2 uses a topology-aware expiration service: - Namespaces are divided into range-based subtasks. - Multiple workers scan and delete expired records concurrently. - Scheduling limits interference with live queries. - Max-version enforcement and targeted deletes help manage write-heavy tables. - The result is faster, more visible, and more scalable retention management. ## Blue/Green Migration - The migration had to handle massive datasets, thousands of tables, and mission-critical traffic with zero data loss and no availability impact. - Because v1 lacked table-level snapshots and CDC, Airbnb built a custom migration pipeline. - Tables were selected and migrated individually according to usage and risk. ### Migration Stages - **Blue:** All production traffic continued serving from v1. - **Shadowing:** Bootstrapped v2 tables processed parallel reads and writes, but v1 still served responses. - **Reverse:** V2 served live traffic while v1 remained available as a fallback. - **Cutover:** After validation, traffic was permanently moved to v2 one dataname at a time. - Automatic circuit breakers and fallback logic enabled rapid rollback if v2 showed errors or replication lag. - Kafka’s replication stream maintained eventual consistency between the two systems throughout the transition. ## Practical Takeaway Mussel v2 demonstrates that large datastore rearchitectures can be made safe through incremental migration, durable event logs, shadow traffic, and reversible per-table cutovers. The key recommendation is to combine a more scalable backend with strong operational automation and migration tooling, rather than attempting a single disruptive replacement.

Read original

Blue Green Deployment Kubernetes Apache Kafka Database Migration+4