gitlabMay 18, 2026

Fix bugs with Codex and GitLab (opens in new tab)

Codex accelerates coding in the terminal, but producing a fix is only one part of shipping software. GitLab supplies the surrounding lifecycle: issues, merge requests, CI/CD, security scanning, code review, and human approval. The tutorial demonstrates this progression through a Rust WebSocket bug, first with local Codex, then with GitLab MCP for issue context, and finally with Codex as an external agent in GitLab Duo Agent Platform. ## Prerequisites and Project Setup - Configure Codex in the terminal, Rust/Cargo, and access to a GitLab project. - Import and clone the Tanuki IoT Platform project, then launch Codex from its repository root. - The tutorial focuses on `backend/`, where: - Sensors submit readings through a REST API. - Dashboards receive live readings through WebSocket streams. - `AGENTS.md` provides Codex with repository structure, toolchain instructions, build commands, and quality expectations. ## Reproducing the WebSocket Filtering Bug - Start the Rust metrics backend on port `9090`: ```bash PORT=9090 cargo run --manifest-path backend/rust-metrics-store/Cargo.toml ``` - Connect to a filtered WebSocket stream: ```bash websocat 'ws://localhost:9090/ws?sensor=arduino-iot-collector&metric=temperature_celsius' ``` - Submit both temperature and humidity readings for the same sensor through the REST API. - The stream incorrectly returns both metrics instead of only `temperature_celsius`, proving that the WebSocket handler does not apply the metric filter. ## Fixing the Bug with Codex - Give Codex a focused request to add metric filtering to `/ws`. - Codex examines the Rust source and identifies that the endpoint already supports `sensor` filtering but lacks an optional `metric` condition. - It updates the handler, adds tests, and keeps documentation aligned with the implementation. - Codex runs formatting, tests, and builds before creating a branch, committing, and pushing the change. - Once the merge request is created, GitLab handles: - CI/CD pipelines - Security scanning - GitLab Duo Code Review - A follow-up WebSocket test confirms that supplying both sensor and metric now returns only the requested metric. ## Adding GitLab Context with MCP - Local Codex can inspect repository files, but it cannot automatically see GitLab issues, requirements, implementation notes, merge-request discussions, or pipeline status. - The GitLab MCP server connects Codex to that development lifecycle context. - Codex can retrieve the existing issue directly instead of requiring the developer to copy its contents into the prompt. - The issue acts as the shared source of truth and includes: - The bug description - Functional behavior requirements - Non-functional requirements - Required tests - Updates to `README.md` and `AGENTS.md` - Implementation notes - This helps Codex produce a fix that satisfies the agreed requirements rather than merely addressing the symptom visible in the local code. ## Using Codex as an External GitLab Agent - The tutorial’s third workflow uses Codex inside GitLab Duo Agent Platform as an external agent. - This allows the agent to participate after the merge request is open, particularly when addressing review feedback. - GitLab remains the system coordinating issues, merge requests, pipelines, reviews, and deployment, while Codex contributes its terminal-oriented coding capabilities. - The overall workflow moves from bug report to implementation, automated validation, review feedback, revisions, and an informed human decision to ship. ## Practical Conclusion Use Codex for fast, repository-local implementation, but connect it to GitLab through MCP or Duo Agent Platform when requirements and review context matter. The strongest workflow combines Codex’s coding speed with GitLab’s issue-aware, automated, and human-governed delivery lifecycle.

websocketsai-agentgitlabci-cd+3

cloudflareMay 13, 2026

Browser Run: now running on Cloudflare Containers, it’s faster and more scalable (opens in new tab)

Browser Run was rebuilt on Cloudflare Containers to improve speed, reliability, and scale. The migration increased capacity to 60 browser launches per minute and 120 concurrent browsers—four times the previous limit—while cutting Quick Action response times by more than 50%. The main architectural changes were global Container deployment, regional pools of pre-warmed browsers, and replacing eventually consistent KV state with transactional D1 and batched Queue updates. ## Browser Run’s Role - Provides programmatic access to headless browsers on Cloudflare’s global network. - Supports: - End-to-end testing - Suspicious URL investigation - PDF rendering - Screenshots and content extraction - Web interaction for AI agents - The goal is to offer secure, responsible browser automation at massive scale. ## Why the Previous Infrastructure Was Limiting - Browser Run originally shared infrastructure with Browser Isolation (BISO). - BISO’s larger container images caused slower startup and development cycles. - Browser Run lacked optimal global distribution, affecting latency and resilience. - BISO’s long-running sessions conflicted with Browser Run’s short, bursty workloads. - These differences created scaling and availability bottlenecks. ## Gradual Migration to Containers - A Worker initially routed a small number of requests to Container-based browsers while others continued using BISO. - This dual-running setup allowed the team to: - Compare performance - Find implementation bugs - Validate stability - Rollout stages included: - Quick Actions - Free-account Workers browser binding connections - Pay-as-you-go accounts - Contract customers - Customers did not need to change code or redeploy Workers. ## Regional Pools for Lower Latency - Durable Object-enabled Containers can create the Durable Object near the request while starting the Container elsewhere. - This is acceptable for one-off commands but inefficient for WebSocket workflows involving many messages. - The team introduced regional pools of pre-warmed, Durable Object-backed browsers. - Requests are assigned to a nearby Durable Object–Container pair, reducing latency between: - The user and Durable Object - The Durable Object and browser Container - The design requires global browser-state observability so capacity can be allocated and reassigned as demand changes. ## Replacing KV with D1 and Queues - Workers KV was initially used to track browser availability. - Its eventual consistency and cache TTL—around 30 seconds or longer—caused race conditions: - A browser could appear available when another request had already claimed it. - Delayed state updates led to over-allocation and limited responsiveness to traffic spikes. - Browser state was moved to D1, whose SQLite transactions provide atomic assignment. - A browser is exclusively assigned to one user, preventing simultaneous claims through transactional updates. Example acquisition logic updates selected candidates to `picked` and returns their data in one operation: ```sql WITH candidate_pool AS (...) UPDATE containers SET status = 'picked' WHERE sessionId IN ( SELECT sessionId FROM candidate_pool ORDER BY RANDOM() LIMIT ?5 ) RETURNING data; ``` ## Batching State Updates - D1 shards are maintained by location. - Thousands of containers report their state every five seconds, which could overload the database if each update were written individually. - Queue-based batching groups 100 updates into a single write. - This increases theoretical capacity from roughly 5,000 containers per location to as many as 500,000. - The team reports a P95 batch-write latency of 0.1 ms. - Queue consumers use: - Maximum batch size: 100 - Maximum batch timeout: 1 second - Maximum retries: 1 The migration is live, requires no customer changes, and gives Browser Run more room to handle demand from AI agents and other high-volume browser automation workloads.

websocketscloudflare-workersdurable-objectssqlite+4

discordApr 29, 2026

You’ve Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage (opens in new tab)

Discord’s March 25, 2026 voice outage began when a Kubernetes configuration change abruptly terminated 17% of session processes. The resulting reconnection storm propagated through Discord’s realtime systems and overloaded voice-routing infrastructure, preventing many users from starting or joining calls. The incident exposed how failures in one distributed subsystem can create cascading load several services away. ## The Infrastructure Background - Discord is migrating stateful Elixir services to Kubernetes. - Each host runs thousands of in-memory processes for guilds, presence, messaging, and calls. - Deployments normally wait for a server’s entity count to reach zero before shutting it down, allowing processes to hand off their state safely. - The sessions service maintains one process for every connected device and carries websocket traffic, messages, presence updates, and other realtime events. - To reduce weekend CPU utilization, Discord planned to increase pod CPU and memory while proportionally reducing the number of pods. ## The Session Loss - The resource change was deployed to the first availability zone at 12:13 PDT. - Kubernetes terminated half of that zone’s pods because of the reduced replica count. - A safety check delayed process handoffs until other events completed, but the Kubernetes termination grace period expired first. - Because the service operated across three balanced zones, approximately 17% of Discord’s sessions stopped without a graceful handoff. - The outage lasted from 12:13 to 15:30 PDT, with users commonly seeing “Awaiting Endpoint.” ## How Elixir Monitoring Amplified the Failure - Discord relies heavily on Elixir `GenServer` processes, which process one mailbox message at a time. - Process monitors notify dependent processes whenever a monitored process exits. - The sudden loss of sessions therefore generated a large number of `{:DOWN, …}` notifications throughout the realtime infrastructure. - Guild and other processes stopped attempting to deliver updates to disconnected users, while the gateway began driving those users to reconnect. ## Reconnecting Users - The gateway handles websocket ingress and egress, creating sessions and maintaining client connections. - Session disconnections are normally expected and recoverable, whether caused by hardware, network problems, software bugs, or temporary connectivity loss. - When a session disappears, the gateway immediately instructs the client to reconnect. - It optimistically tries to resume the session through a gateway instance in the same zone, but the mass failure created a much larger reconnection surge than the system was designed to absorb. The incident demonstrates that reducing pod count can be dangerous in stateful distributed systems: an apparently routine capacity adjustment can cause abrupt process loss, trigger widespread retries, and overload unrelated downstream services. Changes to stateful workloads should be evaluated not only for steady-state resource usage but also for graceful shutdown behavior and synchronized failure scenarios.

websocketsk8sdistributed-systemselixir+2

figmaOct 19, 2022

Making multiplayer more reliable | Figma Blog (opens in new tab)

Figma improved multiplayer reliability by adding a durable write-ahead journal alongside its existing checkpoint system. Instead of relying on full-file snapshots every 30–60 seconds, Figma now records incremental changes frequently, allowing crashed servers to recover nearly to the latest state and reducing deployment-related database spikes. The goal was to reduce potential data loss from up to 60 seconds to less than one second. ## How Figma’s Multiplayer System Worked - Browsers connect to Figma’s multiplayer service over WebSockets. - The service authoritatively handles: - Validation - Ordering - Conflict resolution - Broadcasting updates to connected clients - File state is held in memory for speed. - Every 30–60 seconds, the entire file is: - Encoded into a binary format - Compressed - Uploaded to Amazon S3 as a checkpoint ## Problems with Checkpoint-Only Persistence - A multiplayer crash could lose up to 60 seconds of server-side work. - Checkpoints become increasingly expensive as files grow in size and complexity. - Redeployments caused large write spikes: - All in-memory files had to be closed. - Each file needed a final checkpoint. - The resulting burst increased database load. ## Introducing the Journal - Figma added a durable transaction log, or journal, backed by DynamoDB. - Each accepted change receives an incrementing sequence number. - Checkpoints store the latest sequence number they include. - During recovery: - Multiplayer loads the latest checkpoint. - It queries the journal for entries with higher sequence numbers. - It replays those incremental changes to reconstruct the latest file state. - Journal entries are much smaller than full-file checkpoints, since they contain only user edits. - Figma writes journal data roughly every 0.5 seconds rather than waiting 60 seconds between full snapshots. - The target was less than one second of data loss in rare failure scenarios. ## Smoother Deployments - During deployment, Figma can close connections and wait for unsaved changes to reach the journal. - The 99th percentile persistence time is under one second. - Since journal writes happen continuously during normal operation, deployments no longer create a sudden checkpoint-writing surge. - Database write load is therefore steadier and more predictable. ## Datastore and Batching Decisions - Figma selected DynamoDB because the journal requires a horizontally scalable datastore with high write capacity. - Postgres was considered but rejected because the anticipated write volume exceeded Figma’s current horizontal-scaling approach for Postgres. - Clients send updates at approximately 30 frames per second, or every 33 milliseconds. - The journal does not need that granularity, so multiple changes can be batched before being persisted, improving performance. Figma’s approach combines inexpensive, frequent incremental journal writes with larger periodic checkpoints. This provides faster recovery, minimizes data loss, and avoids deployment-related load spikes while retaining checkpoints for efficient long-term storage and features such as version history.

websocketspostgresqlreal-time-collaborationdynamodb+2

figmaOct 13, 2021

LiveGraph: real-time data fetching at Figma | Figma Blog (opens in new tab)

LiveGraph is Figma’s in-house real-time data-fetching layer built on PostgreSQL. It lets frontend developers declare live data views with GraphQL-like queries, while LiveGraph reads PostgreSQL’s replication stream to deliver updates within milliseconds. Figma built it to replace fragile, manually maintained client events and to support real-time subscriptions at large scale without relying on polling or a new database technology. ## Problems with Figma’s Earlier Real-Time Architecture - React clients initially loaded large data sets through Ruby HTTP endpoints and stored them in Redux. - Backend code manually emitted events whenever database records changed. - Frontends subscribed over WebSockets and applied those events to client state. - As data volumes grew, Figma split requests into incremental loads, making data ownership and availability harder to reason about. - Complex changes—such as permission updates affecting many resources—were difficult to represent with individual events. - Events could arrive out of order or fail to correspond reliably with database writes, causing client state to diverge from server state. ## Why Figma Chose Live Queries - Figma wanted developers to define data subscriptions declaratively rather than manually coordinate fetches and update events. - GraphQL provided a natural interface for describing the relevant portion of the object graph. - LiveGraph uses “live queries,” which keep query results synchronized, rather than GraphQL subscriptions in the narrower sense of consuming event streams. - The system is a query and data-fetching layer over existing PostgreSQL infrastructure, not a replacement persistence layer. ## In-House System Versus Existing Tools - Figma’s multiplayer service handles collaborative writes and conflict resolution within individual files, whereas LiveGraph focuses on reading application data. - Systems such as Hasura, Prisma, and PostGraphile offered GraphQL subscription features but were not designed primarily for Figma’s scale of concurrent live subscriptions. - Polling was rejected because it increases database load and requires developers to choose polling intervals for each query. - Figma’s collaborative product made real-time data central enough to justify building and operating a specialized internal system. - The company did not claim LiveGraph was universally superior; its value came from matching Figma’s specific scale and requirements. ## Replication-Stream-Based Updates - LiveGraph executes queries directly against PostgreSQL. - It tails the database replication log to detect changes instead of repeatedly polling tables. - Reading the replication stream enables update latency measured in milliseconds. - Because the system must process the complete volume of database changes, its architecture needs to distribute updates across machines and database shards. - This approach separates the complexity of detecting database changes from product code, allowing frontend engineers to work with declarative JSON data views. ## Frontend API - Product developers send GraphQL-like queries and receive results as JSON trees. - A schema defines server-side entities and relationships, while views expose queryable subsets of that graph. - The frontend can therefore request the data it needs and rely on LiveGraph to keep the result synchronized as the underlying PostgreSQL data changes. LiveGraph’s central recommendation is architectural: derive live client views from the database’s authoritative change stream rather than maintaining a parallel network of hand-written events. For organizations with similar scale and real-time requirements, this can improve consistency and simplify product development, though Figma’s in-house approach was justified by its unusually collaborative workload.

websocketsreactpostgresqltypescript+4

figmaOct 15, 2019

How Figma’s multiplayer technology works | Figma Blog (opens in new tab)

Figma built a custom multiplayer system because traditional operational transformation (OT) was too complex for its document-editing needs. Its client/server architecture synchronizes document changes over WebSockets, supports offline editing, and separates document collaboration from other data such as comments and users. The system began as a prototype that enabled rapid experimentation before being integrated into production. ## Why Figma Built Its Own Multiplayer System - In 2015, no major design tool offered real-time collaborative editing. - Figma avoided OT, the algorithm used by tools such as Google Docs, because it considered OT unnecessarily complex for its problem space. - The custom approach was designed to be simpler and faster to implement. - Multiplayer eliminated the need to export, email, or manually synchronize design files. - It also allowed non-designers—such as copywriters and developers—to participate or view work without interrupting the designer. ## Figma’s Client/Server Architecture - Figma clients are web pages connected to a server cluster through WebSockets. - Each multiplayer document runs in a separate server process, with all editors connected to that process. - When a document opens, the client downloads an initial copy of the file. - Subsequent changes are synchronized in both directions over the WebSocket connection. - Server performance and scaling were important considerations, later addressed in part through the use of Rust. ## Offline Editing and Reconnection - Clients can continue editing while offline for an arbitrary period. - When reconnecting, the client: - Downloads a fresh version of the document. - Reapplies its locally stored offline edits to that latest state. - Resumes synchronization through a new WebSocket connection. - This keeps connection and reconnection logic relatively simple by concentrating multiplayer complexity on already-connected clients. ## Separate Systems for Different Data - Figma’s multiplayer system is used only for syncing document changes. - Comments, users, teams, projects, and similar information are stored in Postgres. - That data is synchronized through a separate system because it has different requirements involving: - Performance - Offline availability - Security ## Prototyping Before Production - Figma first created a standalone browser-based prototype rather than experimenting directly in the production codebase. - The prototype simulated three clients connected to a server and visualized the complete system state. - Engineers could test: - Offline clients - Bandwidth-limited connections - Different collaborative algorithms - Alternative data structures - Once the design was validated, the ideas were transferred into the main codebase. Figma’s experience demonstrates that collaborative systems do not always require the most established algorithm. A focused, custom protocol—validated through fast prototyping—can provide a simpler solution when its data model and product requirements differ from tools like document editors.

websocketspostgresqlreal-time-collaborationrust+2