gitlab·May 28, 2026·3 min readCurated summary

GitLab Patch Release: 19.0.1, 18.11.4, 18.10.7 | GitLab Docs

GitLab released patch versions 19.0.1, 18.11.4, and 18.10.7 on May 27, 2026, addressing multiple security vulnerabilities and bug fixes across CE and EE. The most serious issue, rated CVSS 8.2, affected Duo AI workflow identity resolution, while other fixes addressed authorization, data exposure, denial-of-service, and CI access issues. GitLab strongly recommends that self-managed installations upgrade immediately. ## Release Scope and Upgrade Guidance - Applies to GitLab Community Edition and Enterprise Edition. - GitLab.com is already patched; GitLab Dedicated customers do not need to act. - The releases include fixes for supported versions and should be applied as soon as possible. - GitLab publishes vulnerability details publicly 30 days after the release containing the fix. - Patch releases are generally issued on the second and fourth Wednesdays, with critical ad-hoc releases possible. ## Security Fixes - **CVE-2026-4868 — Duo AI workflow runners, EE** - CVSS 8.2. - Under certain conditions, an authenticated user could cause workflows to run under another user’s identity. - Affected versions were EE releases before 18.10.7, 18.11.4, and 19.0.1. - **CVE-2026-1402 — Wiki denial of service, CE/EE** - CVSS 6.5. - Insufficient validation could allow an authenticated user to trigger a denial-of-service condition. - **CVE-2026-6713 — GraphQL WorkItem API, CE/EE** - CVSS 5.3. - Incorrect authorization checks could allow unauthorized enumeration of private projects. - **CVE-2026-5296 — Duo Workflows API, EE** - CVSS 4.3. - Developer-level users could bypass restrictions on group-level foundational flows in certain configurations. - **CVE-2026-2601 — Operations, EE** - CVSS 4.3. - Developers could potentially access sensitive deployment data because of missing authorization checks. - **CVE-2026-8716 — Pipelines, CE/EE** - CVSS 4.3. - Incorrect name resolution could expose CI data from an unintended ref type. - **CVE-2026-2710 — Authentication endpoints, CE/EE** - CVSS 4.3. - Blocked project access tokens could continue accessing private resources in certain cases. ## Bug Fixes in 19.0.1 - Added write permissions for repositories in job-token fine-grained permissions. - Fixed broken trial calls to action on the self-managed GitLab Credits dashboard. - Updated API security remediation guidance. - Removed Helm-based release-environment QA coverage. - Backported final 19.0 release notes and related maintenance changes. ## Bug Fixes in 18.11.4 - Updated Ruby thread scheduler behavior and Zlib to version 3.2.3. - Upgraded GitLab Shell to 14.50.0 and Elasticsearch indexer to 5.14.7. - Fixed lost comments when moving wiki pages. - Improved subgroup provisioning, synchronization workers, pipeline cancellation, and build handling. - Corrected issues affecting swimlanes, epic boards, CI catalog filtering, and pipeline traces. - Improved advanced-search bulk indexing by using the primary database connection. - Added AI workflow scope support and endpoint handling for diagram proxies. - Included performance optimizations for license approval workflows. ## Recommended Action Administrators of affected self-managed GitLab installations should upgrade to 19.0.1, 18.11.4, or 18.10.7 immediately, using the appropriate deployment method such as Omnibus, source, or Helm.

Read original(opens in new tab)

ElasticsearchCI/CDGitlabGraphQL+4

discord·May 8, 2026·3 min readCurated summary

How Discord Automates ScyllaDB Clusters at Scale

Discord’s Persistence Infrastructure team replaced fragile, manually sequenced scripts with the Scylla Control Plane (SCP), a framework for safely automating large-scale database operations. The effort was driven by the difficulty of creating shadow clusters and managing hundreds of ScyllaDB nodes with a seven-person team. SCP emphasizes resumability, safety checks, configurable parallelism, and incremental development. ## The Scale of Discord’s Database Operations - Discord operates Elasticsearch, Postgres, and ScyllaDB infrastructure across dozens of clusters and hundreds of nodes. - ScyllaDB stores critical data, including messages, channels, servers, and much of Discord’s user data. - Routine work includes: - Rolling restarts after configuration changes - Cluster expansion as traffic grows - Operating-system upgrades without downtime - Creating test clusters for validating ScyllaDB releases - These operations require careful sequencing and continuous validation rather than simple, fire-and-forget automation. ## From Scripts to the Scylla Control Plane - Discord initially accumulated Python, Bash, and other scripts incrementally. - The scripts were useful but fragile and dependent on institutional knowledge. - As operational demands grew, Discord created the Scylla Control Plane, or SCP, to provide a more structured automation system. ## Shadow Clusters for Safer Upgrades - Shadow clusters are temporary, full replicas of production that receive the same reads and writes as live traffic. - They allow Discord to detect upgrade problems under realistic load before changing production. - Building one manually requires: - Provisioning and configuring nodes - Joining nodes to the cluster - Validating replication - Establishing dual-write pipelines - Eventually tearing the environment down - Repeating this process across every ScyllaDB cluster made automation essential, especially for testing operating-system, hardware, and ScyllaDB version changes. ## Lessons from the Previous Automation Discord identified three major weaknesses in its old scripts: - **Unsafe:** Scripts could be run against the wrong nodes or in the wrong order, often without precondition checks. - **Unrecoverable:** A failure late in a multi-step process required restarting from the beginning. - **Difficult to extend:** New operations often required copying and modifying existing scripts instead of composing reusable components. SCP was designed around four goals: - Provide an extensible task framework that hides orchestration complexity. - Support configurable parallelism, including constraints such as avoiding simultaneous work in different availability zones. - Make safety the default through preconditions, retries, and persisted state. - Deliver functionality incrementally and refine it through real-world use. ## SCP’s Task-Based Architecture - SCP is organized around **tasks, workflows, and jobs**. - A task represents one unit of work, such as draining a node, checking repair status, or running cleanup. - **Node tasks** operate on individual nodes. - **Cluster tasks** coordinate operations across an entire cluster and may run node tasks across many nodes. - SCP also uses **conditions**, which pause execution until a required state is reached. - Conditions poll ScyllaDB APIs or Prometheus metrics. - They either succeed when the criterion is met or fail after a timeout. - For example, after restarting a node, SCP can wait for compactions to settle before continuing. - This avoids unreliable fixed-duration sleeps and reduces the risk of creating cascading pressure during rolling operations. ## Practical Recommendation For large-scale database operations, automation should be built as a reusable, stateful orchestration framework rather than a collection of scripts. Explicit preconditions, observable conditions, retries, controlled parallelism, and resumable state make complex infrastructure changes safer and more repeatable.

Read original(opens in new tab)

ElasticsearchPostgreSQLRustPrometheus+3

netflix·Apr 4, 2026·3 min readCurated summary

Powering Multimodal Intelligence for Video Search

Video search is difficult because it must combine many kinds of information—characters, scenes, dialogue, labels, and embeddings—across enormous volumes of footage. The post argues that solving this problem requires a distributed pipeline that separates reliable ingestion, computationally intensive data fusion, and low-latency search indexing. Temporal bucketing, hybrid ranking, and deduplication turn billions of model outputs into searchable moments for editors. ## Why Video Search Is Complex - Video contains multiple overlapping modalities, each analyzed by specialized models. - Models produce different outputs, including: - Text labels such as characters or objects - Scene classifications - High-dimensional embedding vectors - Time ranges with varying boundaries - Overlapping model timelines must be synchronized into a chronological representation. - A 2,000-hour archive may contain more than 216 million frames, expanding to billions of records after multimodal processing. - Search must avoid returning thousands of redundant clips from continuous shots. - Ranking therefore combines: - Symbolic text matching for precision and interpretability - Semantic vector similarity for contextual relevance - Clustering and deduplication to identify the best moments - Sub-second response times are essential because delays interrupt editors’ creative workflows. ## Three-Stage Ingestion and Fusion Pipeline ### Transactional Persistence - Raw model annotations are ingested through highly available pipelines. - Apache Cassandra stores the annotations with an emphasis on: - Data integrity - Distributed availability - High write throughput - An annotation can include a type, nanosecond time range, embedding vector, label, and confidence score. ### Offline Data Fusion - After persistence, Apache Kafka publishes an event that starts asynchronous processing. - The offline pipeline performs expensive temporal intersections without slowing ingestion or search. - Model outputs are normalized into fixed one-second time buckets. - The fusion process: - Maps continuous detections into discrete intervals - Intersects annotations sharing a bucket - Combines them into unified records - Writes the enriched records back to Cassandra - For example, a “Joey” character detection from seconds 2–8 can be combined with a “kitchen” scene detection from seconds 4–9 to create a fused record for the 4–5 second interval. - Each fused record retains links to the original annotations and source asset. ### Real-Time Search Indexing - Enriched buckets are later sent from Cassandra to Elasticsearch. - Upserts use a composite key consisting of the asset ID and time bucket. - If a bucket already exists, it is updated rather than duplicated. - This creates one consistent record for each second of footage while allowing new model results to be incorporated. The overall recommendation is to treat multimodal video search as a distributed data-fusion problem rather than a single-model retrieval task. Decoupling ingestion, offline processing, and indexing allows the system to handle massive archives while preserving reliable data capture and fast, context-rich search.

Read original(opens in new tab)

ElasticsearchDatabase DesignApache KafkaApache Cassandra+4

toss·Dec 15, 2025Original article

Customers Never Wait: How to Skyrocket (opens in new tab)

Toss Payments addressed the challenge of serving rapidly growing transaction data within a microservices architecture (MSA) by evolving their data platform from simple Elasticsearch indexing to a robust CQRS pattern. While Apache Druid initially provided high-performance time-series aggregation and significant cost savings, the team eventually integrated StarRocks to overcome limitations in data consistency and complex join operations. This architectural journey highlights the necessity of balancing real-time query performance with operational scalability and domain decoupling. ### Transitioning to MSA and Early Search Solutions * The shift from a monolithic structure to MSA decoupled application logic but created "data silos" where joining ledgers across domains became difficult. * The initial solution utilized Elasticsearch to index specific fields for merchant transaction lookups and basic refunds. * As transaction volumes doubled between 2022 and 2024, the need for complex OLAP-style aggregations led to the adoption of a CQRS (Command Query Responsibility Segregation) architecture. ### Adopting Apache Druid for Time-Series Data * Druid was selected for its optimization toward time-series data, offering low-latency aggregation for massive datasets. * It provided a low learning curve by supporting Druid SQL and featured automatic bitmap indexing for all columns, including nested JSON keys. * The system decoupled reads from writes, allowing the data team to serve billions of records without impacting the primary transaction databases' resources. ### Data Ingestion: Message Publishing over CDC * The team chose a message publishing approach via Kafka rather than Change Data Capture (CDC) to minimize domain dependency. * In this model, domain teams publish finalized data packets, reducing the data team's need to maintain complex internal business logic for over 20 different payment methods. * This strategy simplified system dependencies and leveraged Druid’s ability to automatically index incoming JSON fields. ### Infrastructure and Cost Optimization in AWS * The architecture separates computing and storage, using AWS S3 for deep storage to keep costs low. * Performance was optimized by using instances with high-performance local storage instead of network-attached EBS, resulting in up to 9x faster I/O. * The team utilized Spot Instances for development and testing environments, contributing to a monthly cloud cost reduction of approximately 50 million KRW. ### Operational Challenges and Druid’s Limitations * **Idempotency and Consistency:** Druid struggled with native idempotency, requiring complex "Merge on Read" logic to handle duplicate messages or state changes. * **Data Fragmentation:** Transaction cancellations often targeted old partitions, causing fragmentation; the team implemented a 60-second detection process to trigger automatic compaction. * **Join Constraints:** While Druid supports joins, its capabilities are limited, making it difficult to link complex lifecycles across payment, purchase, and settlement domains. ### Hybrid Search and Rollup Performance * To ensure high-speed lookups across 10 billion records, a hybrid architecture was built: Elasticsearch handles specific keyword searches to retrieve IDs, which are then used to fetch full details from Druid. * Druid’s "Rollup" feature was utilized to pre-aggregate data at ingestion time. * Implementing Rollup reduced average query response times from tens of seconds to under 1 second, representing a 99% performance improvement for aggregate views. ### Moving Toward StarRocks * To solve Druid's limitations regarding idempotency and multi-table joins, Toss Payments began transitioning to StarRocks. * StarRocks provides a more stable environment for managing inconsistent events and simplifies the data flow by aligning with existing analytical infrastructure. * This shift supports the need for a "Unified Ledger" that can track the entire lifecycle of a transaction—from payment to net profit—across disparate database sources.

ElasticsearchApache KafkaData ArchitectureApache Druid+4

daangn·Dec 15, 2025Original article

Easily Operating Karrot Search Engine (opens in new tab)

This blog post by the Daangn (Karrot) search platform team details their journey in optimizing Elasticsearch operations on Kubernetes (ECK). While their initial migration to ECK reduced deployment times, the team faced critical latency spikes during rolling restarts due to "cold caches" and high traffic volumes. To achieve a "deploy anytime" environment, they developed a data node warm-up system to ensure nodes are performance-ready before they begin handling live search requests. ## Scaling Challenges and Operational Constraints - Over two years, Daangn's search infrastructure expanded from a single cluster to four specialized clusters, with peak traffic jumping from 1,000 to over 10,000 QPS. - The initial strategy of "avoiding peak hours" for deployments became a bottleneck, as the window for safe updates narrowed while total deployment time across all clusters exceeded six hours. - Manual monitoring became a necessity rather than an option, as engineers had to verify traffic conditions and latency graphs before and during every ArgoCD sync. ## The Hazards of Rolling Restarts in Elasticsearch - Standard Kubernetes rolling restarts are problematic for stateful systems because a "Ready" Pod does not equate to a "Performant" Pod; Elasticsearch relies heavily on memory-resident caches (page cache, query cache, field data cache). - A version update in the Elastic Operator once triggered an unintended rolling restart that caused a 60% error rate and 3-second latency spikes because new nodes had to fetch all data from disk. - When a node restarts, the cluster enters a "Yellow" state where remaining replicas must handle 100% of the traffic, creating a single point of failure and increasing the load on the surviving nodes. ## Strategy for Reliable Node Warm-up - The primary goal was to reach a state where p99 latency remains stable during restarts, regardless of whether the deployment occurs during peak traffic hours. - The solution involves a "Warm-up System" designed to pre-load frequently accessed data into the filesystem and Elasticsearch internal caches before the node is allowed to join the load balancer. - By executing representative search queries against a newly started node, the system ensures that the necessary segments are already in the page cache, preventing the disk I/O thrashing that typically follows a cold start. ## Implementation Goals - Automate the validation of node readiness beyond simple health checks to include performance readiness. - Eliminate the need for human "eyes-on-glass" monitoring during the 90-minute deployment cycles. - Maintain high availability and consistent user experience even when shards are being reallocated and replicas are temporarily unassigned. To maintain a truly resilient search platform on Kubernetes, it is critical to recognize that for stateful applications, "available" is not the same as "ready." Implementing a customized warm-up controller or logic is a recommended practice for any high-traffic Elasticsearch environment to decouple deployment schedules from traffic patterns.

ElasticsearchKubernetesInfrastructureSearch Engine+3

toss·Nov 13, 2025Original article

Frontend Code That Lasts 1 (opens in new tab)

Toss Payments evolved its Payment SDK to solve the inherent complexities of integrating payment systems, where developers must navigate UI implementation, security flows, and exception handling. By transitioning from V1 to V2, the team moved beyond simply providing a library to building a robust, architecture-driven system that ensures stability and scalability across diverse merchant environments. The core conclusion is that a successful SDK must be treated as a critical infrastructure layer, relying on modular design and deep observability to handle the unpredictable nature of third-party runtimes. ## The Unique Challenges of SDK Development * SDK code lives within the merchant's runtime environment, meaning it shares the same lifecycle and performance constraints as the merchant’s own code. * Internal logging can inadvertently create bottlenecks; for instance, adding network logs to a frequently called method can lead to "self-DDoS" scenarios that crash the merchant's payment page. * Type safety is a major hurdle, as merchants may pass unexpected data types (e.g., a number instead of a string), causing fatal runtime errors like `startsWith is not a function`. * The SDK acts as a bridge for technical communication, requiring it to function as both an API consumer for internal systems and an API provider for external developers. ## Ensuring Stability through Observability * To manage the unpredictable ways merchants use the SDK, Toss implemented over 300 unit tests and 500 E2E integration tests based on real-world use cases. * The team utilizes a "Global Trace ID" to track a single payment journey across both the frontend and backend, allowing for seamless debugging across the entire system. * A custom Monitoring CLI was developed to compare payment success rates before and after deployments, categorized by merchant and runtime environment (e.g., PC Chrome vs. Android WebView). * This observability infrastructure enables the team to quickly identify edge-case failures—such as a specific merchant's checkout failing only on mobile WebViews—which are often missed by standard QA processes. ## Scaling with Modular Architecture * To avoid "if-statement hell" caused by merchant-specific requirements (e.g., fixing installment months or custom validation for a specific store), Toss moved to a "Lego-block" architecture. * The SDK is organized into three distinct layers based on the "reason for change" principle: * **Public Interface Layer:** Manages the contract with the merchant, validating inputs and translating them into internal domain models. * **Domain Layer:** Encapsulates core business logic and payment policies, keeping them isolated from external changes. * **External Service Layer:** Handles dependencies like Server APIs and Web APIs, ensuring technical shifts don't leak into the business logic. * This separation allows the team to implement custom merchant logic by swapping specific blocks without modifying the core codebase, reducing the risk of regressions and lowering maintenance costs. For developers building SDKs or integration tools, the shift from monolithic logic to a layered, observable architecture is essential. Prioritizing the separation of domain logic from public interfaces and investing in environment-specific monitoring allows for a highly flexible product that remains stable even as the client-side environment grows increasingly complex.

ElasticsearchObservabilityTypeScriptSoftware Architecture+3

datadog·Nov 4, 2025·3 min readCurated summary

Replication redefined: How we built a low-latency, multi-tenant data replication platform

Datadog built a managed, multi-tenant data replication platform to move data reliably across thousands of services without brittle, point-to-point integrations. The effort began by separating analytical search workloads from a shared PostgreSQL database, then evolved into automated pipeline provisioning with Temporal. The platform favors asynchronous replication to improve scalability and resilience, accepting limited replication lag in exchange for lower application latency and reduced operational coupling. ## Scaling Search Beyond PostgreSQL - A shared PostgreSQL database initially provided low-latency access, ACID guarantees, and low operational cost. - As data volumes grew, complex joins and aggregations became increasingly slow. - Datadog’s Metrics Summary page had to join: - 82,000 active metrics - 817,000 metric configurations - Page latency reached approximately 7 seconds at p90, while repeated facet changes generated additional expensive queries. - Index and disk bloat, memory pressure, VACUUM and ANALYZE overhead, and rising I/O wait further reduced throughput. - Rather than continuing to optimize PostgreSQL for analytical search, Datadog moved search and aggregation workloads to a dedicated search platform. - Data was denormalized during replication, producing document-oriented indexes better suited to faceted search. - The resulting system reduced page-load times by as much as 97%—from roughly 30 seconds to 1 second—while maintaining about 500 ms of replication lag. ## Automating Pipeline Provisioning with Temporal Provisioning a replication pipeline required coordinating multiple systems and configuration steps: - Enabling PostgreSQL logical replication with `wal_level`. - Creating users and assigning replication permissions. - Configuring publishers and replication slots. - Deploying Debezium instances to capture PostgreSQL changes. - Creating Kafka topics and mapping them to Debezium instances. - Adding heartbeat tables to monitor replication and prevent excessive WAL retention. - Configuring sink connectors to write Kafka data into the search platform. Manual management became increasingly difficult across many pipelines and data centers. Datadog used Temporal workflows to split provisioning into modular, repeatable tasks and combine them into higher-level orchestrations. This reduced errors, improved consistency, and allowed engineers to create and modify pipelines without repeating complex operational procedures. ## Choosing Asynchronous Replication - Synchronous replication provides strong consistency by waiting for replicas to acknowledge each write. - However, it increases latency and operational complexity, particularly across distributed environments. - Asynchronous replication allows the primary system to acknowledge writes immediately while replicas catch up afterward. - Datadog selected the asynchronous model because it decouples application performance from network latency and replica availability. - The trade-off is temporary replication lag during failures or periods of pressure, but the model offers better scalability and resilience for high-throughput systems. Datadog’s experience suggests that replication should be treated as a managed platform rather than a collection of custom integrations. Separating workloads, automating provisioning, and choosing asynchronous delivery can improve performance and reliability while reducing the operational burden on individual engineering teams.

Read original(opens in new tab)

ElasticsearchDatabase DesignPostgreSQLApache Kafka+4

discord·May 1, 2025·3 min readCurated summary

How Discord Indexes Trillions of Messages

Discord’s original Elasticsearch-based search system worked well for billions of messages but became fragile as message volume and cluster size grew. Redis queues could drop messages, bulk operations failed too broadly, large clusters were difficult to operate, and individual indices could hit Lucene’s roughly two-billion-document limit. Discord’s response was to modernize the platform with Kubernetes, the Elastic Kubernetes Operator, and a multi-cluster “cell” architecture built from smaller clusters. ## The Original Search Architecture - Messages were stored in Elasticsearch indices distributed across two clusters. - Data was sharded by Discord server (guild) or direct message, keeping each guild’s messages together for efficient queries. - Messages were indexed lazily because not every message is searched. - Redis-backed queues supplied workers with message batches for Elasticsearch bulk indexing. ## Problems with the Existing System ### Redis Queue Message Loss - The realtime indexing queue relied on Redis. - When Elasticsearch failures caused the queue to back up, Redis CPU usage could reach its limit. - Once overloaded, Redis began dropping messages, making the indexing pipeline unreliable. ### Fault-Intolerant Bulk Indexing - A batch could contain messages belonging to many different Elasticsearch indices and nodes. - A batch of 50 messages might fan out to dozens of nodes. - If one message failed because its target node was unavailable, Elasticsearch treated the entire bulk request as failed. - All messages were then re-enqueued, increasing queue pressure. - In a 100-node cluster with batches of 50 messages, a single failed node gave each batch roughly a 40% chance of encountering a failure. ### Large-Cluster Overhead - Adding nodes and indices enabled horizontal scaling but increased coordination overhead. - Bulk operations fanned out across more nodes, slowing indexing. - Larger clusters also had a higher probability that some node would fail. ### Difficult Upgrades and Restarts - The system lacked sufficient resilience to individual node outages, making rolling restarts unsafe. - Clusters exceeding 200 nodes and containing terabytes of data would have taken too long to drain gracefully. - Discord therefore remained on outdated operating-system and Elasticsearch versions. - Addressing the Log4Shell vulnerability required taking the entire search system offline while every node was restarted. ### Oversized Indices - Some indices accumulated messages from extremely large guilds. - Each Elasticsearch index is backed by a Lucene index with a limit of approximately two billion documents. - Once that limit was reached, all further indexing failed. - Discord temporarily recovered by identifying and deleting guilds created primarily for message spam, but this was not viable for legitimate high-volume communities. ## Moving Elasticsearch to Kubernetes - Discord chose Kubernetes to improve operational flexibility and resource efficiency. - The Elastic Cloud on Kubernetes (ECK) Operator could define cluster topology and configuration declaratively. - Kubernetes would automate operating-system upgrades. - ECK provided tools for safer rolling restarts and Elasticsearch upgrades. - This marked Discord’s first move toward managing stateful Elasticsearch infrastructure on Kubernetes. ## Smaller Multi-Cluster Cells - Discord planned to replace very large clusters with a larger number of smaller Elasticsearch clusters. - Smaller clusters reduce coordination overhead and limit the impact of individual node failures. - A cell-based design also provides a more manageable scaling and operational boundary than clusters with hundreds of nodes. Discord’s experience demonstrates that scaling Elasticsearch is not only a matter of adding nodes. Reliable operation requires isolating failures, avoiding oversized indices and fan-out-heavy batches, and designing deployment infrastructure that supports upgrades without taking search offline.

Read original(opens in new tab)

ElasticsearchKubernetesDistributed SystemsRedis+2

figma·Oct 9, 2024·2 min readCurated summary

The Search for Speed in Figma | Figma Blog

Figma’s search investigation revealed that OpenSearch itself was responsible for less than 30% of total search latency. The larger costs came from query construction and especially permission checks before and after searches. By measuring the correct end-to-end metrics, Figma identified misleading monitoring data and established a foundation for improving search performance at scale. ## Diagnosing the Latency Gap - Figma migrated from an older Elasticsearch version to AWS-managed OpenSearch, a fork created after Elasticsearch’s 2021 license change. - OpenSearch reported an average search time of roughly **8 ms**, while Figma’s API showed: - About **150 ms average latency** - **200–400 ms** latency at the 99th percentile - Minimum latency above **40 ms** - Search performance also varied significantly depending on traffic levels, with peak periods much slower than weekends. - Additional instrumentation showed that substantial time was spent both before and after the OpenSearch request. ## Understanding OpenSearch’s Metrics - OpenSearch distributes a query through a coordinator node to worker nodes, typically sending one request per index shard. - It then gathers, sorts, and fetches results during the query and fetch phases. - The reported 8 ms metric measured only the average time for individual shard queries—not the total time required to coordinate hundreds of shard requests. - Figma’s queries could involve as many as **500 shard-level requests**, many of which ran in parallel but not all. - OpenSearch did not provide built-in metrics or logs for overall query duration. - Figma instead extracted the `took` value from each search response, producing a backend latency measure that aligned more closely with application-level timing. ## Permission Processing as the Main Bottleneck - Less than 30% of total query API time was spent waiting for OpenSearch. - Pre-processing: - Retrieved information about files the user could access. - Built an OpenSearch filter intended to exclude inaccessible files. - Post-processing: - Performed additional permission checks on every returned file. - Was especially slow and consumed more time than the search itself. - The investigation demonstrated that optimizing the search engine alone would not solve Figma’s overall latency problem. Figma’s experience highlights the importance of measuring end-to-end request latency rather than relying on subsystem metrics. Accurate coordinator-level and application-level instrumentation is essential, particularly when distributed searches involve many shards and expensive authorization work.

Read original(opens in new tab)

ElasticsearchObservabilityAWSOpensearch+2

figma·Jul 20, 2020·2 min readCurated summary

A deep dive on deep search | Figma Blog

Figma’s deep search lets users find files by searching text inside them rather than relying on file names or metadata. Building it required extending infrastructure originally created for Design System Analytics to process `.fig` files stored in Amazon S3. Because analyzing large file trees is expensive, Figma accepted briefly stale results and processed deduplicated changes hourly. ## Deep Search in a Browser-Based Product - Figma’s web-based architecture provides detailed access to files and usage data. - This enables features such as: - Component usage analytics - File-view frequency - Inspection of file structure - Searching content inside files - Deep search builds on the browser’s collaboration and discoverability advantages. ## Reusing Design System Analytics Infrastructure - Design System Analytics already opened recently edited files, retrieved them from storage, and traversed their contents. - Analytics extracted shared-library usage information. - Deep search applies the same general workflow to extract text from Figma files. - The existing file-analyzer worker platform provided support for computationally intensive, periodic processing. ## Regular Search vs. Deep Search - Regular search indexes database metadata, including: - File name - Creator - Folder ID - Team ID - Its pipeline: - Database changes are streamed into a messaging system. - Search indexers retrieve current records. - The metadata is indexed in Elasticsearch. - Deep search cannot rely on database metadata because the actual file contents are stored as `.fig` documents in Amazon S3. - A `.fig` file is represented as a tree of nodes, such as frames, rectangles, vectors, ellipses, and text objects, each with its own properties. ## Managing the Cost of File Analysis - Retrieving and traversing a complete Figma file is significantly more expensive than reading database records. - Files may contain thousands of nodes, and users can trigger saves approximately every 30 seconds. - Re-indexing every save would produce substantial duplicated computation. - Figma therefore: - Deduplicates file changes over one-hour windows. - Sends changed files to file-analyzer workers. - Allows deep-search results to be temporarily stale. - This tradeoff reduces server workload while maintaining useful search functionality. Deep search demonstrates how content-aware features require different infrastructure from conventional metadata search. Periodic, deduplicated processing offers a practical balance between timely results and the high computational cost of analyzing complete design files.

Read original(opens in new tab)

ElasticsearchDistributed SystemsAmazon S3Data Pipeline+2