Caching | Techlist.io

cloudflareJul 23, 20263 min readCurated summary

Introducing Cache Response Rules

Cache Response Rules let Cloudflare modify an origin response after it arrives but before it is stored in cache. They address response-header problems—such as accidental `Set-Cookie`, restrictive `Cache-Control`, or problematic validators—that can unnecessarily reduce cacheability. This provides an origin-independent fix while preserving the distinction between request-time and response-time caching decisions. ## When Caching Decisions Are Made - CDN caches aim to serve content from the edge and contact the origin only on misses. - Origin response headers determine: - Whether content can be cached - How long it remains fresh - When it should be revalidated - Whether it should be cached at all - Common problems include: - `Set-Cookie` on static assets such as `/static/app.js`, making them uncacheable - `Cache-Control: no-cache` on content that is safe to cache at the CDN - Browser-oriented cache directives that are unsuitable for Cloudflare - Overly aggressive `ETag` values causing repeated revalidation - These issues often require coordination between separate origin and CDN teams, delaying simple fixes. ## Cache Response Rules - Run after the origin responds but before Cloudflare writes the response to cache. - Can: - Rewrite `Cache-Control` directives - Strip `Set-Cookie`, `ETag`, and `Last-Modified` - Manage cache tags for purging - Apply entirely within Cloudflare, without requiring origin code changes. - They solve problems that request-time rules cannot, because response headers are unavailable until after the origin request completes. ## The Missing Piece in Cloudflare’s Cache Controls - Earlier caching behavior was largely handled through Page Rules, which combined caching with unrelated features. - Cloudflare later introduced more focused controls, including: - Cache Rules - CDN-Cache-Control - Custom cache keys - Other cache-specific settings - Most existing controls operate during the request phase. - Before contacting the origin, Cloudflare can evaluate only request information such as the URL, headers, file extension, geography, and device type. - Previously, response-header problems required: - Changing the origin - Deploying a Worker to re-fetch and rewrite responses - Accepting a lower cache hit ratio ## Two Phases, Two Questions - **Cache Rules** run before the origin request and determine: - Whether the response is eligible for caching - What cache key identifies the object - How it should be cached, including TTL and stale-serving behavior - **Cache Response Rules** run after the origin response and determine whether caching behavior should be adjusted: - Remove headers that make content ineligible - Change origin cache directives - Set cache tags - When the two rule types conflict, the Cache Response Rule takes precedence. - Response rules cannot change the cache key, since that must already be established during the request phase. - They can make an otherwise eligible response non-cacheable with `no-store`, or make content eligible by removing `Set-Cookie`, but they cannot recover the latency of a request that was already excluded from caching at request time. Cache Response Rules complement rather than replace Cache Rules. Use request-phase rules for cache eligibility, keys, and general caching behavior; use response-phase rules to correct origin headers before they damage cacheability.

Read original

Caching Cloudflare Cdn Cache Response Rules+2

googleJun 25, 20263 min readCurated summary

Optimizing cloud economics with linear elastic caching

Linear elastic caching treats cache memory as a variable cost rather than a fixed allocation. It dynamically adjusts how long pages remain in memory by balancing ongoing memory expense against the cost of fetching evicted data again, using the ski rental problem as its theoretical foundation. Experiments in Spanner and public cache traces show meaningful cost reductions with only modest increases in misses. ## Fixed-Size Cache Limitations - Traditional caches allocate a fixed amount of RAM and use policies such as LRU when space runs out. - Undersizing the cache causes excessive disk or storage access and poor performance. - Oversizing it wastes money during periods of low demand; some serverless providers charge up to $3 per day for 1 GiB of memory. - Fixed sizing therefore creates a “Goldilocks” problem as workloads fluctuate. ## Ski Rental Model for Cache Eviction - Each cached page presents two choices: - **Rent:** Keep it in RAM and continuously pay for its memory footprint. - **Buy the miss:** Evict it and risk a latency and I/O penalty if it is requested again. - A ski rental algorithm assigns each page a time-to-live (TTL). - If the page is not accessed before its TTL expires, it is evicted. - If the cache becomes physically full, a conventional policy such as LRU handles capacity pressure. - The researchers prove that eviction policy and rental duration can be optimized separately, simplifying implementation. - Unlike worst-case break-even or randomized ski rental strategies, lightweight machine learning can exploit predictable workload patterns. ## Lightweight TTL Prediction - In Spanner, each page receives a TTL based on: - Page size - Cost of a cache miss - Type of database operation - Observed access behavior - A shallow decision tree was chosen because Spanner processes billions of requests per second. - The model can be translated into a few lines of interpretable C++ code. - Its cost-aware decisions allow extra misses mainly for data that is inexpensive to retrieve. ## Spanner Production Results - Compared with a standard fixed-size cache: - Memory usage fell by **15.5%**. - Cache misses increased by only **5.5%**. - Total cost of ownership fell by approximately **5%**. - The additional misses increased actual I/O costs by only **0.5%**, because they were concentrated on cheap-to-fetch data. - The policy was deployed on production Spanner servers and evaluated over several months. ## Public Trace Evaluation - The approach was tested on public industry cache traces using GDSF as the fixed-size baseline. - GDSF generalizes LRU to account for pages with different sizes. - Researchers evaluated four elastic-cache variants using: - Break-even or randomized ski rental policies - Learned or non-learned TTL selection - Because public traces lacked application-level features, learning used the first half of each trace to calculate the best TTL for individual pages. - Caches were warmed with one day of requests before performance measurement began. ## Overall Results - Elastic caching consistently produced lower total cost across diverse workloads. - Its advantage increased as memory became more expensive relative to cache misses. - At comparable cache sizes, elastic policies also achieved substantially lower miss rates than fixed-size approaches. Linear elastic caching is most useful when memory costs vary significantly or workloads are bursty and predictable. Dynamically assigning page TTLs offers a practical way to reduce memory spending while limiting performance impact, especially when the system can estimate the cost of each miss.

Read original

Caching Machine Learning C Plus Plus Database Systems+4

figmaJun 11, 20262 min readCurated summary

Improving Performance in the Layers Panel | Figma Blog

Figma rebuilt its layers panel to handle files containing tens of thousands of layers. The old architecture recomputed too much data too often, slowing both panel interactions and broader editor operations. A two-pass computation model and cached derived properties now make some interactions 30–50% faster. ## Why the Original Architecture Slowed Down - Figma files are trees of nodes with properties and children. - The panel previously built a large JavaScript object in one recursive pass. - Every expanded node—and often all of its descendants—had its display data recomputed after changes. - This created two problems: - Data was computed for hundreds of thousands of nodes even though only 20–30 rows were visible. - Small changes, such as expanding a node, triggered broad recomputation because incremental results were rarely cached. ## Two-Pass Computation - The first pass computes only the ordered list of row IDs shown in the panel. - Determining that order still requires handling complex rules, including: - Reversed child ordering in autolayout frames. - Node types such as widgets and FigJam stickies that hide children. - Fixed and scrolling headers that divide prototype-frame children. - Sticky top-level frames and components. - The second pass gathers display data—names, icons, lock and visibility state, and selection state—only for rows inside the visible window. - This makes windowing effective: previously, Figma computed data for off-screen rows even though they were not rendered. ## Caching Derived Data - Figma introduced its “derived properties” platform primitive to avoid recomputing unchanged data. - Nodes store mutable fields, while other values are calculated from those fields and related properties. - For example, a node’s absolute position can be derived from its parent’s absolute position and its relative position: ```text Self.AbsolutePosition = Parent.AbsolutePosition + Self.RelativePosition ``` - Derived properties: - Track their dependencies through an optimized dependency graph. - Support different caching strategies balancing speed and memory. - Are lazy by default, computing values only when they are read. - Because the layers panel is itself a tree, this dependency-aware system lets Figma update only affected rows while keeping stable portions cached. Figma’s results show the value of combining virtualization with incremental, dependency-based computation: large hierarchical interfaces can remain responsive when they calculate only visible data and preserve everything that has not changed.

Read original

Caching JavaScript Virtualization Dependency Graph+2

netflixApr 6, 20264 min readCurated summary

Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale

Netflix’s Druid deployment now exceeds 10 trillion rows and can ingest 15 million events per second, but repetitive dashboard queries became a scaling problem. Its new experimental caching layer handles rolling time windows by reusing settled historical results and querying Druid only for recent, changing data. Netflix accepts up to five seconds of additional staleness in exchange for substantially lower query load. ## The Scaling Problem - A dashboard with 26 charts can issue 64 queries per load. - Viewed by 30 people and refreshed every 10 seconds, that becomes roughly 192 queries per second. - Druid’s full-result cache misses whenever a rolling time interval changes. - Druid avoids caching realtime segments to preserve result correctness and determinism. - Per-segment caching reduces historical scans but still requires brokers to gather and merge results for every request. - Adding hardware to handle this redundant workload would be prohibitively expensive. ## Caching Only the Unsettled Data - In a three-hour query, most data is already stable; only the newest minutes are likely to change. - The cache stores previously returned historical portions and sends Druid only the uncached interval. - This approach is designed for time-grouped queries such as timeseries and groupBy queries. ## Deliberate Staleness - The cache can make the newest data up to five seconds stale. - This is acceptable because dashboards typically refresh every 10–30 seconds. - Netflix’s pipeline already has up to roughly five seconds of latency at P90. - Many queries also intentionally end at `now-1m` or `now-5s` to avoid unstable, newly arriving data. ## Exponential TTLs - Cache lifetimes increase with the age of each data point because older data is less likely to change. - Data under two minutes old has a minimum TTL of five seconds. - After that, TTL doubles for each additional minute: - 10 seconds at two minutes old - 20 seconds at three minutes - 40 seconds at four minutes - TTLs are capped at one hour. - Fresh data is refreshed frequently to account for late-arriving events, while older data remains cached longer. ## Time-Based Bucketing - A single cache entry per query and interval would still miss whenever a rolling window shifted. - Netflix instead uses a map-of-maps: - The outer key is a hash of the query excluding its time interval. - Inner keys represent timestamps bucketed by query granularity or at least one minute. - Big-endian timestamp encoding preserves chronological order for efficient range scans. - A three-hour query at one-minute granularity becomes 180 independently cached buckets. - When the window moves, most buckets can be reused and only the newly exposed range must be fetched. ## Router-Integrated Cache Service - The cache currently operates as an external service behind the Druid Router. - Cacheable requests are intercepted transparently: - Fully cached requests are answered directly. - Partially cached requests are narrowed to the missing interval and sent to Druid. - Metadata queries and queries without time-based grouping bypass the cache. - The proxy can be enabled or disabled without changing clients. - Netflix views this as an interim design while exploring deeper integration with Druid. ## Query Identification and Lookup - Incoming queries are parsed to extract their interval, granularity, and structure. - A SHA-256 hash is generated from the query’s logical contents, including datasource, filters, aggregations, and relevant context properties, while excluding the time interval. - The cache looks for buckets within the requested range. - Lookup requires cached buckets to be contiguous from the beginning of the requested interval; the provided article text ends while explaining the handling of expired or missing buckets. Netflix’s approach is best suited to frequently repeated rolling-window dashboards where a small, slightly stale tail is acceptable. Segmenting results by time and assigning age-based TTLs allows the system to preserve freshness where it matters while eliminating most redundant Druid work.

Read original

Caching Database Design Query Optimization Data Ingestion+3

figmaMar 25, 20263 min readCurated summary

Figma's Next-Generation Data Caching Platform | Figma Blog

Figma built FigCache to address scalability, reliability, and operational weaknesses in its Redis-based caching infrastructure. The stateless proxy provides a unified Redis data plane, decouples Redis connections from volatile client fleets, centralizes routing and security, and standardizes observability. After rollout to Figma’s main API in 2025, the caching layer reached six nines of uptime. ## Growing pains in caching - Redis evolved from a secondary component into a critical dependency for site availability. - Redis clusters were nearing connection limits as Figma’s infrastructure grew. - Rapid client-service scaling caused thundering herds of new connections, creating I/O bottlenecks and reducing availability. - Decentralized traffic management allowed applications to pollute or corrupt data across clusters. - Client libraries provided inconsistent observability, complicating incident diagnosis and mitigation. - A fragmented client ecosystem made it difficult to guarantee correct client-side behavior during failovers and topology changes. - Figma initially reduced Redis dependency in core API subsystems and created service-specific connection pooling, but pursued a broader platform redesign for long-term scalability. ## Design goals for a durable platform Figma defined several objectives for a caching platform capable of supporting future growth: - **Decouple Redis from client volatility:** Redis connection volume should not rise directly with elastic application fleets. - **Provide built-in observability:** Service owners and platform operators should receive consistent, granular visibility across workloads in a multitenant environment. - **Hide Redis Cluster complexity:** Clients should not need to manage topology changes such as scaling, failovers, or shard loss. - **Offer a universal endpoint:** Applications should access multiple Redis clusters through a centralized routing layer rather than managing separate endpoints and clients. - **Enable alternative backends:** New storage technologies, including durable systems, should be usable behind the same protocol and API. - **Remain extensible:** Cross-cutting capabilities such as encryption, guardrails, and traffic backpressure should be implemented centrally rather than repeatedly in applications. ## FigCache’s foundational architecture - Figma identified the need for a caching proxy that would serve as: - A unified Redis data plane. - An ingress layer for applications. - A connection multiplexer shielding Redis from client connection spikes. - A language-agnostic interface that hides cluster routing and management. - The platform was designed to centralize traffic decisions and abstract the underlying Redis topology from application developers. - FigCache is stateless and communicates using the Redis RESP wire protocol, allowing existing Redis-compatible clients and first-party libraries to use it. - Its broader platform role includes centralized security, routing, and end-to-end observability across the caching stack. ## Results - FigCache was rolled out to Figma’s main API service during the second half of 2025. - The caching layer subsequently achieved six nines of uptime. - The system established a foundation for more reliable, scalable, and interchangeable ephemeral storage across Figma. Figma’s approach demonstrates that Redis reliability at large scale requires more than larger clusters: a dedicated platform layer can isolate connection volatility, simplify client behavior, and centralize operational controls.

Read original

Caching Observability Distributed Systems Redis+2

gitlabFeb 25, 20262 min readCurated summary

GitLab metrics and registry features help reduce CI/CD bottlenecks

GitLab’s two new beta features target common CI/CD bottlenecks without requiring additional third-party tools. CI/CD Job Performance Metrics provides job-level visibility into duration and failures, while Container Virtual Registry centralizes pulls from multiple registries through a cached GitLab endpoint. Together, they help platform teams identify pipeline problems faster and simplify container management. ## CI/CD Job Performance Metrics - Available in GitLab Premium and Ultimate. - Limited beta on GitLab.com; available on Self-Managed and Dedicated with ClickHouse configured. - Adds a job-focused panel to **Analyze > CI/CD analytics**. - Shows, for the previous 30 days by default: - Median (P50) and worst-case (P95) job duration - Failure rate - Job name and pipeline stage - Supports sorting, searching, and pagination to identify slow or unreliable jobs. - GitLab plans to add stage-level aggregation for build, test, and deploy bottlenecks. ## Container Virtual Registry - Available in GitLab Premium and Ultimate; API-ready in GitLab 18.9. - Provides one GitLab endpoint for pulling images from multiple upstream registries. - Supports registries such as Docker Hub, Harbor, Quay, and other sources using long-lived token authentication. - Uses pull-through caching to: - Reduce repeated downloads and bandwidth costs - Improve availability and reliability - Centralize authentication and registry configuration - Currently configured through the API, with UI management in development. - Cloud registries requiring IAM authentication, including Amazon ECR, Google Artifact Registry, and Azure Container Registry, may be supported later. ## Beta Access and Feedback - GitLab.com users can request access through their customer success manager or the feature’s feedback issue. - Self-managed users can enable the feature flag and configure the virtual registry through the API. - GitLab is seeking feedback to guide future improvements to both features. These betas are worth evaluating if your team needs better visibility into pipeline performance or manages images across several registries. The metrics feature can replace custom dashboards, while the virtual registry can reduce registry-related configuration and operational overhead.

Read original

Caching CI/CD Gitlab Docker & Containers+4

lineFeb 13, 20263 min readCurated summary

Slow Query Resolution: Optimizing Bit

LINE VOOM’s post server experienced intermittent timeouts when loading profiles belonging to users with hundreds of thousands of posts. The root cause was bitwise filtering on `category_flag` and `access_flag`, which prevented MySQL from efficiently using indexes and forced scans of all posts for a user. The team resolved the issue with MySQL 8.0.13 functional indexes and by changing the query predicates to exact decimal comparisons, reducing scanned rows from 805 to 31 in testing. ## The Slow Query and Its Root Cause - Post metadata was distributed across shards and partitioned tables. - `category_flag` and `access_flag` were stored as `bit(64)` values containing multiple status flags. - The problematic query filtered by: - `user_id` - `category_flag & 0x0100` - `access_flag & 0x0001` - For heavy users, the query scanned hundreds of thousands of posts and ran for more than 30 seconds. - Bitwise expressions operated on computed results rather than raw column values, preventing normal indexes from filtering efficiently. ## Choosing Functional Indexes - The team considered hardware upgrades, caching, and additional partitioning, but none addressed the root cause adequately. - MySQL 8.0.13 functional indexes could index expression results without changing the table schema. - The proposed composite index was: ```sql ALTER TABLE post_metadata ADD INDEX idx_user_premium_searchable ( user_id, (category_flag & 0x0100), (access_flag & 0x0001) ); ``` - Functional indexes rely on the query expression matching the index definition precisely. ## Discovering the Required Query Form - Initial attempts failed to use the index: - Truthy checks such as `category_flag & 0x0100` - Comparisons using `> 0` - Equality against hexadecimal values such as `= 0x0100` - The successful form used decimal equality: ```sql WHERE user_id = '{user_id}' AND (category_flag & 0x0100) = 256 AND (access_flag & 0x0001) = 1 ``` - In testing, scanned rows dropped from 805 to 31. - Index storage increased by approximately 24%, but the DBA team determined that production capacity was sufficient. ## Rolling Out the Indexes in Production - Indexes were created before changing the application queries. - The team used online schema changes to avoid service downtime and support pausing or rollback during replication problems. - Because dozens of tables across multiple shards were affected: - One shard was handled first for validation. - Only one or two tables were processed per day. - Work was avoided during periods when emergency DBA support was unavailable. - Index creation increased replication lag, causing newly created posts to temporarily disappear from read replicas. - The team reduced the cache expiration time for the affected post lists and accepted the remaining replication delay before resuming the rollout. ## Gradual Query Deployment and a Bitwise Logic Bug - Query changes were deployed gradually through a dynamic configuration system. - Each query pattern was tested on one shard before being expanded to the remaining shards. - This allowed changes to be rolled back immediately through configuration. - During rollout, a serious visibility bug was found. - The original condition: ```sql category_flag & 0x0110 ``` matched when either `0x0100` or `0x0010` was present, effectively representing an OR condition. - Rewriting it as: ```sql (category_flag & 0x0110) = 272 ``` required both bits to be set, creating an AND condition. - Because production data stored only the premium bit, some profiles returned no content. - The incident highlighted the need to verify the semantic meaning of bit flags before converting bitwise predicates into equality comparisons. ## Practical Recommendation For slow queries involving bit flags, consider functional indexes when using MySQL 8.0.13 or later. Ensure the query expression exactly matches the index definition, validate bitwise logic carefully, and use staged schema and query rollouts with monitoring and fast rollback mechanisms.

Read original

Caching Database Design Mysql Sharding+4

naverNov 25, 2025Original article

@RequestCache: Developing a Custom Annotation (opens in new tab)

The development of `@RequestCache` addresses the performance degradation and network overhead caused by redundant external API calls or repetitive computations within a single HTTP request. By implementing a custom Spring-based annotation, developers can ensure that specific data is fetched only once per request and shared across different service layers. This approach provides a more elegant and maintainable solution than manual parameter passing or struggling with the limitations of global caching strategies. ### Addressing Redundant Operations in Web Services * Modern web architectures often involve multiple internal services (e.g., Order, Payment, and Notification) that independently request the same data, such as a user profile. * These redundant calls increase response times, put unnecessary load on external servers, and waste system resources. * `@RequestCache` provides a declarative way to cache method results within the scope of a single HTTP request, ensuring the actual logic or API call is executed only once. ### Limitations of Manual Data Passing * The common alternative of passing response objects as method parameters leads to "parameter drilling," where intermediate service layers must accept data they do not use just to pass it to a deeper layer. * In the "Strategy Pattern," adding a new data dependency to an interface forces every implementation to change, even those that have no use for the new parameter, which violates clean architecture principles. * Manual passing makes method signatures brittle and increases the complexity of refactoring as the call stack grows. ### The TTL Dilemma in Traditional Caching * Using Redis or a local cache with Time-To-Live (TTL) settings is often insufficient for request-level isolation. * If the TTL is set too short, the cache might expire before a long-running request finishes, leading to the very redundant calls the system was trying to avoid. * If the TTL is too long, the cache persists across different HTTP requests, which is logically incorrect for data that should be fresh for every new user interaction. ### Leveraging Spring’s Request Scope and Proxy Mechanism * The implementation utilizes Spring’s `@RequestScope` to manage the cache lifecycle, ensuring that data is automatically cleared when the request ends. * Under the hood, `@RequestScope` uses a Singleton Proxy that delegates calls to a specific instance stored in the `RequestContextHolder` for the current thread. * The cache relies on `RequestAttribute`, which uses `ThreadLocal` storage to guarantee isolation between different concurrent requests. * Lifecycle management is handled by Spring’s `FrameworkServlet`, which prevents memory leaks by automatically cleaning up request attributes after the response is sent. For applications dealing with deep call stacks or complex service interactions, a request-scoped caching annotation provides a robust way to optimize performance without sacrificing code readability. This mechanism is particularly recommended when the same data is needed across unrelated service boundaries within a single transaction, ensuring consistency and efficiency throughout the request lifecycle.

Caching Performance Optimization Java Spring Framework+3

slackNov 6, 20253 min readCurated summary

Build better software to build software better

Slack’s backend build pipeline for Quip and Slack Canvas once took 60 minutes, delaying feedback and slowing delivery. The team improved build performance by applying familiar software-engineering techniques—caching, parallelization, precise interfaces, and careful decomposition—using Bazel. The central argument is that build systems should be designed like high-performance programs: do less work, distribute unavoidable work, and define work units rigorously. ## Modeling Builds as Dependency Graphs - Applications can be represented as directed acyclic graphs of source files, intermediate artifacts, and deployable outputs. - A backend artifact depends on Python files, while a frontend artifact depends on TypeScript files. - Changing a Python file should rebuild the backend but not unrelated frontend components. - Clearly defined graph nodes allow build systems to optimize work rather than rebuilding everything. ## Caching and Hermetic Work - Caching avoids repeating expensive operations by storing outputs for known inputs. - The article uses a cached recursive `factorial()` function as an analogy: - The input is the cache key. - The return value is the cached artifact. - Effective caching requires work to be: - **Hermetic:** dependent only on explicitly provided inputs. - **Idempotent:** producing the same output for the same inputs. - Cache hit rate matters: poorly defined work units produce more cache misses. ## Granular Cache Units - Caching an entire `process_images(images, transforms)` operation is inefficient because changing one image invalidates the result for every image. - A more granular design caches `process_image(image, transform)` independently. - The higher-level operation can then reuse cached results and process only new image-transform combinations. - Smaller, well-defined units generally improve cache reuse and reduce rebuild time. ## Parallelizing Independent Work - Image processing can also be distributed across CPU threads using `ThreadPoolExecutor`. - Parallel work requires: - Completely specified inputs and outputs. - The ability to transfer data across thread, process, or network boundaries. - Handling completion and failure in any order. - APIs must document ordering guarantees; the threaded example returns images in completion order rather than input order. - Work-unit granularity affects scalability: - Too few large tasks limit available parallelism. - Too many tiny tasks may introduce coordination overhead. - The appropriate balance depends on the workload. ## Applying These Principles to Bazel - Bazel represents builds as directed acyclic graphs made of targets. - Each target defines: - Its input or dependency files. - Its output files. - The commands that transform inputs into outputs. - This structure provides the foundation for caching and parallel execution, just as explicit function inputs and outputs enable those optimizations in application code. The practical recommendation is to design build steps as small, hermetic, idempotent, and independently executable units. Combined with Bazel’s dependency graph, this lets teams avoid unnecessary work, maximize cache hits, and run independent tasks concurrently—turning slow build pipelines into faster sources of developer feedback.

Read original

Caching TypeScript Python Bazel+4

netflixOct 21, 2025Original article

Behind the Streams: Real-Time Recommendations for Live Events Part 3 | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix manages the massive surge of concurrent users during live events by utilizing a hybrid strategy of prefetching and real-time broadcasting to deliver synchronized recommendations. By decoupling data delivery from the live trigger, the system avoids the "thundering herd" effect that would otherwise overwhelm cloud infrastructure during record-breaking broadcasts. This architecture ensures that millions of global devices receive timely updates and visual cues without requiring linear, inefficient scaling of compute resources. ### The Constraint Optimization Problem To maintain a seamless experience, Netflix engineers balance three primary technical constraints: time to update, request throughput, and compute cardinality. * **Time:** The specific duration required to coordinate and push a recommendation update to the entire global fleet. * **Throughput:** The maximum capacity of cloud services to handle incoming requests without service degradation. * **Cardinality:** The variety and complexity of unique requests necessary to serve personalized updates to different user segments. ### Two-Phase Recommendation Delivery The system splits the delivery process into two distinct stages to smooth out traffic spikes and ensure high availability. * **Prefetching Phase:** While members browse the app normally before an event, the system downloads materialized recommendations, metadata, and artwork into the device's local cache. * **Broadcasting Phase:** When the event begins, a low-cardinality "at least once" message is broadcast to all connected devices, triggering them to display the already-cached content instantaneously. * **Traffic Smoothing:** This approach eliminates the need for massive, real-time data fetches at the moment of kickoff, distributing the heavy lifting of data transfer over a longer period. ### Live State Management and UI Synchronization A dedicated Live State Management (LSM) system tracks event schedules in real time to ensure the user interface stays perfectly in sync with the production. * **Dynamic Adjustments:** If a live event is delayed or ends early, the LSM adjusts the broadcast triggers to preserve accuracy and prevent "spoilers" or dead links. * **Visual Cues:** The UI utilizes "Live" badging and dynamic artwork transitions to signal urgency and guide users toward the stream. * **Frictionless Playback:** For members already on a title’s detail page, the system can trigger an automatic transition into the live player the moment the broadcast begins, reducing navigation latency. To support global-scale live events, technical teams should prioritize edge-heavy strategies that pre-position assets on client devices. By shifting from a reactive request-response model to a proactive prefetch-and-trigger model, platforms can maintain high performance and reliability even during the most significant traffic peaks.

Caching Distributed Systems Recommendation Systems Scalability+4

lineJul 18, 2025Original article

Making the Most of Flutter (opens in new tab)

Riverpod is a powerful state management library for Flutter designed to overcome the limitations of its predecessor, Provider, by offering a more flexible and robust framework. By decoupling state from the widget tree and providing built-in support for asynchronous data, it significantly reduces boilerplate code and improves application reliability. Ultimately, it allows developers to focus on logic rather than the complexities of manual state synchronization and resource management. ### Modern State Management Architecture Riverpod introduces a streamlined approach to state by separating the logic into Models, Providers, and Views. Unlike the standard `setState` approach, Riverpod manages the lifecycle of state automatically, ensuring resources are allocated and disposed of efficiently. * **Providers as Logic Hubs:** Providers define how state is built and updated, supporting synchronous data, Futures, and Streams. * **Consumer Widgets:** Views use `ref.watch` to subscribe to data and `ref.read` to trigger actions, creating a clear reactive loop. * **Global Access:** Because providers are not tied to the widget hierarchy, they can be accessed from anywhere in the app without passing context through multiple layers. ### Optimization for Server Data and Asynchronous Logic One of Riverpod's strongest advantages is its native handling of server-side data, which typically requires manual logic in other libraries. It simplifies the user experience during network requests by providing built-in states for loading and error handling. * **Resource Cleanup:** Using `ref.onDispose`, developers can automatically cancel active API calls when a provider is no longer needed, preventing memory leaks and unnecessary network usage. * **State Management Utilities:** It natively supports "pull-to-refresh" functionality through `ref.refresh` and allows for custom data expiration settings. * **AsyncValue Integration:** Riverpod wraps asynchronous data in an `AsyncValue` object, making it easy to check if a provider `hasValue`, `hasError`, or `isLoading` directly within the UI. ### Advanced State Interactions and Caching Beyond basic data fetching, Riverpod allows providers to interact with each other to create complex, reactive workflows. This is particularly useful for features like search filters or multi-layered data displays. * **Cross-Provider Subscriptions:** A provider can "watch" another provider; for example, a `PostList` provider can automatically rebuild itself whenever a `Filter` provider's state changes. * **Strategic Caching:** Developers can implement "instant" page transitions by yielding cached data from a list provider to a detail provider immediately, then updating the UI once the full network request completes. * **Offline-First Capabilities:** By combining local database streams with server-side Futures, Riverpod can display local data first to ensure a seamless user experience regardless of network connectivity. ### Seamless Data Synchronization Maintaining consistency across different screens is simplified through Riverpod's centralized state. When a user interacts with a data point on one screen—such as "starring" a post on a detail page—the change can be propagated globally so that the main list view is updated instantly without additional manual refreshes. This synchronization ensures the UI remains a "single source of truth" across the entire application. For developers building data-intensive Flutter applications, Riverpod is a highly recommended choice. Its ability to handle complex asynchronous states and inter-provider dependencies with minimal code makes it an essential tool for creating scalable, maintainable, and high-performance mobile apps.

Caching State Management Flutter Dependency Injection+3

datadogApr 24, 20183 min readCurated summary

Using Datadog APM to improve the performance of Homebrew

Andrew Robert McBurney describes using Datadog APM to diagnose and optimize Homebrew’s slow `brew linkage` command. Instrumentation identified `LinkageChecker#check_dylibs` as the main bottleneck, and replacing repeated dynamic-library processing with persistent caching reduced execution time from 11.5 seconds to 182 milliseconds for 106 packages. A later implementation used Ruby’s built-in PStore instead of SQLite3 to avoid an additional gem dependency. ## Finding the Bottleneck with APM - Homebrew is widely used at Datadog, so improving its performance provides broad benefits. - The `brew linkage` command checks the library links of installed formulas and can identify when a reinstall is needed. - The target was to scan roughly 50 packages, including large packages such as Boost, in under five seconds. - The author instrumented Homebrew with Datadog’s Ruby `ddtrace` gem. - Flame graphs showed that most execution time was spent in `LinkageChecker#check_dylibs`. ## Why Multithreading Was Not Effective - The author tested Ruby threads as a way to process libraries concurrently. - Ruby’s Global Interpreter Lock limited the achievable parallelism. - Threading failed to meet the required performance target, so a different approach was needed. ## SQLite3-Based Caching - The expensive library-processing results were stored in an on-disk SQLite database. - A `linkage` table recorded: - Formula names and library paths - Linkage categories such as `system_dylibs`, `broken_dylibs`, `undeclared_deps`, and `brewed_dylibs` - Optional labels for selected linkage types - A uniqueness constraint on `(name, path, type, label)` prevented duplicate cache entries. - Homebrew could insert and retrieve linkage data using SQL queries. ## Performance Improvements - Without caching, processing 106 packages took 11.5 seconds. - Boost alone required about 1.01 seconds for dynamic-library checks. - With caching enabled: - The full command completed in 182 milliseconds. - Boost’s check took approximately 1.38 milliseconds. - The cached implementation significantly exceeded the original five-second performance requirement. ## Moving to PStore - After submitting the SQLite3 implementation for review, Homebrew maintainers recommended Ruby’s PStore. - PStore provides file-based persistence built around Ruby’s `Hash` data structure. - Its main advantage is avoiding a third-party SQLite3 gem dependency while preserving the benefits of caching. The central lesson is that profiling should guide optimization: rather than adding ineffective threading, the author located the true bottleneck and achieved a dramatic speedup through persistent caching. For similar command-line performance problems, instrument the complete execution path first, then choose the simplest cache or storage mechanism that satisfies both speed and dependency constraints.

Read original

Caching Datadog Application Performance Monitoring Performance Optimization+4

datadogJun 15, 20173 min readCurated summary

Hackathon project: Viewing Datadog metrics in Minecraft

Datadog engineers used a two-day hackathon to display real-time Datadog metrics inside Minecraft. They connected Minecraft’s Python API with Datadog’s metrics API, then built configurable, live-updating graphs and monitor indicators in the game world. The project demonstrated that even an unconventional visualization environment can be practical to prototype with familiar tools. ## Controlling Minecraft with Python - The team used Raspberry Juice and a Minecraft Pi Edition server to expose Minecraft controls. - They ran the setup on laptops for better performance and faster development. - The `py3minepi` library enabled Python code to create, remove, and query blocks. - Creating a block required only a connection to the server and a call such as `mc.setBlock(...)`. ## Retrieving Datadog Metrics - The Datadog Python library provided access to the Metrics API. - The prototype authenticated with an API key and application key. - It queried recent data, such as average system CPU idle time over the previous five minutes. - The Minecraft and Datadog components were then combined so metric values could be rendered as blocks and structures. - Monitor status indicators changed between green and red depending on whether an alert was active. ## YAML-Based Dashboard Configuration - The team moved dashboard definitions out of Python code into YAML files. - Configuration specified: - Graph position, size, and orientation - Visual properties such as colors, transparency, and borders - Datadog queries and time ranges - Monitor IDs and display locations - This allowed complete dashboards containing multiple graphs and monitor indicators to be updated in real time. ## Handling Minecraft’s Persistence - Minecraft blocks remain in the world after being created, while metric graphs change constantly. - Early experiments left behind random cubes that made the world difficult to navigate. - The team implemented “vacuum” functions to remove everything generated by the visualization code before redrawing it. ## Rendering and Performance Challenges - Without browser technologies such as JavaScript and CSS, graphs had to be reduced to rows of data and represented with Minecraft blocks. - Large graphs could overwhelm the data pipeline. - Caching was added to reduce bandwidth usage and avoid repeatedly requesting the same data. - The performance concerns mirrored Datadog’s everyday engineering work, where caching and efficient data handling are essential. ## Hackathon Experience - The first four hours focused on configuring the environment and connecting the systems. - The remaining time was spent experimenting with building, viewing, destroying, and rebuilding metric displays. - The project’s main value was creative exploration rather than production monitoring. The prototype shows how quickly APIs can be combined to create unusual monitoring interfaces. While Minecraft is not intended to replace conventional dashboards, the project is a playful demonstration of real-time data visualization and rapid experimentation.

Read original

Caching Database Design Datadog Python+3