discord·Mar 4, 2026·2 min readCurated summary

Tracing Discord's Elixir Systems (Without Melting Everything)

Discord runs each guild independently using Elixir’s concurrency model, helping chats and reactions feel instantaneous at scale. When a guild becomes overloaded, metrics and logs can reveal activity spikes but often fail to show the actual user experience or downstream effects. To fill this gap, Discord built distributed tracing for its Elixir services and integrated it without downtime. ## Guild-Level Isolation and Outages - Each Discord server, or “guild,” runs independently from others. - This isolation supports high concurrency and limits failures to individual guilds. - A guild may become laggy or go offline when user activity exceeds its processing capacity. - If it cannot recover automatically, on-call engineers investigate the incident. ## Limits of Metrics and Logs - Engineers inspect metrics showing: - How often each user action type is processed. - How long processing takes. - These metrics can identify bursts of activity, such as sudden waves of reactions or messages. - However, they do not clearly show how those conditions affected users. - Metrics are comparable to a car dashboard: they expose internal conditions but not necessarily the consequences. ## Guild Timings - Discord’s custom “guild timings” tool records the amount of each minute spent processing different action types. - The data is stored in memory and provides more detail than standard metrics. - Its high volume makes long-term storage impractical, so data is frequently rotated. - The tool also focuses on guild-local processing and does not capture downstream effects or complete end-to-end request experience. ## Building Distributed Tracing for Elixir - Distributed tracing shows how long each part of an operation takes across services. - Other Discord teams had already benefited from tracing and application performance monitoring. - Typical tracing systems propagate operation context through metadata such as HTTP headers. - Elixir’s built-in communication mechanisms do not provide an equivalent metadata layer. - Discord therefore built its own mechanism for propagating tracing information between services. ## Deployment Without Downtime - Although the tracing system changed how Discord services communicate, it was integrated without taking the platform offline. - The result gives engineers a more complete view of request paths, helping them understand both the source of guild problems and their impact on users. Discord’s experience suggests that detailed distributed tracing is essential when local metrics and logs cannot explain end-to-end behavior. For highly concurrent systems, investing in tracing infrastructure tailored to the platform can significantly improve incident diagnosis without requiring disruptive deployment changes.

Read original(opens in new tab)

Database DesignApplication Performance MonitoringDistributed TracingElixir+3

gitlab·Mar 4, 2026·4 min readCurated summary

How GitLab built a security control framework from scratch

GitLab created its own security control framework after finding that existing frameworks were too broad, rigid, or insufficiently granular for its multi-product, cloud-native environment. The GitLab Control Framework (GCF) combines industry best practices with product-specific implementations and extensive operational metadata. This lets GitLab manage multiple certifications and internal risks through one scalable framework rather than maintaining separate frameworks for each product. ## Why Existing Frameworks Were Insufficient - GitLab initially used the Secure Controls Framework, then adopted NIST SP 800-53 in preparation for FedRAMP. - NIST’s more than 1,000 controls were comprehensive but included requirements that did not apply to GitLab. - Broad controls often combined several distinct activities: - NIST AC-2, “Account Management,” covers account creation, modification, disabling, termination, shared accounts, and monitoring. - GitLab treated these as separate controls because they have different owners, risks, testing methods, and evidence requirements. - Repeatedly customizing NIST controls effectively meant GitLab was building its own framework, leading to the decision to formalize one. ## Establishing the GitLab Control Framework GitLab developed the GCF through five major steps: ### Assessing Requirements - The team mapped requirements from existing and planned certifications, including: - SOC 2 Type II - ISO 27001, ISO 27017, ISO 27018, and ISO 42001 - PCI DSS - TISAX - Cyber Essentials - FedRAMP - Internal requirements covered mission-critical systems outside certification scopes and systems handling sensitive data. - This analysis established the minimum controls GitLab needed to meet compliance and risk-management obligations. ### Learning from Industry Frameworks - GitLab compared its requirements with: - NIST SP 800-53 - NIST Cybersecurity Framework - Secure Controls Framework - Adobe and Cisco Common Controls Framework - The goal was to reuse proven structures and ensure important security domains and practices were not omitted. ### Creating Custom Domains - The team organized the framework into 18 custom control domains. - Each domain groups related controls according to how GitLab’s security program is managed. - The structure supports adding, changing, or retiring controls as the business evolves. ## Separating Framework Requirements from Implementations GitLab operates several products with different infrastructure and compliance scopes: - GitLab.com is a multi-tenant SaaS platform hosted on GCP. - GitLab Dedicated is single-tenant SaaS hosted on AWS. - GitLab Dedicated for Government is a FedRAMP offering hosted on AWS. To avoid duplicating the framework, the GCF uses two control levels: - **Level 1:** Defines what must be implemented at the organizational framework level. - **Level 2:** Describes how each product fulfills the requirement. - Entity-level controls apply across the organization and are inherited by all product offerings. - This model supports product-specific audits while preserving a single source of control requirements. ## Adding Operational Metadata Rather than tracking only a control ID, description, and owner, the GCF records detailed context for each control: - Responsible owner and risk accountability - Applicable environment or product - Covered assets and systems - Performance or testing frequency - Manual, semi-automated, or automated nature - External certification or internal-risk classification - Testing procedures and required evidence This turns the framework into an operational control inventory. Teams can filter it to identify controls for a particular audit, determine ownership, or find manual controls that may be candidates for automation. ## Designing for Growth - The GCF is intended to evolve with GitLab’s products, risks, and certification goals. - Its structured metadata helps GitLab assess scope and identify gaps when pursuing additional certifications such as ISMAP, IRAP, or C5. - The framework’s modular design makes it easier to extend compliance coverage without creating entirely new control systems. GitLab’s experience suggests that organizations should consider a custom framework when standard frameworks require extensive modification. The most effective approach is to retain useful industry guidance while tailoring control granularity, product implementations, ownership, testing, and metadata to the organization’s actual operating environment.

Read original(opens in new tab)

Database DesignGitlabCloud NativeSecurity Compliance+4

datadog·Mar 4, 2026·3 min readCurated summary

Designing MCP tools for agents: Lessons from building Datadog's MCP server

Datadog’s initial MCP server simply exposed existing APIs, but real-world agent use revealed major problems with context limits, inaccurate trend analysis, and tool overload. The team redesigned its tools around token efficiency, query-based analysis, and a smaller, more deliberate tool surface. These changes improved both answer quality and cost, though emerging agent features may eventually reduce the need for some optimizations. ## Context Efficiency Matters - Observability results can be extremely large: a log record may range from roughly 100 characters to 1 MB. - CSV or TSV is more token-efficient than JSON for tabular data, often using about half as many tokens per record. - YAML can reduce token usage for nested data by around 20% compared with JSON. - Removing rarely used fields from default responses, while allowing agents to request them when needed, further reduces output size. - Combined formatting and field-trimming improvements allowed some tools to return approximately five times more records within the same token budget. - Pagination by record count is unreliable when records vary greatly in size. Datadog instead paginates by token budget and returns a cursor when the limit is reached. - Tools such as Cursor and Claude Code increasingly write long results to disk, which could make response-format efficiency less important in the future. ## Let Agents Query Data - Retrieval-only tools forced agents to infer trends from incomplete samples, such as guessing which services generated the most errors. - Agents sometimes repeatedly fetched logs to compensate, wasting tokens and producing unreliable answers. - SQL lets agents aggregate and filter data directly: ```sql SELECT service, COUNT(*) AS error_count FROM logs WHERE status = 'error' GROUP BY service ORDER BY error_count DESC LIMIT 10 ``` - Agents can select only necessary fields, limit row counts, and calculate aggregates without loading raw data. - SQL improved correctness and reduced costs; some evaluation scenarios became about 40% cheaper. - Supporting SQL at Datadog’s scale required significant infrastructure work because traditional relational databases were insufficient. ## Tools Are Not Free - Exposing every API endpoint as a separate tool increases tool-selection errors and consumes context through tool descriptions. - Flexible tools can support multiple related workflows through carefully designed schemas, reducing the total tool count. - Toolsets provide a core collection by default while allowing users to opt into specialized capabilities, though users must anticipate their needs. - Layered tools can first explain how to accomplish a task and then execute it, keeping specialized functionality out of the initial context. - Layering introduces additional tool calls and therefore increases latency. - Improving agent context management, including tool search and dynamically loaded skills, may reduce the need for aggressive tool minimization over time. The practical recommendation is to design MCP tools for how agents actually reason: minimize and control output size, provide query and aggregation capabilities instead of raw retrieval alone, and expose a focused set of flexible tools rather than mirroring every API endpoint.

Read original(opens in new tab)

Database DesignDatadogObservabilityAPI Design+4

netflix·Mar 3, 2026·3 min readCurated summary

Optimizing Recommendation Systems with JDK’s Vector API

Netflix’s Ranker service used significant CPU for video serendipity scoring, which compares candidate-title embeddings with a member’s viewing history. The team reduced CPU usage by progressively replacing scalar dot products with batched computation, improving memory layout, reusing buffers, and investigating optimized matrix-multiplication libraries. The main lesson was that mathematical optimization alone is insufficient; allocation behavior, cache locality, SIMD support, and runtime overhead all matter. ## The Serendipity Scoring Hotspot - Each candidate title and history item is represented by a vector embedding. - The service computes cosine similarity between every candidate and every history item. - It selects the maximum similarity and converts it into a novelty score: - `serendipity = 1.0 - maxSimilarity` - The original implementation performed `M × N` individual dot products, creating: - Sequential computational work - Repeated embedding lookups - Scattered memory access - Poor cache locality - This logic consumed roughly 7.5% of CPU per Ranker node. - Although 98% of requests contained one video, large batch requests represented about half of the total videos processed. ## Batching Similarity Computations - The team reorganized the calculation as matrix multiplication: - Candidate embeddings form an `M × D` matrix. - History embeddings form an `N × D` matrix. - Rows are normalized to unit length. - Similarities are computed as `C = A × Bᵀ`. - This replaces many separate dot products with one larger operation better suited to CPU-optimized kernels. - The implementation added `batchEncode()` while preserving the existing `encode()` path for single-video requests. ## Why the First Batched Version Regressed - Initial canary tests showed a 5% performance regression. - The batched implementation created `double[][]` arrays for candidates, history, and results on every request. - These allocations: - Increased garbage-collection pressure - Used non-contiguous memory - Added pointer chasing and reduced cache efficiency - The matrix multiplication itself was scalar Java code and did not exploit SIMD hardware. - Batching therefore introduced overhead without delivering corresponding compute gains. ## Flat Buffers and Thread-Local Reuse - The team replaced multidimensional arrays with flat `double[]` buffers in row-major order. - Contiguous storage improved predictability and cache locality. - A `ThreadLocal<BufferHolder>` was used to retain reusable candidate, history, and scratch buffers per thread. - Buffers grow when necessary but do not shrink, avoiding repeated allocations while preventing cross-thread contention. - This reduced GC pressure and made batch performance more stable. ## Evaluating BLAS - BLAS appeared promising in isolated microbenchmarks but did not provide the expected production improvement. - The default `netlib-java` configuration used F2J, a Java implementation rather than truly native BLAS. - Native BLAS introduced setup costs and JNI transition overhead. - Java’s row-major data layout also created an impedance mismatch with common BLAS expectations.

Read original(opens in new tab)

Database DesignJavaCosine SimilarityMatrix Multiply+4

cloudflare·Mar 3, 2026·3 min readCurated summary

Introducing the 2026 Cloudflare Threat Report

Cloudflare’s 2026 Threat Report argues that cyberattacks are shifting from brute-force intrusion toward high-trust exploitation. Attackers increasingly prioritize “Measure of Effectiveness” (MOE)—the greatest operational result for the least effort—using stolen tokens, AI, trusted cloud services, and social engineering rather than costly custom exploits. The report concludes that defenders must focus on identity, integrations, infrastructure resilience, and continuous monitoring of legitimate tools. ## Measure of Effectiveness (MOE) - MOE measures the ratio between an attacker’s effort and the operational outcome. - Threat actors favor: - Stolen session tokens over expensive zero-day exploits. - Reputation-based infrastructure such as LotX over custom servers. - AI-assisted automation over manually written tooling. - The most dangerous actors are those able to combine intelligence and technology into continuous, high-speed operations. ## Eight trends shaping the 2026 threat landscape - **AI-driven attacker operations** - Generative AI supports real-time network mapping, exploit development, and deepfake creation. - Lower-skilled attackers can now conduct more sophisticated, high-impact campaigns. - **State-sponsored infrastructure pre-positioning** - Groups such as Salt Typhoon and Linen Typhoon are targeting North American telecommunications, government, commercial, and IT services. - Their goal is to maintain access that can provide long-term geopolitical leverage. - **Over-privileged SaaS integrations** - Third-party APIs can expand a single compromise across hundreds of organizations. - The GRUB1 breach of Salesloft demonstrates the risks created by excessive integration privileges. - **Weaponized trusted cloud tools** - Attackers use services such as Google Calendar, Dropbox, GitHub, Google Drive, Microsoft Teams, and Amazon S3 to conceal malicious activity. - Legitimate enterprise traffic makes command-and-control communications harder to distinguish from normal use. - **Deepfake-based insider placement** - North Korean operators are using fraudulent identities and deepfakes to place remote IT workers inside Western companies. - These operatives support espionage and illicit revenue generation. - **Session-token theft** - Infostealers such as LummaC2 harvest active authentication tokens. - Attackers can then bypass multi-factor authentication and begin post-authentication activity. - **Internal brand spoofing** - Phishing-as-a-service tools exploit mail-relay blind spots where sender identity is not re-verified. - This enables convincing impersonation messages to arrive directly in trusted user inboxes. - **Hyper-volumetric DDoS attacks** - Botnets such as Aisuru are generating increasingly large distributed denial-of-service attacks. - The speed and scale of these attacks can overwhelm infrastructure before human responders can react. ## Living off legitimate cloud infrastructure - Attackers increasingly avoid known malicious servers and instead use legitimate SaaS, IaaS, and PaaS platforms. - Cloud services can be used to host payloads, redirect victims, deliver malware, or scale campaigns. - Amazon SES and SendGrid, for example, can be abused for phishing and malware distribution. - This “living off the land” approach—or “living off anything-as-a-service”—allows attackers to hide behind the reputation and normal traffic patterns of trusted providers. - Cloud-resource abuse is evolving from opportunistic infrastructure misuse into a deliberate nation-state strategy. Defenders should treat identity tokens, SaaS permissions, cloud activity, and trusted integrations as critical security boundaries. Organizations need least-privilege access, stronger token protection, continuous monitoring, automated DDoS mitigation, and detection that evaluates behavior—not just whether a service is legitimate.

Read original(opens in new tab)

Database DesignAISaasDdos+3

cloudflare·Mar 3, 2026·3 min readCurated summary

Evolving Cloudflare’s Threat Intelligence Platform: actionable, scalable, and ETL-less

Cloudflare’s Threat Intelligence Platform (TIP) is designed to turn massive volumes of security telemetry into actionable intelligence without relying on traditional ETL pipelines. Its sharded, SQLite-backed architecture uses Durable Objects and edge-based GraphQL to provide near-real-time analysis across millions of events. By combining automated telemetry with analyst investigations, the platform aims to help security teams understand threats and block them proactively. ## Motivation for Building the Platform - Cloudflare began developing the TIP after launching Cloudforce One in 2022 and discovering that existing tools could not adequately track adversary infrastructure. - The platform models the full threat lifecycle, connecting: - Threat actors to malware - Cases to indicators - Events to broader campaigns - It is designed for: - Multiple datasets and tenants - Group-based and tenant-to-tenant sharing - Extensibility and edge-scale performance - Visual analysis and automated response - Cloudflare Workers allow the platform to evolve with the runtime and support features such as Smart Placement, higher CPU limits, and Hyperdrive. ## Beyond the SIEM - The TIP complements rather than replaces a SIEM: - SIEMs focus on real-time log aggregation and alerting. - The TIP provides long-term retention, specialized threat schemas, and historical context. - Analysts can enrich alerts with: - Indicator history - Known threat-actor associations - Campaign relationships - Risk scores and intelligence context - Findings from analysts feed new indicators of compromise back into the platform. - This feedback loop keeps intelligence current and helps organizations move from reactive investigation to proactive defense. ## Sharded Storage Without ETL Bottlenecks - Cloudflare distributes Threat Events across many logical shards instead of using one centralized database. - Each shard is a Durable Object with a private SQLite database, providing transactional consistency and avoiding a single database bottleneck. - Cloudflare Queues handle asynchronous ingestion, helping absorb high-volume attack spikes. - R2 stores data for long-term retention, while SQLite maintains a hot index for fast access. - Because data is available directly in the platform’s operational store, complex ETL pipelines and synchronization delays are avoided. ## Parallel Queries at the Edge - GraphQL runs in the same Worker-based system that powers the Threat Events platform, keeping data live from ingestion through querying. - Queries are fanned out to relevant Durable Objects in parallel rather than executed against one large table. - The platform first verifies permissions and excludes shards that cannot contain matching events, such as shards outside the requested date range. - Results from multiple shards are aggregated with `Promise.all`, enabling low-latency searches across global datasets. - Smart Placement positions query Workers near the Durable Objects they access, reducing tail latency. Cloudflare’s approach combines edge-native storage, parallel execution, and analyst-driven enrichment to make threat intelligence both scalable and actionable. The practical goal is a unified system that explains not only what is malicious, but also why it matters and how to automatically prevent it.

Read original(opens in new tab)

Database DesignCloudflare WorkersGraphQLSqlite+4

cloudflare·Mar 3, 2026·3 min readCurated summary

See risk, fix risk: introducing Remediation in Cloudflare CASB

Cloudflare CASB now lets customers remediate risky SaaS file-sharing directly from the Cloudflare One dashboard, rather than merely identifying problems. The initial release targets Microsoft 365 and Google Workspace, removing public, organization-wide, or external sharing without deleting files or changing ownership. Cloudflare concludes that durable, workflow-based execution makes remediation scalable, observable, and easier to operate. ## CASB as a Centralized SaaS Risk View - CASB connects to services including Microsoft 365, Google Workspace, Slack, Salesforce, Box, GitHub, Jira, and Confluence through APIs. - It provides: - A consolidated view of misconfigurations, overshared files, and risky access. - Continuous scanning as users collaborate and adopt new tools. - Searchable and exportable findings for triage and reporting. - Previously, fixing findings required using each application’s admin interface or submitting tickets to application owners. ## File-Sharing Remediation - The new **Remove sharing** action can address: - Public links that allow anyone to view or edit files. - Company-wide sharing when only a few users need access. - Sharing with external domains or personal accounts. - Any of these risks involving files that match a DLP profile, such as customer records, credentials, or financial data. - Remediation removes the risky sharing configuration only: - Files are not deleted. - Ownership is not changed. - Progress and outcomes are tracked in CASB, while actions are recorded in Cloudflare One Admin logs and can be exported to a SIEM. ## Microsoft 365 and Google Workspace - The initial integrations focus on business-critical documents stored in: - OneDrive and SharePoint. - Google Drive, including Docs, Sheets, and Slides. - Common examples include temporary public editing links, company-wide documents forgotten after an event, and sensitive spreadsheets shared with contractors’ personal accounts. - Teams can now resolve findings directly in CASB instead of exporting CSVs and relying on application owners to make changes. ## Durable Remediation Architecture - Cloudflare designed the system for speed, resilience, and ease of use using: - Workers - Workflows - Queues - Workers KV - Secrets Store - Hyperdrive - The process is: - An API call sends a remediation job to a Worker. - The Worker places it on a Queue. - A second Worker starts a Workflow. - Credentials are securely provided through Workers KV and Secrets Store. - The Workflow gathers information and calls third-party APIs. - Hyperdrive records the final result. - Workflows’ native retries handle vendor API rate limits such as HTTP 429 responses, while built-in step logging shows retry activity. - Load testing and early customer usage produced a median completion time of 48 seconds and a p90 of 72 seconds. ## Planned Expansion - Cloudflare plans to add: - Quarantine actions that move or isolate high-risk files. - Custom Webhooks for ticketing, chat notifications, and external automation. - Carefully scoped autoremediation policies. - Custom CASB findings based on organization-specific patterns, data types, or access conditions. Organizations using Microsoft 365 or Google Workspace can use CASB Remediation to turn detected sharing risks into tracked, auditable fixes. The planned quarantine, webhook, and automated-policy features could further position CASB as an active security control plane rather than a passive reporting tool.

Read original(opens in new tab)

Database DesignCloudflareAPISaas+4

netflix·Feb 28, 2026·3 min readCurated summary

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs

Netflix’s effort to modernize its container runtime exposed a hardware-level bottleneck rather than an application problem. Under heavy startup concurrency, containers with many image layers triggered massive mount and unmount activity, causing kernel lock contention, systemd stalls, and container startup failures. The issue was especially severe on older dual-socket NUMA instances, while newer single-socket systems scaled much more reliably. ## Container Startup at Netflix - New AWS capacity is rapidly filled with pods as applications scale. - Some nodes became unresponsive, with: - Health checks timing out for more than 30 seconds - Kubelet requests to containerd timing out - systemd processing huge numbers of mount events - The mount table taking tens of seconds to read - The problem primarily affected `r5.metal` instances running images with more than 50 layers. ## Mount Lock Contention - With user namespaces, containerd performs several mount operations for every image layer: - `open_tree()` references the layer. - `mount_setattr()` applies the container’s ID mapping. - `move_mount()` creates an ID-mapped bind mount. - These bind mounts become OverlayFS lower directories and are later unmounted. - The Linux VFS uses global mount-related locks, so concurrent container creation causes CPUs to contend on the same kernel locks. - For 100 containers with 50 layers each, containerd performs the process twice: - `100 × 2 × (1 + 50 + 50) = 20,200` mount operations - This makes startup cost depend heavily on both container concurrency and image layer count. ## Why the New Runtime Exposed the Problem - The old Docker-based runtime shifted file ownership while unpacking images. - All containers shared one host user range, avoiding repeated per-container mount work. - The new containerd-based runtime assigns each container a unique host user range for stronger isolation. - Instead of rewriting file ownership during extraction, it uses Linux ID-mapped mounts to apply ownership mappings efficiently. - This improves security and avoids expensive image copying, but creates many additional mount operations during startup. ## Differences Between AWS Instance Types Netflix compared: - `r5.metal`: 5th-generation Intel, dual-socket, multiple NUMA domains - `m7i.metal-24xl`: 7th-generation Intel, single-socket, single NUMA domain - `m7a.24xlarge`: 7th-generation AMD, single-socket, single NUMA domain Results showed: - At low concurrency—around 20 containers or fewer—all systems performed similarly. - `r5.metal` began failing at roughly 100 concurrent container launches. - Newer Intel instances maintained lower startup times and better success rates. - AMD-based `m7a` instances scaled most consistently and had the fewest failures. ## Kernel and CPU-Level Diagnosis - Profiling showed that containerd spent most of its time in Linux VFS path lookup code. - Specifically, threads were spinning in `path_init()` while waiting on a sequence lock. - Intel Topdown Microarchitecture Analysis found: - 95.5% of pipeline slots stalled on contested accesses - 57% attributed to false sharing - Cache-line bouncing and global lock contention, rather than raw CPU capacity, dominated performance. ## NUMA as a Contributing Factor - NUMA systems divide memory among processor sockets. - Local memory access is faster, while remote access crosses an interconnect and introduces additional latency. - The dual-socket layout of `r5.metal` amplified contention around shared mount-related data. - The better behavior of newer single-socket instances indicated that CPU topology and memory locality were key contributors to the container startup bottleneck. ## Practical Conclusion High-concurrency container launches can overwhelm kernel mount infrastructure, especially when using per-container ID mapping and images with many layers. Netflix’s results suggest minimizing image layers, controlling startup concurrency, and favoring newer single-socket hardware can substantially improve reliability and scaling.

Read original(opens in new tab)

Database DesignKubernetesAWSDocker & Containers+4

netflix·Feb 24, 2026·3 min readCurated summary

MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

Netflix’s Media Foundational Model (MediaFM) is a tri-modal AI system that combines video, audio, and timed text to understand long-form entertainment. It represents sequences of shots while using title-level metadata and temporal context to produce richer content embeddings. Netflix concludes that these contextual embeddings improve many downstream tasks, including advertising relevance, clip selection, tone classification, and popularity prediction. ## Motivation for MediaFM - Netflix needs machine-readable understanding of its expanding catalog, including films, series, live events, and podcasts. - Long-form media requires recognizing narrative dependencies, emotional arcs, scene transitions, and subtle tones across entire episodes or films. - Combining visual, audio, and textual signals provides a more complete understanding than relying on video alone. - The resulting embeddings support applications such as: - Cold-start recommendations for new titles - Promotional art and trailer optimization - Advertising relevance - Clip tagging and internal content analysis ## Multimodal Input Representation - The model uses a shot as its fundamental unit, with titles segmented using shot-boundary detection. - Each shot receives three modality-specific embeddings: - **Video:** Frames sampled from the shot are encoded with SeqCLIP, Netflix’s video-retrieval model. - **Audio:** Sound is encoded using Meta FAIR’s wav2vec2. - **Timed text:** Captions, subtitles, or audio descriptions are encoded with OpenAI’s `text-embedding-3-large`. - The three embeddings are concatenated and unit-normalized into a 2,304-dimensional fused vector. - Training examples consist of temporally ordered shot sequences from a movie or episode, with up to 512 shots. - Title metadata, such as synopses and tags, is also embedded and supplied as global context. ## Transformer Architecture - MediaFM uses a BERT-like Transformer encoder. - Fused shot embeddings are first projected into the model’s hidden dimension. - Two special tokens are prepended: - `[CLS]`, a learnable sequence-level embedding - `[GLOBAL]`, containing projected title-level metadata - Positional embeddings and self-attention allow each shot representation to incorporate surrounding narrative context. - A final projection maps contextualized representations back into the original 2,304-dimensional embedding space. ## Masked Shot Modeling - The model masks 20% of shot embeddings in each training sequence. - Masked inputs are replaced with a learnable `[MASK]` embedding. - The Transformer must reconstruct the original fused embedding for each masked shot. - Training minimizes cosine distance between predicted and ground-truth embeddings. - Hidden parameters are optimized with Muon, while other parameters use AdamW; Netflix reports noticeable gains after adopting Muon. ## Evaluation Through Linear Probes - Netflix evaluates MediaFM by freezing its representations and training task-specific linear layers on top. - Most evaluation tasks involve short clips extracted from larger titles. - Embedding a clip within the context of its surrounding episode or film performs better than embedding the clip in isolation, demonstrating the value of long-range contextualization. ## Downstream Applications - **Ad relevancy:** Multilabel classification identifies clips suitable for relevant advertising; MediaFM helps retrieve candidate clips before ad-serving optimization. - **Clip popularity ranking:** The model predicts relative clip performance and click-through rate within a title, evaluated using Kendall’s tau. - **Clip tone:** Clips are classified into 100 categories, such as creepy, scary, or humorous. - **Clip genre:** Clips are assigned to core genres including Action, Comedy, Documentary, Drama, Horror, Romance, and Thriller. - **Clip retrieval:** The system distinguishes “clip-worthy” content from unsuitable clips based on human annotations, using Average Precision. MediaFM’s main practical lesson is that effective media understanding depends on fusing all available modalities and preserving long-form temporal context. Netflix’s approach provides a reusable embedding foundation for recommendation, promotion, advertising, and content-analysis systems rather than building a separate representation for every task.

Read original(opens in new tab)

Database DesignTransformerMultimodalVideo Embedding+4

gitlab·Feb 19, 2026Original article

GitLab Threat Intelligence Team reveals North Korean tradecraft (opens in new tab)

The GitLab Threat Intelligence Team has detailed its efforts to disrupt North Korean (DPRK) cyber campaigns, specifically focusing on "Contagious Interview" malware distribution and fraudulent IT worker schemes. By analyzing internal platform data, GitLab identified that these state-sponsored actors leverage legitimate tools and fake recruitment scenarios to compromise software developers and generate illicit revenue for the regime. The report concludes that while these operations are sophisticated and persistent, proactive monitoring and cross-industry intelligence sharing are essential to mitigating these evolving threats. ### Contagious Interview Mechanics * Threat actors pose as recruiters to trick software developers into executing malicious JavaScript projects under the guise of technical interviews. * The primary goal is to deploy malware families such as BeaverTail and Ottercookie, which facilitate credential theft and provide remote control of the victim's device. * A notable evolution in tradecraft includes the use of "ClickFix," a compiled BeaverTail variant identified in late 2025. * Malicious repositories often use a specific execution pattern where base64-encoded URLs and secret headers are hidden within `.env` files, masquerading as benign configuration variables. * To execute the payload, actors utilize `Function.constructor` to load strings as executable code, often triggered by custom error handlers designed to source remote content. ### 2025 Campaign Trends and Infrastructure * GitLab banned 131 unique accounts linked to these campaigns in 2025, with activity peaking in September and averaging 11 bans per month. * Nearly 90% of malicious accounts were created using Gmail addresses, and actors typically accessed the platform through consumer VPNs or dedicated VPS infrastructure. * In more than 80% of cases, malware payloads were not stored on GitLab. Instead, actors used concealed loaders to fetch content from legitimate hosting services, most commonly Vercel. * Recent tactics include the creation of malicious NPM dependencies immediately before use and the exploitation of VS Code tasks to pipe remote content into native shells. ### IT Worker Campaigns and Sanctions Evasion * Beyond malware distribution, DPRK actors use GitLab to support "IT worker" cells that generate revenue and evade international sanctions. * One identified pipeline involved the creation of at least 135 synthetic identities, automated to generate professional connections and contact leads at scale. * Threat actors have been observed adding their own images to stolen U.S. identity documents to bypass employment verification processes. * Forensic analysis revealed financial records from cell managers detailing revenue proceeds from 2022 through 2025, often earned while operating from locations like Moscow, Russia. Organizations should remain vigilant against recruitment-themed social engineering and scrutinize unexpected requests to run external code. GitLab recommends that the security community use the provided indicators of compromise to update defensive posture, as these actors continue to refine their ability to hide malicious intent within legitimate development workflows.

Database DesignGitlabJavaScriptNpm+3

spotify·Feb 19, 2026·4 min readCurated summary

Inside the Archive: The Tech Behind Your 2025 Wrapped Highlights | Spotify Engineering

Spotify’s 2025 Wrapped Archive identified up to five remarkable listening days for each eligible user and turned them into personalized, LLM-generated stories. A distributed pipeline, carefully designed prompts, model distillation, and massive-scale pre-generation made it possible to create roughly 1.4 billion reports before launch. The system prioritized factual grounding, creative consistency, safety, and reliable parallel storage. ## Identifying Remarkable Listening Days - Spotify used a priority-ordered set of heuristics to evaluate each user’s full year of listening. - Straightforward categories included: - Biggest Music Listening Day - Biggest Podcast Listening Day - Biggest Discovery Day, based on first-time artists - Biggest Top Artist Day - Biggest Top Genre Day - More nuanced categories detected: - Nostalgic listening and throwback-heavy sessions - Unusual listening patterns that differed from a user’s typical taste - Contextual dates such as birthdays and New Year’s Day - Candidate days were ranked by narrative potential and statistical strength, reducing hundreds of millions of events to as many as five standout days per user. - A distributed data pipeline aggregated the results and stored listening data in object storage. - Messaging queues then moved each user’s data asynchronously into report generation. ## Prompt Engineering for Reliable Stories - Spotify spent more than three months iterating on prompts and evaluating edge cases. - The system prompt established: - Traceability to real listening behavior - A witty, sincere, and quietly playful tone - Safety constraints excluding references to drugs, alcohol, sex, violence, and offensive language - User prompts supplied: - Detailed daily listening logs - Precomputed statistics, since LLMs are unreliable at arithmetic - Overall Wrapped data - The remarkable-day category - Previously generated reports to reduce repetition - The user’s country for appropriate spelling and vocabulary - Outputs were improved through prototype comparisons, LLM-based judging, human review, and feedback from creative, technical, and safety teams. ## Distilling the Model for Scale - Larger frontier models produced strong results during prototyping but were too expensive for more than a billion generations. - Spotify generated high-quality reference outputs and curated them into a reviewed “gold” dataset. - A smaller, faster production model was fine-tuned on that dataset. - Direct Preference Optimization (DPO), based on curated human A/B evaluations, further aligned the smaller model with the preferred output style. - The resulting model achieved preference performance comparable to the larger baseline. ## Generating 1.4 Billion Reports - Approximately 350 million users were eligible, with up to five reports each. - Spotify pre-generated about 1.4 billion reports before Wrapped launch. - The system sustained thousands of model requests per second over several days. - After remarkable days were computed, snapshots were published to a pub/sub queue. - Reports were generated sequentially per user so earlier reports could inform later ones and prevent repetition. - Real-time dashboards tracked throughput, reliability, errors, and projected completion time. - The generation engine ran continuously for four days, followed by checks for missing reports, inconsistencies, and necessary re-generation. ## Designing Storage for Concurrent Writes - Completed reports were stored in a distributed, column-oriented key-value database optimized for high-throughput writes. - Each user occupied a single row, with separate columns representing completed remarkable days. - Instead of maintaining a serialized list—which could cause race conditions during read-modify-write operations—each date received its own column qualifier in `YYYYMMDD` format. - Independent reports could therefore be written concurrently to separate cells without locks or coordination. - Report content was written first, followed by lightweight metadata marking the report complete. - This ordering prevented the system from exposing a completion marker before the underlying report was safely stored. ## Practical Conclusion Building Wrapped Archive required treating creative AI generation as a large-scale production system: ground outputs in structured data, use smaller specialized models when volume demands it, evaluate continuously, and design storage schemas that make concurrency safe by default.

Read original(opens in new tab)

Database DesignLarge Language ModelsPrompt EngineeringHigh Throughput+4

gitlab·Feb 17, 2026Original article

Claude Opus 4.6 now available in GitLab Duo Agent Platform (opens in new tab)

GitLab has integrated Anthropic’s Claude Opus 4.6 into its Duo Agent Platform, providing developers with a high-intelligence frontier model designed for complex agentic workflows. By combining a 1-million-token context window with native access to DevSecOps data, the update enables more autonomous task execution and deeper reasoning within the software development lifecycle. This integration allows teams to delegate multi-step tasks to AI agents that can now process entire codebases and project histories in a single interaction. ## Advanced Agentic Capabilities and Reasoning * Claude Opus 4.6 features enhanced "agentic" behavior, meaning it can proactively take actions and drive tasks forward with minimal human intervention. * The model supports multi-agent orchestration, allowing it to spin up subagents and coordinate parallel workstreams to solve complex, multi-step problems. * Adaptive thinking capabilities allow the model to calibrate its reasoning depth based on the query, using extended thinking for difficult tasks while maintaining speed for simpler ones. * Deep reasoning via test-time compute helps the model navigate challenging development bottlenecks and architectural decisions. ## Full-Context DevSecOps Integration * The model boasts a 1-million-token context window—a fivefold increase over Opus 4.5—enabling the processing of massive codebases and extensive documentation. * Integration with the GitLab Duo Agent Platform provides the model with direct access to repositories, merge requests, pipelines, and security findings. * Enterprise-grade security features, including human-in-the-loop controls and group-based access, ensure that agentic actions remain transparent and governed. * Native integration ensures developers can utilize these frontier capabilities without leaving their established GitLab workflows. ## Availability and Resource Consumption * Opus 4.6 is currently available for GitLab.com users via the Duo Agent Platform and Agentic Chat, though it is not supported for GitLab Duo Classic features. * Support for the model within various Integrated Development Environments (IDEs) is expected to be released in the near future. * Usage is managed via GitLab credits, with multipliers determined by the size of the prompt. * Prompts containing 200k tokens or fewer are charged at 1.2 requests per credit, while larger prompts exceeding 200k tokens are charged at 0.7 requests per credit. Organizations aiming to automate complex development workstreams should migrate their specialized agents to Claude Opus 4.6 to take advantage of its superior orchestration and context handling. By leveraging the model's ability to coordinate parallel subagents, teams can significantly reduce the manual effort required for codebase-wide refactors and security remediation.

Database DesignMachine LearningGitlabDevSecOps+3

netflix·Feb 13, 2026·4 min readCurated summary

Scaling LLM Post-Training at Netflix

Netflix argues that LLM post-training at production scale is as much an infrastructure challenge as a modeling challenge. Its internal framework abstracts distributed data processing, model sharding, GPU orchestration, checkpointing, and complex training workflows so developers can focus on experimentation. The result is a flexible system supporting SFT, DPO, reinforcement learning, and knowledge distillation across hundreds of GPUs. ## Why Post-Training Becomes an Engineering Problem - Pre-training provides general language ability, but post-training adapts models to Netflix’s catalog, member histories, recommendation tasks, personalization, and search. - Production-scale training introduces challenges involving: - Large proprietary datasets - Multi-node GPU coordination - Distributed model state - Workflows that combine training and inference - Failure recovery and experiment tracking - A simple Hugging Face fine-tuning script is insufficient for reliable, large-scale jobs. ## Preparing Data Correctly - Chat templates serialize conversations but do not determine which tokens should contribute to the loss. - Netflix applies explicit loss masking so training focuses on assistant responses rather than prompts or other non-target text. - Variable-length examples can waste GPU memory through padding and create synchronization overhead across FSDP workers. - Sequence packing combines multiple samples into fixed-length sequences. - A document mask prevents attention across separately packed samples while improving GPU utilization. ## Loading and Optimizing Large Models - Models that do not fit on one GPU require sharding strategies such as FSDP or tensor parallelism. - Partial weights should be loaded directly onto the device mesh rather than materializing the entire checkpoint on a single device. - Developers can choose full fine-tuning or LoRA and use: - Activation checkpointing - Compilation - Appropriate precision settings - Reinforcement learning requires compatible precision between rollout generation and policy training. - Large vocabularies create memory pressure because logits have dimensions `[batch, seq_len, vocab]`. - The framework reduces peak memory by removing ignored tokens before projection and computing logits and loss in sequence chunks. ## Distributed Training and Workflow Management - The framework supports standard forward/backward training for SFT as well as workflows that interleave: - Rollout generation - Reward-model and reference-model inference - Policy updates - Ray actors orchestrate distributed jobs while keeping hardware concerns separate from modeling code. - Experiment tracking covers both quality metrics, such as loss, and efficiency metrics, such as Model FLOPS Utilization (MFU). - Standardized checkpointing allows jobs to resume after failures. ## Netflix’s Post-Training Framework - The stack is built on: - Mako for AWS GPU provisioning - PyTorch, Ray, and vLLM - Netflix’s framework library for reusable utilities and training recipes - Jobs are generally defined through configuration files that select a recipe and provide task-specific components. - Unlike narrower fine-tuning systems, the framework supports: - Custom output heads - Expanded vocabularies and semantic IDs - Special tokens - Transformer models trained on non-natural-language sequences - This flexibility is important for Netflix-specific recommendation and personalization use cases. ## Four Core Abstractions ### Data - Dataset abstractions cover SFT, reward modeling, and RL. - Streaming supports datasets larger than local disk capacity. - Asynchronous sequence packing overlaps CPU preprocessing with GPU execution to reduce idle time. ### Model - The framework supports architectures such as Qwen3 and Gemma3, including Mixture-of-Experts variants. - LoRA is integrated into model definitions. - High-level sharding APIs distribute models across device meshes without requiring developers to write low-level distributed code. ### Compute - A unified job interface scales from one node to hundreds of GPUs. - MFU measurement remains accurate for custom architectures and LoRA configurations. - Checkpoints include parameters, optimizer state, dataloader state, and data-mixer state, enabling exact resumption. ### Workflow - The system supports SFT, DPO, RL, and knowledge distillation. - Online RL uses a hybrid architecture combining a single controller with Single Program, Multiple Data (SPMD) workers. - This extends conventional SPMD training to multi-stage workflows that cannot be represented as a simple training loop. Netflix’s approach is to standardize the difficult operational parts of post-training while preserving enough flexibility for unconventional models and objectives. A framework built around reusable data, model, compute, and workflow abstractions can help teams iterate faster and scale experiments without repeatedly rebuilding distributed infrastructure.

Read original(opens in new tab)

Database DesignLarge Language ModelsCloud ComputingPytorch+4

line·Feb 13, 2026·3 min readCurated summary

Claude Code Action: Platformizing AI Code

LINE NEXT transformed Claude Code from an individual productivity tool into an organization-wide code review platform integrated with GitHub Actions. The goal was to reduce review-quality variation, standardize policies, and make AI feedback part of the existing pull request workflow. Its central design separates simple repository-level invocation from centrally managed execution, prompts, permissions, and infrastructure. ## Why AI Code Review Needed to Be Platformized - As LINE NEXT’s services and repositories grew, human code review quality varied according to each reviewer’s experience and preferences. - Developers were already using Claude Code locally, but individual usage created several problems: - Inconsistent review criteria and perspectives - No organization-wide quality process - AI feedback disconnected from pull request workflows - Difficulty providing new employees with a consistent review experience - DevOps therefore treated the issue as a decentralized quality-process problem rather than merely a tooling problem. ## Why GitHub Actions and Claude Code - GitHub Actions was already the foundation for CI/CD and automation across LINE NEXT repositories. - It allowed the team to: - Apply a common workflow repository by repository - Centrally manage execution environments and permissions - Avoid requiring each service team to build additional infrastructure - Claude Code Action integrated directly with pull requests: - Developers could trigger reviews with an `@claude` mention. - Results appeared as GitHub comments or PR reviews. - Developers did not need to learn a separate interface. - A shared GitHub App Runner environment provided consistent execution and centralized security controls. ## Centralized Caller–Executor Architecture - Service repositories act as **callers**: - They invoke the standard workflow. - They provide only basic parameters such as service name and review type. - A centrally managed DevOps repository acts as the **executor**: - Stores prompts and review personas - Defines review policies and priorities - Manages permissions and authentication - Contains the actual execution logic - This design makes AI review an organization-wide platform capability rather than a separate configuration maintained by every project. ### Benefits of Central Control - **Consistent quality:** Central prompts and personas ensure common review depth, tone, security checks, stability checks, and priorities. - **Faster adoption:** New repositories need only add the standard workflow and specify a few parameters. - **Improved governance:** GitHub Apps, centrally managed secrets, and shared runners make it possible to track who accessed which code and with what permissions. - **Lower operational overhead:** Service teams use the platform without managing AI infrastructure themselves. ## Handling Fork-Based Pull Requests - The official Claude Code Action initially assumed that a PR branch existed in the base repository’s `origin`. - For pull requests created from forks, this caused failures such as: ```text couldn't find remote ref ``` - The original implementation fetched and checked out the branch by name: ```text git fetch origin <branch> git checkout <branch> ``` - This failed because fork branches exist in the external repository, not necessarily in the base repository. - From a platform perspective, this was a structural limitation because it blocked external contributors and collaboration repositories. - The proposed direction was to redesign the execution flow rather than simply add an exception, using GitHub’s special pull-request reference: ```text refs/pull/<PR number>/head ``` This approach allows the workflow to retrieve the actual pull request head commit regardless of whether the PR originated from the main repository or a fork.

Read original(opens in new tab)

Database DesignKubernetesCI/CDClaude Code+4

line·Feb 13, 2026·3 min readCurated summary

Slow Query Resolution: Optimizing Bit

LINE VOOM’s post server experienced intermittent timeouts when loading profiles belonging to users with hundreds of thousands of posts. The root cause was bitwise filtering on `category_flag` and `access_flag`, which prevented MySQL from efficiently using indexes and forced scans of all posts for a user. The team resolved the issue with MySQL 8.0.13 functional indexes and by changing the query predicates to exact decimal comparisons, reducing scanned rows from 805 to 31 in testing. ## The Slow Query and Its Root Cause - Post metadata was distributed across shards and partitioned tables. - `category_flag` and `access_flag` were stored as `bit(64)` values containing multiple status flags. - The problematic query filtered by: - `user_id` - `category_flag & 0x0100` - `access_flag & 0x0001` - For heavy users, the query scanned hundreds of thousands of posts and ran for more than 30 seconds. - Bitwise expressions operated on computed results rather than raw column values, preventing normal indexes from filtering efficiently. ## Choosing Functional Indexes - The team considered hardware upgrades, caching, and additional partitioning, but none addressed the root cause adequately. - MySQL 8.0.13 functional indexes could index expression results without changing the table schema. - The proposed composite index was: ```sql ALTER TABLE post_metadata ADD INDEX idx_user_premium_searchable ( user_id, (category_flag & 0x0100), (access_flag & 0x0001) ); ``` - Functional indexes rely on the query expression matching the index definition precisely. ## Discovering the Required Query Form - Initial attempts failed to use the index: - Truthy checks such as `category_flag & 0x0100` - Comparisons using `> 0` - Equality against hexadecimal values such as `= 0x0100` - The successful form used decimal equality: ```sql WHERE user_id = '{user_id}' AND (category_flag & 0x0100) = 256 AND (access_flag & 0x0001) = 1 ``` - In testing, scanned rows dropped from 805 to 31. - Index storage increased by approximately 24%, but the DBA team determined that production capacity was sufficient. ## Rolling Out the Indexes in Production - Indexes were created before changing the application queries. - The team used online schema changes to avoid service downtime and support pausing or rollback during replication problems. - Because dozens of tables across multiple shards were affected: - One shard was handled first for validation. - Only one or two tables were processed per day. - Work was avoided during periods when emergency DBA support was unavailable. - Index creation increased replication lag, causing newly created posts to temporarily disappear from read replicas. - The team reduced the cache expiration time for the affected post lists and accepted the remaining replication delay before resuming the rollout. ## Gradual Query Deployment and a Bitwise Logic Bug - Query changes were deployed gradually through a dynamic configuration system. - Each query pattern was tested on one shard before being expanded to the remaining shards. - This allowed changes to be rolled back immediately through configuration. - During rollout, a serious visibility bug was found. - The original condition: ```sql category_flag & 0x0110 ``` matched when either `0x0100` or `0x0010` was present, effectively representing an OR condition. - Rewriting it as: ```sql (category_flag & 0x0110) = 272 ``` required both bits to be set, creating an AND condition. - Because production data stored only the premium bit, some profiles returned no content. - The incident highlighted the need to verify the semantic meaning of bit flags before converting bitwise predicates into equality comparisons. ## Practical Recommendation For slow queries involving bit flags, consider functional indexes when using MySQL 8.0.13 or later. Ensure the query expression exactly matches the index definition, validate bitwise logic carefully, and use staged schema and query rollouts with monitoring and fast rollback mechanisms.

Read original(opens in new tab)

Database DesignMysqlCachingSharding+4