cloudflare·Mar 9, 2026·3 min readCurated summary

Complexity is a choice. SASE migrations shouldn’t take years.

Cloudflare argues that SASE and zero trust migrations do not need to take years. Its partners, TachTech and Adapture, reportedly reduced deployments from around 18 months to four–six weeks by using Cloudflare One’s unified, cloud-native architecture. The post concludes that programmable security infrastructure can accelerate zero trust adoption while also enabling safer use of AI. ## Faster Zero Trust Deployments - Traditional Secure Web Gateway (SWG) and Zero Trust Network Access (ZTNA) migrations can take up to 18 months for large organizations. - TachTech reduced comparable Cloudflare One deployments to four–six weeks. - Cloudflare Access is presented as lightweight and largely “no-touch” after deployment, reducing ongoing operational effort. ## Why Legacy Migrations Stall - Legacy architectures often treat migration as hardware replacement rather than software transformation. - Complex service chaining creates a “trombone effect,” increasing latency and making troubleshooting difficult. - Cloudflare’s partners accelerate migrations through: - **Identity-first on-ramps:** Existing identity-provider groups define access policies instead of rebuilding network segments. - **Consolidated policy engines:** SWG and ZTNA policies are handled together, avoiding synchronization between separate products. - **Cloud-native connectors:** Tools such as `cloudflared` provide connectivity without opening inbound firewall ports. ## Scaling Quickly - Adapture expanded one Cloudflare Access deployment from 600 contractors to 5,000 users. - The company describes the expansion as seamless compared with the lengthy implementation cycles associated with legacy SASE platforms. - Cloudflare positions rapid elasticity as important for organizations whose workforce and security needs change quickly. ## A Programmable, Extensible Edge - Cloudflare One is described as software-defined and composable, allowing partners to adapt it to specialized environments. - TachTech supported Arch Linux developer workstations by extracting binaries from an Ubuntu `.deb` package and creating a custom `PKGBUILD`. - This approach preserved device-posture checks, including disk-encryption and firewall-status verification, without creating a security exception. ## Supporting Safe AI Adoption - Cloudflare says the Secure Web Gateway is evolving from simple URL filtering toward controlling data flows to large language models. - Its AI security capabilities include: - **Shadow AI visibility:** Identifying unauthorized AI tools in use across the organization. - **AI confidence scores:** Evaluating models based on standards such as SOC 2 and ISO 42001, as well as data-handling practices. - **DLP prompt protection:** Blocking sensitive source code, personally identifiable information, and financial data from being submitted to public AI services. - **LLM discovery:** Finding and labeling internet-exposed LLM endpoints to reveal the organization’s AI attack surface. - **Request validation:** Intended to defend AI applications against prompt injection and related attacks. Cloudflare’s central recommendation is to replace fragmented, hardware-oriented security deployments with a unified, programmable platform. Doing so can shorten zero trust migrations, simplify operations, preserve consistent security controls across unusual environments, and establish a faster foundation for responsible AI adoption.

Read original(opens in new tab)

PostgreSQLDatabase DesignCloudflareZero Trust+4

line·Mar 9, 2026·4 min readCurated summary

Building an Enterprise LLM Service Part

FAA achieves a 96.1% response rate by favoring simple, maintainable techniques over complex AI architectures. Its design choices were to use RAG instead of knowledge-focused fine-tuning, retrieve complete documents before cutting them into question-relevant sections, and rely on a basic ReAct agent loop rather than elaborate workflows or multiple agents. The article concludes that improving documentation is more valuable than adding complexity when unanswered questions mainly result from missing source material. ## RAG Instead of Fine-Tuning - Fine-tuning was rejected as the primary method for injecting enterprise knowledge. - Research cited in the article found that fine-tuning was highly effective for changing a model’s style—about 97% success—but achieved only about 11% accuracy when teaching new factual knowledge. - FAA’s experiment with approximately 40 examples showed that the model answered the exact training question correctly but failed when the wording changed slightly. - Maintaining larger fine-tuning datasets would require experts to create, verify, and continuously update training examples whenever product documentation changes. - RAG is better suited to frequently changing product information because only the source documents need to be updated. - Fine-tuning may still be useful for domain-specific terminology or reasoning patterns, but not for keeping FAA’s product knowledge current. ## Retrieving Whole Documents Instead of Pre-Chunking - Conventional RAG systems split documents into small chunks before embedding them, improving semantic search precision. - Pre-chunking can remove essential context, especially when references such as “this case” or “the following settings” are separated from the text they depend on. - FAA’s documents are generally short, well-structured, focused on one product and topic, making whole-document retrieval practical. - Instead of chunking before search, FAA embeds and retrieves complete documents, then splits them after the relevant document is known. - The post-split process has two stages: - Split the document by Markdown headers into meaningful sections. - Use a lightweight LLM to select only the sections relevant to the user’s question. - For a question about creating and deleting a VM, the main model might receive only the “VM creation” and “VM deletion” sections. - This extra filtering call remains inexpensive because the lightweight model outputs only section indexes rather than generating a full response. - The key advantage is that splitting happens after the system understands the question, preserving context while delivering only the necessary information. ## ReAct Instead of Complex Agent Workflows - FAA tested plan-and-execute workflows, in which the model first creates a multi-step plan and then carries it out. - Planning and replanning increased system complexity without producing a noticeable improvement in answer quality. - With well-designed tools and carefully filtered context, the model was able to determine tool order on its own. - FAA therefore uses ReAct: the model reasons, takes an action, observes the result, and decides what to do next. - This approach allowed the agent to handle troubleshooting questions without a separate planning layer. ## Rejecting Multi-Agent Architectures - The team also tested specialized agents, such as separate VM and Kubernetes experts. - Delegating questions and assembling the results required additional LLM calls, increasing response time from roughly 9 seconds to 14 seconds in one test. - Multi-agent routing performed poorly for cross-domain questions, such as moving data from a VM to object storage. - Specialists could miss information outside their assigned domain, whereas a single agent could maintain the complete context. - FAA therefore kept one agent with access to progressively disclosed tools and relevant documentation. ## Documentation as the Main Bottleneck - Analysis of unanswered questions showed that about 50% were caused by a documentation gap: no reference document existed. - Other failures were mostly temporary API issues or questions outside FAA’s intended scope. - This suggests the core retrieval and agent system performs well when documentation is available. - The team shares missing questions with product teams, whose updated documents are then re-embedded and incorporated into future evaluations. The practical recommendation is to start with the simplest architecture that fits the data: use RAG for changing knowledge, preserve document context during retrieval, and let a capable model operate through a ReAct loop. In enterprise systems, improving the underlying documentation may produce greater gains than adopting more sophisticated AI frameworks.

Read original(opens in new tab)

PostgreSQLDatabase DesignKubernetesReact+4

datadog·Mar 9, 2026·3 min readCurated summary

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos

Datadog describes how AI-powered attackers targeted its open-source repositories through malicious issues, pull requests, and comments. The campaign, attributed to the “hackerbot-claw” agent, focused on weaknesses in GitHub Actions and LLM-powered workflows. Datadog’s LLM-based review system and layered CI security controls detected the activity and helped limit its impact, while prompting further hardening. ## Why Open-Source Repositories Attract Attackers - Public repositories are attractive targets because automated CI/CD pipelines often build and execute code from external contributions. - Common attack techniques include: - Injecting user-controlled values, such as PR titles, into workflow scripts. - Using indirect poisoned pipeline execution to introduce malicious dependencies or build instructions. - Abusing `pull_request_target` workflows, which may run untrusted code with elevated permissions. - Prompt-injecting LLM-powered GitHub Actions used for issue triage, labeling, or code assistance. - Attackers may also disguise malicious changes through: - Large or obfuscated diffs. - Invisible Unicode characters. - Malicious libraries. - Imposter commits that resemble legitimate dependency references. ## Datadog’s LLM-Based Contribution Detection - Datadog receives dozens of external PRs each week across projects such as the Agent, tracers, SDKs, Vector, chaos-controller, and Stratus Red Team. - Its BewAIre system monitors GitHub events and selects security-relevant activity, including PRs and pushes. - BewAIre: - Extracts, normalizes, and enriches code diffs. - Sends them through a two-stage LLM pipeline. - Classifies changes as benign or malicious. - Produces a structured explanation for each verdict. - Malicious verdicts are forwarded to Datadog Cloud SIEM, where detection rules create enriched signals for the Security Incident Response Team to investigate. ## Hardening CI and Development Workflows - Datadog reduces the potential impact of successful attacks through multiple preventive controls: - Its `dd-octo-sts-action` generates minimally scoped, short-lived GitHub credentials using OIDC. - Long-lived and overly broad personal access tokens and GitHub Apps are being replaced. - Unused GitHub Actions secrets are identified and removed across thousands of repositories. - Organization-wide controls enforce branch protection, mandatory PR approval, commit signing, and lower-privilege default `GITHUB_TOKEN` permissions. - Engineers are provided with documented best practices and secure “golden paths” for CI development. ## The Hackerbot-Claw Campaign - Modern AI models are increasingly capable of offensive security tasks, especially when given tools, feedback loops, and autonomy. - StepSecurity reported an AI agent attacking open-source CI systems on March 1. - Between February 27 and March 2, the actor: - Opened 16 pull requests. - Created two issues and eight comments. - Targeted nine repositories across six organizations. - The activity was later linked to the hackerbot-claw agent, whose GitHub account was removed. - Datadog’s investigation began after BewAIre alerted the team to a suspicious contribution in the newly public `datadog-iac-scanner` repository on February 27. ## Practical Takeaway Organizations that accept public contributions should combine automated, AI-assisted review with least-privilege credentials, strict workflow permissions, secret management, mandatory approvals, and human incident response. Detection alone is insufficient; CI pipelines should be designed so that a malicious contribution has limited access and minimal opportunity to compromise secrets or production systems.

Read original(opens in new tab)

PostgreSQLDatabase DesignLarge Language ModelsCI/CD+4

netflix·Mar 6, 2026·3 min readCurated summary

Scaling Global Storytelling: Modernizing Localization Analytics at Netflix

Netflix is modernizing its localization analytics to support more than 300 million members across 190+ countries and 50+ languages. Rapid growth created duplicated pipelines, inconsistent business logic, and siloed dashboards, making basic questions such as who produced a dub difficult to answer reliably. The company’s solution is to consolidate data foundations, improve usability, and centralize reusable business logic. ## The Challenge of Fragmented Localization Data - Localization metrics were historically built independently across different teams and workflows. - Determining who created a dub or subtitle required combining multiple sources with complex, frequently changing rules. - Duplicated logic led to: - Inconsistent reporting across tools - High maintenance costs when upstream systems changed - Siloed analytics and dashboards ## Auditing and Consolidating Analytics - Netflix audited more than 40 dashboards and tools for usage, quality, and code health. - The focus shifted from repeatedly fixing frontend visualizations to consolidating backend data pipelines. - Three legacy dashboards covering dubbing-partner operations, capacity, and finances are being unified around a shared data and backend layer. - This foundation can support multiple future frontend experiences instead of forcing each dashboard to maintain separate logic. ## Reducing User Experience Debt - Netflix defines “Not-So-Tech Debt” as stakeholder friction caused by confusing tools or weak analytical storytelling. - The Language Asset Consumption tool was redesigned to combine audio and text languages into a single consumption-language view. - This distinguishes: - Original-language viewing from localized consumption - Subtitle, dubbing, or combined preferences - Recurring member preferences for a given language - The result is more intuitive analysis aligned with real stakeholder questions. ## Centralizing Reusable Business Logic - Netflix is adopting a “write once, read many” architecture. - Shared tables, including a Language Asset Producer table, solve common questions in one centralized location. - The same trusted data can feed downstream domains such as Dub Quality and Translation Quality. - Updates to business rules propagate across the analytics ecosystem instead of requiring changes in multiple pipelines. ## Moving Toward Event-Level Analytics - Future work will analyze individual timed-text events rather than only complete language assets. - A generic model will capture details such as individual subtitle lines and reading speed. - Netflix plans to connect subtitle characteristics with member engagement. - These findings can improve style guidelines for subtitle linguists and ultimately enhance the localized viewing experience. Netflix’s recommendation is to treat analytics modernization as both a technical and product-quality effort: consolidate data foundations, centralize business logic, and design tools around how stakeholders actually make decisions. This creates more trustworthy reporting while enabling deeper analysis of how localization affects member enjoyment.

Read original(opens in new tab)

PostgreSQLDatabase DesignData ModelingData Pipelines+4

gitlab·Mar 4, 2026·2 min readCurated summary

10 AI prompts to speed your team’s software delivery

AI-assisted coding can accelerate code production without accelerating delivery, because review, security, documentation, and planning often become the new bottlenecks. The post recommends applying AI across the full software lifecycle, using targeted prompts to reduce routine work and let teams focus on architecture, risk, and business decisions. ## Code Review as an Accelerator - AI can review merge requests (MRs) for: - Logical errors, edge cases, and potential bugs. - API changes, altered return types, schema modifications, and configuration changes that may break consumers. - Catching these issues before human review reduces repeated review cycles and helps prevent deployment-time rollbacks. ## Shifting Security Left - Security scan analysis can use AI to: - Distinguish real vulnerabilities from false positives. - Explain risks and recommend remediation. - Prioritize findings by severity and exploitability. - AI-assisted code reviews can identify injection flaws, authorization problems, data exposure, insecure dependencies, and cryptographic weaknesses before an MR is created. - This reduces security-team backlogs and limits late-stage developer/security rework. ## Keeping Documentation Current - AI can generate release notes from merged MRs, organizing changes into features, fixes, performance improvements, breaking changes, and deprecations. - It can also identify which README files, API references, architecture diagrams, and onboarding guides need updates after code changes. - Automating these checks helps prevent documentation drift without creating a separate manual task. ## Breaking Down Complex Planning - An AI planning prompt can decompose an epic into implementable issues by considering: - Technical dependencies. - Appropriate issue sizes. - Acceptance criteria. - Implementation order. - The goal is to replace lengthy planning meetings with an initial AI-generated breakdown followed by team review. The practical recommendation is to treat AI as a team workflow accelerator, not merely a code generator. Applying focused prompts to review, security, documentation, and planning can help prevent increased coding speed from creating larger downstream bottlenecks.

Read original(opens in new tab)

PostgreSQLDatabase DesignMachine LearningCode Review+4

line·Feb 23, 2026·4 min readCurated summary

Sharing the journey of LINE DEV

AI adoption at LY Corporation has moved beyond experimentation toward learning how to use these tools effectively in real work. The LINE DEV AI Reporters program connects scattered individual and team experiences through internal sharing sessions, helping practical lessons spread across the organization. Its central conclusion is that AI productivity depends not only on tools, but also on clear specifications, sound engineering practices, and a culture of continuous sharing. ## Turning Individual Experiments into Organizational Knowledge - AI enthusiasts across LY Corporation were independently experimenting with tools such as ChatGPT and Claude Code. - These experiences often remained limited to individuals or small teams. - AI Reporters brought together members of different roles and seniority who had experience sharing AI-related work. - Their goal was to turn personal trial and error into reusable organizational knowledge. ## Starting with Informal Personal Experiments - Early AI sharing sessions emphasized accessibility rather than polished success stories. - Multimedia Platform Dev’s Choi Jeong-min shared a “one service a day” vibe-coding experiment using Claude Code and Antigravity. - The experiment demonstrated that rapid implementation increases the importance of clearly defining what to build. - Sharing failures and unfinished experiments reduced the pressure to perform and encouraged more employees to try AI themselves. ## Applying AI to Real Development Work - As interest grew, discussions shifted from fun experiments to practical workplace applications. - Data Dev4’s Lee Yun-seong shared more than a month of project experience using Claude Code, project templates, and Vibe Kanban. - Developers spent more time on planning, design, review, and coordination while agents handled implementation. - Because the current codebase becomes the context for future agent work, poor architecture and coding styles can quickly be reproduced and amplified. - Continuous testing, refactoring, documentation, interface management, and architectural cleanup are therefore essential. - Skipping automated tests before commits led to increasing numbers of broken changes during later merges. - Humans remain responsible for ensuring that AI-generated code actually contributes to the project. - The most valuable skills increasingly involve task design, project management, system context, and meta-programming rather than implementation alone. - Developers can work in parallel with agents by planning the next task, researching requirements, and reviewing completed code while agents execute current work. ## Expanding from Teams to Organization-Wide Programs - Fintech Engineering organized a hands-on workshop covering the full path from idea to deployment. - Participants connected ChatGPT, Claude Code, and Stitch AI to plan, design, build, and complete a working service. - The integrated workflow helped participants understand how AI tools can support an entire product-development process, not just prototyping. - The GAI Study Group in the advertising organization broadened discussions to AI strategy, trends, agent behavior, developer workflows, and business applications. - Topics included: - AI agent reliability - Implementing interactions between PyTorch-based LLMs and MCP servers - Senior and junior developers’ vibe-coding workflows - NotebookLM-based RAG using wiki pages and Slack conversations - One session examined MCP internals by implementing JSON-RPC messaging and session-state management directly, revealing complexities hidden by libraries such as FastMCP. - Sessions were opened to participants and presenters from other teams, with some content published online for wider access. ## Building a Culture of Continuous Sharing - The most useful AI knowledge came from real workplace attempts, failures, and revisions—not only from polished documentation or external trends. - AI Reporters made existing but scattered experiences visible and connected them through presentations, Slack discussions, and monthly meetings. - Informal conversations such as “I tried this—how did it work for you?” helped normalize experimentation and learning from mistakes. - AI adoption is treated as an ongoing practice because tools and workflows continue to change. LY Corporation’s experience suggests that organizations should create lightweight, recurring forums where employees can share practical AI experiments. The combination of rapid experimentation, disciplined engineering, and open knowledge exchange allows individual discoveries to become lasting organizational capability.

Read original(opens in new tab)

PostgreSQLAIClaude CodeChatgpt+4

netflix·Feb 12, 2026·3 min readCurated summary

Automating RDS Postgres to Aurora Postgres Migration

Netflix standardized on Amazon Aurora PostgreSQL after finding that PostgreSQL already supported most relational workloads and that Aurora offered stronger scalability, availability, elasticity, and ecosystem alignment. To migrate nearly 400 RDS PostgreSQL clusters efficiently, Netflix built a self-service workflow that automates replication, traffic quiescence, validation, and cutover while minimizing downtime and eliminating data loss. The Aurora read-replica method is preferred over snapshot migration because it keeps the target nearly synchronized while production continues running. ## Why Netflix Chose Aurora PostgreSQL - PostgreSQL already supported the majority of Netflix’s relational workloads. - Internal evaluations found Aurora PostgreSQL could support more than 95% of workloads running on other relational database systems. - PostgreSQL benefits from: - A broad open-source ecosystem - Strong community adoption - Compatibility with modern data platforms - Aurora’s distributed, cloud-native architecture provides: - Better scalability and elasticity - High availability - Support for globally distributed applications - The migration effort began with RDS PostgreSQL and is intended to expand to other relational systems. ## Database Migration Requires More Than Data Copying A safe migration must move both data and database functionality while preserving correctness, availability, and performance. - **Data replication:** Copy existing data and continuously apply source changes to the destination. - **Quiescence:** Stop writes to the source so the destination can catch up completely. - **Validation:** Confirm that source and destination data are synchronized. - **Cutover:** Redirect applications to the new Aurora database as the system of record. ## Operational and Technical Challenges - Manually migrating almost 400 PostgreSQL clusters would be slow, error-prone, and operationally expensive. - Coordinating downtime across dependent services is difficult. - Netflix therefore created a self-service workflow that handles orchestration, safety checks, and correctness guarantees automatically. - The system must guarantee: - Zero data loss - Extremely short downtime, especially for critical services - No performance degradation during or after migration - Migration of related resources such as parameter groups, read replicas, and replication slots - Application teams control database clients, so the platform cannot depend on them manually pausing writes. - The migration system must provide control-plane mechanisms to halt traffic safely during validation and cutover. - The workflow must operate without obtaining RDS credentials from users, since databases may be tightly secured and the migration platform may lack direct database access. - Because non-experts operate the process, the experience must be self-guided and require minimal user effort. ## Snapshot-Based Migration The snapshot approach is straightforward but requires stopping writes before migration. - Halt write traffic to the RDS PostgreSQL source. - Create a manual snapshot. - Convert the snapshot into an Aurora-compatible format. - Create an Aurora PostgreSQL cluster from the converted snapshot. - Validate the new cluster. - Redirect applications to the Aurora endpoint. This method is simple but can involve a longer interruption because the target is not continuously updated while the snapshot is created and converted. ## Aurora Read-Replica Migration The read-replica approach reduces downtime by continuously replicating the RDS database into Aurora. - Create an Aurora PostgreSQL read replica from the RDS source. - Stream changes asynchronously from RDS to Aurora while applications continue using the source. - Provision and validate Aurora configuration, connectivity, and performance in advance. - When replication lag is sufficiently low, briefly pause writes. - Allow the replica to catch up fully. - Promote it to a standalone Aurora PostgreSQL cluster. - Redirect application traffic to the Aurora endpoint. This approach keeps the destination nearly synchronized before cutover, making it substantially less disruptive than snapshot-based migration. Netflix’s automation focuses on making the read-replica migration process safe, repeatable, and self-service, with the platform handling replication, traffic control, validation, and cutover rather than relying on manual application-team coordination.

Read original(opens in new tab)

PostgreSQLDatabase DesignAWSDatabase Migration+4

line·Feb 6, 2026·4 min readCurated summary

Creating the Cloud of the Future

LY Corporation is consolidating Yahoo! JAPAN and LINE’s internal cloud services into Flava, a private cloud for application development. The article outlines how Flava could evolve over the next two to three years through unified developer platforms, stronger yet more usable security, scalable multimedia storage, AI infrastructure, and intelligent cloud management. Its ultimate goal is to make complex infrastructure easier to consume while automating operational work. ## Platform Flavaization - Flava currently focuses on infrastructure, databases, and containers, while other development services are spread across separate internal platforms. - Developers must learn different systems for: - Access control and approvals - Logging, monitoring, metering, and billing - APIs, CLIs, and user interfaces - Multi-region and availability-zone operations - “Flavaization” means offering all development platforms through a consistent cloud experience. - LY expects much of this integration to be completed within the next one to two years. ## Stronger, More Usable Security - Flava incorporates security governance from the architecture and product-planning stages, working with the CISO organization. - Data environments are separated by security level: - Default - Secret - Top secret - Sensitive changes require role-based permissions, organizational reporting, expert review, and formal approval. - The main challenge is usability: - Resources can now be provisioned within minutes, but access may still require around ten workflows, such as VDI and Box account creation, taking up to two months. - VPC ACL controls can add several milliseconds of latency, which may affect latency-sensitive services such as LINE messaging. - Flava must provide “usable security” that preserves strong governance without making development excessively slow or difficult. ## Storage for Growing Multimedia Data - Users continuously generate and retain large volumes of photos, videos, and other multimedia content. - Storage demand can grow even when service traffic remains stable. - Flava needs storage technologies suited to different data lifecycles, balancing: - Cost - Throughput and latency - Searchability - Compression and deduplication - Encryption - Efficient tiered storage will be essential for managing long-lived user data economically. ## AI Operations Platforms - LY is adopting AI tools and agents across its organizations, creating demand for shared AIOps infrastructure. - Potential platform capabilities include: - Approved MCP server development and management - Vector databases - AI observability tools such as Langfuse - AI model management - Because AI systems handle internal data, these platforms must comply with company security and data-processing policies. - Flava aims to rapidly evaluate emerging AI technologies and provide compliant, standardized services across the company. ## Network and Storage Infrastructure for AI - AI workloads process larger datasets while requiring very low network latency and high throughput. - Relevant technologies include: - DPUs - Smart NICs - High-speed NVMe storage - Automated storage tiering - Operating networks and storage at cloud scale introduces major challenges in latency, reliability, fault tolerance, throughput, change management, and security. - Flava’s existing network and storage engineering teams have experience supporting LINE and Yahoo! JAPAN at large scale and will adapt that expertise for AI workloads. ## The Intelligent Cloud - Future users may describe infrastructure requirements in natural language rather than manually configuring resources through consoles, APIs, CLIs, or Terraform. - For example, Flava could translate requirements for image processing, AI-based content labeling, messaging, and tiered storage into an architecture and deployable system. - An intelligent Flava could also: - Generate network diagrams and ACL matrices - Identify vulnerabilities and prioritize remediation - Recommend cost optimizations - Detect underutilized resources - Find unencrypted personal information - Manage OSS vulnerability responses - Chatbots could automate tasks such as identifying low-utilization resources while excluding standby failover servers or proposing cost reductions for them. - Operational campaigns currently requiring substantial engineer participation could increasingly be handled by AI agents. Flava’s recommended direction is to combine a unified cloud experience with practical security, lifecycle-aware storage, AI-ready infrastructure, and natural-language automation. The article argues that building this future cloud requires both deep infrastructure expertise and strong attention to developer and user experience.

Read original(opens in new tab)

PostgreSQLKubernetesCloud SecurityInfrastructure As Code+4

line·Feb 4, 2026·4 min readCurated summary

Scaling to Infinity: LY Corporation’s

LY Corporation’s observability team evolved its time-series database to handle rapidly growing infrastructure and Kubernetes workloads. After outgrowing MySQL and OpenTSDB, the team built an engine optimized for high-cardinality metrics, low-latency queries, and seamless API compatibility. Its architecture now combines in-memory, Cassandra, and S3-compatible storage, enabling cost-efficient scaling while supporting trillions of daily metrics. ## Why Time-Series Storage Matters - Metrics record system state as timestamped numerical values. - They support dashboards, threshold-based alerts, and predictive analysis using tools such as ARIMA and Prophet. - Even a small metric record can consume about 280 bytes when timestamps, values, and tags are included. - One CPU metric collected every 15 seconds requires roughly 562 MiB per server annually; across 1,000 servers, this grows to about 548 GiB before adding memory, disk, and network metrics. - High-cardinality cloud environments make both storage cost and query latency critical operational concerns. ## Moving Beyond MySQL and OpenTSDB - MySQL initially became inadequate as the organization moved from SOA to MSA: - Write load increased sharply. - Storage costs and capacity requirements grew. - Query latency worsened for large datasets. - Rigid schemas could not easily represent changing cloud resources. - MySQL sharding provided temporary relief but could not support high-resolution metrics collected at intervals under one minute. - OpenTSDB, introduced in 2016 on Apache HBase, improved write performance but had important limitations: - Tag growth harmed UID-table lookup performance. - Metadata was restricted to a narrow character set. - Large queries required cache warm-up procedures. - These constraints led to the development of an internal database beginning in 2018. ## Building the Internal Time-Series Database - The 2019 engine was designed around: - Flexible protocol support independent of a particular agent. - Linear scalability without downtime. - Low-latency processing of high-resolution metrics. - Strong availability during failures. - Inspired by Meta’s Gorilla research, the team used access patterns in which most queries target recent data. - Frequently accessed metrics were kept in an in-memory database, while colder data was stored in Apache Cassandra. - The new engine enabled metric volumes to grow by more than 200 billion records annually while preserving existing APIs. - Users benefited from the new backend without migration work or code changes. ## Scaling for Kubernetes Workloads - Kubernetes introduced rapidly changing pods, dynamically allocated volumes, and much higher metric churn. - Both major storage layers encountered scaling problems: - IMDB initially required adding identical hardware, limiting expansion options. - Cassandra rebalancing could take tens of hours because of its data volume. - The team improved IMDB with weighted load balancing so nodes with different capacities could be used effectively. - Storage was divided into tiers: - Recent 14-day data remained in Cassandra for high-performance access. - Older data was moved to S3-compatible storage. - This reduced Cassandra dependency, lowered costs, simplified operations, and enabled more flexible hardware and Kubernetes-based deployment. ## Writing and Reading Through S3 - The write path separates data processing from long-term storage: - A Dumper reads metric slots from IMDB. - It converts them into internally defined sub-blocks. - A Block Dumper combines sub-blocks into blocks and writes them to S3. - A Storage Gateway reads the blocks for queries and caches them on local disks. - Disk caching initially caused excessive page-cache use and rapid memory exhaustion. - Direct I/O was considered but withdrawn after the cloud storage team warned that it consumed too much shared bandwidth. - Through cross-team collaboration, the team adopted a B+ tree-based cache that made better use of the kernel page cache without overloading infrastructure. ## Future Direction: From Storage to Intelligence - The team aims to move beyond recording metrics toward prediction and AI-assisted operations. - Achieving this requires consolidating time-series data currently scattered across internal systems. - A key requirement is to perform this integration without imposing migration work or breaking changes on users. - The broader goal is an observability platform that turns unified metrics into predictive and intelligent operational capabilities. The main recommendation is to design time-series platforms around real access patterns, tier storage according to data age, and preserve compatibility while evolving the backend. At extreme scale, careful storage architecture and collaboration across infrastructure teams are as important as raw database performance.

Read original(opens in new tab)

PostgreSQLAIDatabase DesignKubernetes+4

aws·Jan 15, 2026Original article

Amazon EC2 X8i instances powered by custom Intel Xeon 6 processors are generally available for memory-intensive workloads (opens in new tab)

Amazon has announced the general availability of EC2 X8i instances, specifically engineered for memory-intensive workloads such as SAP HANA, large-scale databases, and data analytics. Powered by custom Intel Xeon 6 processors with a 3.9 GHz all-core turbo frequency, these instances provide a significant performance leap over the previous X2i generation. By offering up to 6 TB of memory and substantial improvements in throughput, X8i instances represent the highest-performing Intel-based memory-optimized option in the AWS cloud. ### Performance Enhancements and Processor Architecture * **Custom Silicon:** The instances utilize custom Intel Xeon 6 processors available exclusively on AWS, delivering the fastest memory bandwidth among comparable Intel cloud processors. * **Memory and Bandwidth:** X8i provides 1.5 times more memory capacity (up to 6 TB) and 3.4 times more memory bandwidth compared to previous-generation X2i instances. * **Workload Benchmarks:** Real-world performance gains include a 50% increase in SAP Application Performance Standard (SAPS), 47% faster PostgreSQL performance, 88% faster Memcached performance, and a 46% boost in AI inference. ### Scalable Instance Sizes and Throughput * **Flexible Sizing:** The instances are available in 14 sizes, including new larger formats such as the 48xlarge, 64xlarge, and 96xlarge. * **Bare Metal Options:** Two bare metal sizes (metal-48xl and metal-96xl) are available for workloads requiring direct access to physical hardware resources. * **Networking and Storage:** The architecture supports up to 100 Gbps of network bandwidth with Elastic Fabric Adapter (EFA) support and up to 80 Gbps of Amazon EBS throughput. * **Bandwidth Control:** Support for Instance Bandwidth Configuration (IBC) allows users to customize the allocation of performance between networking and EBS to suit specific application needs. ### Cost Efficiency and Use Cases * **Licensing Optimization:** In preview testing, customers like Orion reduced SQL Server licensing costs by 50% by maintaining performance thresholds with fewer active cores compared to older instance types. * **Enterprise Applications:** The instances are SAP-certified, making them ideal for RISE with SAP and other high-demand ERP environments. * **Broad Utility:** Beyond databases, the instances are optimized for Electronic Design Automation (EDA) and complex data analytics that require massive memory footprints. For organizations managing massive datasets or expensive licensed database software, migrating to X8i instances offers a clear path to both performance optimization and infrastructure cost reduction. These instances are currently available in the US East (N. Virginia), US West (Oregon), and Europe (Ireland) regions through On-Demand, Spot, and Reserved purchasing models.

PostgreSQLCloud ComputingAmazon Ec2Ai Inference+4

aws·Dec 15, 2025Original article

AWS Weekly Roundup: Amazon ECS, Amazon CloudWatch, Amazon Cognito and more (December 15, 2025) (opens in new tab)

The AWS Weekly Roundup for mid-December 2025 highlights a series of updates designed to streamline developer workflows and enhance security across the cloud ecosystem. Following the momentum of re:Invent 2025, these releases focus on reducing operational friction through faster database provisioning, more granular container control, and AI-assisted development tools. These advancements collectively aim to simplify infrastructure management while providing deeper cost visibility and improved performance for enterprise applications. ## Database and Developer Productivity * **Amazon Aurora DSQL** now supports near-instant cluster creation, reducing provisioning time from minutes to seconds to facilitate rapid prototyping and AI-powered development via the Model Context Protocol (MCP) server. * **Amazon Aurora PostgreSQL** has integrated with **Kiro powers**, allowing developers to use AI-assisted coding for schema management and database queries through pre-packaged MCP servers. * **Amazon CloudWatch SDK** introduced support for optimized JSON and CBOR protocols, improving the efficiency of data transmission and processing within the monitoring suite. * **Amazon Cognito** simplified user communications by enabling automated email delivery through Amazon SES using verified identities, removing the need for manual SES configuration. ## Compute and Networking Optimizations * **Amazon ECS on AWS Fargate** now honors custom container stop signals, such as SIGQUIT or SIGINT, allowing for graceful shutdowns of applications that do not use the default SIGTERM instruction. * **Application Load Balancer (ALB)** received performance enhancements that reduce latency for establishing new connections and lower resource consumption during traffic processing. * **AWS Fargate** cost optimization strategies were highlighted in new technical guides, focusing on leveraging Graviton processors and Fargate Spot to maximize compute efficiency. ## Security and Cost Management * **Amazon WorkSpaces Secure Browser** introduced Web Content Filtering, providing category-based access control across 25+ predefined categories and granular URL policies at no additional cost. * **AWS Cost Management** tools now feature **Tag Inheritance**, which automatically applies tags from resources to cost data, allowing for more precise tracking in Cost Explorer and AWS Budgets. * **Amazon Step Functions** integration with Amazon Bedrock was further detailed in community resources, showcasing how to build resilient, long-running AI workflows with integrated error handling. To take full advantage of these updates, organizations should review their Fargate task definitions to implement custom stop signals for better application stability and enable Tag Inheritance to improve the accuracy of year-end cloud financial reporting.

PostgreSQLCloud ComputingAmazon CloudwatchAmazon Ecs+3

datadog·Nov 4, 2025·3 min readCurated summary

Replication redefined: How we built a low-latency, multi-tenant data replication platform

Datadog built a managed, multi-tenant data replication platform to move data reliably across thousands of services without brittle, point-to-point integrations. The effort began by separating analytical search workloads from a shared PostgreSQL database, then evolved into automated pipeline provisioning with Temporal. The platform favors asynchronous replication to improve scalability and resilience, accepting limited replication lag in exchange for lower application latency and reduced operational coupling. ## Scaling Search Beyond PostgreSQL - A shared PostgreSQL database initially provided low-latency access, ACID guarantees, and low operational cost. - As data volumes grew, complex joins and aggregations became increasingly slow. - Datadog’s Metrics Summary page had to join: - 82,000 active metrics - 817,000 metric configurations - Page latency reached approximately 7 seconds at p90, while repeated facet changes generated additional expensive queries. - Index and disk bloat, memory pressure, VACUUM and ANALYZE overhead, and rising I/O wait further reduced throughput. - Rather than continuing to optimize PostgreSQL for analytical search, Datadog moved search and aggregation workloads to a dedicated search platform. - Data was denormalized during replication, producing document-oriented indexes better suited to faceted search. - The resulting system reduced page-load times by as much as 97%—from roughly 30 seconds to 1 second—while maintaining about 500 ms of replication lag. ## Automating Pipeline Provisioning with Temporal Provisioning a replication pipeline required coordinating multiple systems and configuration steps: - Enabling PostgreSQL logical replication with `wal_level`. - Creating users and assigning replication permissions. - Configuring publishers and replication slots. - Deploying Debezium instances to capture PostgreSQL changes. - Creating Kafka topics and mapping them to Debezium instances. - Adding heartbeat tables to monitor replication and prevent excessive WAL retention. - Configuring sink connectors to write Kafka data into the search platform. Manual management became increasingly difficult across many pipelines and data centers. Datadog used Temporal workflows to split provisioning into modular, repeatable tasks and combine them into higher-level orchestrations. This reduced errors, improved consistency, and allowed engineers to create and modify pipelines without repeating complex operational procedures. ## Choosing Asynchronous Replication - Synchronous replication provides strong consistency by waiting for replicas to acknowledge each write. - However, it increases latency and operational complexity, particularly across distributed environments. - Asynchronous replication allows the primary system to acknowledge writes immediately while replicas catch up afterward. - Datadog selected the asynchronous model because it decouples application performance from network latency and replica availability. - The trade-off is temporary replication lag during failures or periods of pressure, but the model offers better scalability and resilience for high-throughput systems. Datadog’s experience suggests that replication should be treated as a managed platform rather than a collection of custom integrations. Separating workloads, automating provisioning, and choosing asynchronous delivery can improve performance and reliability while reducing the operational burden on individual engineering teams.

Read original(opens in new tab)

PostgreSQLDatabase DesignApache KafkaElasticsearch+4

datadog·Jun 17, 2025·3 min readCurated summary

Breaking up a monolith: How we’re unwinding a shared database at scale

Datadog is moving away from a large shared relational database because its benefits eventually give way to coordination costs, schema fragility, noisy-neighbor problems, and scaling limits. Splitting the database is difficult and expensive, but platform investments in service development and managed Postgres can make independently owned databases practical. The key is to establish functional boundaries, provide safe cross-domain access, and automate migrations. ## Why Shared Databases Persist - Shared databases reduce operational overhead for small or fast-moving organizations. - A single database enables simple, low-latency joins across all data. - Workload isolation and access management often matter less when systems are small. - Because the cost of splitting a database is high, organizations commonly keep the shared model longer than they should. ## Signs It Is Time to Split the Database - Data grows beyond the capacity of one machine, or replication becomes too slow. - Noisy-neighbor effects make performance unpredictable. - Schema changes by one team unexpectedly affect others. - Security requirements such as access-control lists are difficult to enforce. - These issues create engineering costs, incidents, and degraded user experiences across teams. ## What Database Decomposition Requires - Identify functional ownership boundaries. - Build services for cross-domain queries where necessary. - Require consumers to use those services instead of querying another domain’s tables directly. - Provision new database instances. - Migrate data and traffic carefully from the shared database to the new instances. Datadog had previously split off large portions of its database into only a few separate databases. The experience showed that finding boundaries, enforcing them, and migrating without incidents is difficult and highly manual. ## Why Teams Resist Leaving Shared Infrastructure - Building a service may jeopardize existing product goals. - Operating a service can introduce significant maintenance and on-call work. - Cross-domain data access may be unclear or cause unacceptable latency or user impact. - Owning a database creates additional operational responsibility. - Migrations are often handcrafted, risky, and difficult to repeat. - Forcing the transition can cost more than tolerating the existing problems and create organizational resistance. ## Platform Investments That Enable Change Datadog addressed these obstacles through two major initiatives: - **Rapid:** An opinionated framework for building and operating API and gRPC services. - **OrgStore:** A managed platform for Postgres databases. Rapid reduces the cost of creating and maintaining services by providing shared configuration, common data-access patterns, and operational support. OrgStore reduces the burden of owning separate database instances. Together, these platforms make it more attractive for new projects to avoid the legacy shared database and allow existing domains to migrate incrementally. The broader lesson is that database decomposition becomes realistic when platform engineering makes service ownership, database operations, cross-domain access, and migrations safe enough to fit into normal product development.

Read original(opens in new tab)

PostgreSQLDatabase DesignObservabilityGrpc+4

figma·May 28, 2025·4 min readCurated summary

From Multi-Day Latency to Near Real-Time Insights: Figma’s Data Pipeline Upgrade | Figma Blog

Figma replaced a daily full-table export system that could take hours or days with an incremental synchronization pipeline designed for near real-time analytics. The new architecture combines database snapshots, change data capture (CDC), and Snowflake merge logic to transfer only recent changes. By building the system in-house, Figma gained greater flexibility, lower projected costs, and a design that can scale with continued growth. ## Why the Legacy Pipeline Failed - Since 2020, a daily cron job ran `SELECT * FROM <TABLE>`, exported results to S3, and loaded them into Snowflake. - As Figma’s tables and insert volume grew: - Daily syncs reached roughly six hours by 2023. - The largest tables took several days or longer. - Additional database replicas were required for exports. - Replica maintenance cost millions of dollars annually. - The delays limited access to timely company KPIs and analytical insights. ## Choosing Incremental Synchronization Figma evaluated three options: - Continue using the legacy process, which was increasingly expensive and too slow. - Add parallelism, which might improve throughput temporarily but would not scale sustainably. - Rebuild the synchronization system around incremental updates. Incremental synchronization transfers only new and changed records instead of repeatedly copying entire tables, reducing data movement, processing time, and infrastructure usage. ## Buy vs. Build Figma decided to build the pipeline internally because available proprietary tools did not meet its requirements. - **Flexibility:** Generic SQL tools did not take advantage of capabilities such as Amazon RDS for PostgreSQL snapshot exports. - **Cost:** Commercial solutions were projected to cost five to ten times more than an in-house implementation. - **Scale:** Building internally allowed Figma to optimize the system for its infrastructure and adapt it as the company grows. ## Pipeline Components The bespoke system combines several lower-level technologies: - **Snapshots:** Amazon RDS exports initial table copies to S3. - **Change data capture:** Kafka Connect streams database changes through Amazon MSK. - **Warehouse ingestion:** A Snowflake Connector loads CDC events into Snowflake. - **Incremental merging:** Custom Snowflake stored procedures and scheduled tasks merge changes into base tables. ## Architecture Principles The redesign was guided by four goals: - Reduce end-to-end synchronization latency. - Control costs as data volume increases. - Meet regulatory and compliance requirements. - Preserve data accuracy, completeness, consistency, and trustworthiness. The resulting architecture uses two workflows: a bootstrap workflow for onboarding tables and a validation workflow for checking data correctness. ## Bootstrap Workflow The automated onboarding process includes: - The CDC service begins capturing the new Postgres table and publishes events to a per-table Kafka topic. - Amazon RDS exports the latest database snapshot to S3. - Snowflake’s `COPY INTO <table>` loads the snapshot into a per-entity base table. - An MSK Connect Snowflake Sink Connector streams Kafka events into a separate CDC table, with offsets arranged so changes before the snapshot timestamp are retained. - A scheduled Snowflake task runs a custom `MERGE` procedure to combine the snapshot and CDC data. - Once the process catches up with current changes, Figma creates a lightweight user-facing view over the base table. ## Zero-Downtime Re-Bootstrapping - Bootstrap artifacts are versioned, while the final user-facing view remains stable. - New versions can be built in parallel without interrupting queries. - Promotion is completed through an atomic view update. - This supports schema evolution and other situations requiring a fresh bootstrap without downtime. ## Data Validation - Even well-designed pipelines can suffer corruption from partial failures, configuration errors, software bugs, or unexpected source-data anomalies. - Figma therefore added a validation workflow to verify correctness as data moves through snapshot exports, CDC capture, and incremental merging. Figma’s experience shows that incremental synchronization is a more sustainable alternative to repeated full-table exports. Combining managed infrastructure with custom orchestration can deliver lower latency, better cost control, and stronger operational flexibility than a one-size-fits-all commercial pipeline.

Read original(opens in new tab)

PostgreSQLApache KafkaAmazon S3Data Pipeline+4

datadog·Jan 15, 2025·1 min readCurated summary

Unraveling a Postgres segfault that uncovered an Arm64 JIT compiler bug | Datadog

The provided content does not include the blog post itself; it contains Datadog’s navigation menu and a link titled “Unraveling a Postgres Segfault.” The only substantive claim shown is that Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. ### Datadog’s Observability Offering - The navigation lists products for: - Infrastructure and Kubernetes monitoring - Application performance monitoring and profiling - Database, log, and data observability - Security and cloud protection - Digital experience monitoring - CI/CD and software delivery - Incident response and service management - AI-agent and GPU observability - It also highlights platform capabilities such as dashboards, alerts, workflow automation, access control, and governance. ### Missing Post Content - No discussion of the PostgreSQL segmentation fault is included. - The supplied text does not describe the failure’s cause, investigation process, technical diagnosis, or resolution. Please provide the article body or a complete excerpt for an accurate technical summary.

Read original(opens in new tab)

PostgreSQLCDebuggingSegmentation Fault+1