aws·Jan 20, 2026Original article

AWS Weekly Roundup: Kiro CLI latest features, AWS European Sovereign Cloud, EC2 X8i instances, and more (January 19, 2026) (opens in new tab)

The January 19, 2026, AWS Weekly Roundup highlights significant advancements in sovereign cloud infrastructure and the general availability of high-performance, memory-optimized compute instances. The update also emphasizes the maturing ecosystem of AI agents, focusing on enhanced developer tooling and streamlined deployment workflows for agentic applications. These releases collectively aim to satisfy stringent regulatory requirements in Europe while pushing the boundaries of enterprise performance and automated productivity. ## Developer Tooling and Kiro CLI Enhancements * New granular controls for web fetch URLs allow developers to use allowlists and blocklists to strictly govern which external resources an agent can access. * The update introduces custom keyboard shortcuts to facilitate seamless switching between multiple specialized agents within a single session. * Enhanced diff views provide clearer visibility into changes, improving the debugging and auditing process for automated workflows. ## AWS European Sovereign Cloud General Availability * Following its initial 2023 announcement, this independent cloud infrastructure is now generally available to all customers. * The environment is purpose-built to meet the most rigorous sovereignty and data residency requirements for European organizations. * It offers a comprehensive set of AWS services within a framework that ensures operational independence and localized data handling. ## High-Performance Computing with EC2 X8i Instances * The memory-optimized X8i instances, powered by custom Intel Xeon 6 processors, have moved from preview to general availability. * These instances feature a sustained all-core turbo frequency of 3.9 GHz, which is currently exclusive to the AWS platform. * The hardware is SAP certified and engineered to provide the highest memory bandwidth and performance for memory-intensive enterprise workloads compared to other Intel-based cloud offerings. ## Agentic AI and Productivity Updates * Amazon Quick Suite continues to expand as a workplace "agentic teammate," designed to synthesize research and execute actions based on organizational insights. * New technical guidance has been released regarding the deployment of AI agents on Amazon Bedrock AgentCore. * The integration of GitHub Actions is now supported to automate the deployment and lifecycle management of these AI agents, bridging the gap between traditional DevOps and agentic AI development. These updates signal a strategic shift toward highly specialized infrastructure, both in terms of regulatory compliance with the Sovereign Cloud and raw performance with the X8i instances. Organizations looking to scale their AI operations should prioritize the new deployment patterns for Bedrock AgentCore to ensure a robust CI/CD pipeline for their autonomous agents.

AWSAIAI AgentsAmazon Bedrock+5

toss·Jan 8, 2026Original article

Managing Thousands of API/ (opens in new tab)

Toss Payments manages thousands of API and batch server configurations that handle trillions of won in transactions, where a single typo in a JVM setting can lead to massive financial infrastructure failure. To solve the risks associated with manual "copy-paste" workflows and configuration duplication, the team developed a sophisticated system that treats configuration as code. By implementing layered architectures and dynamic templates, they created a testable, unified environment capable of managing complex hybrid cloud setups with minimal human error. ## Overlay Architecture for Hierarchical Control * The team implemented a layered configuration system consisting of `global`, `cluster`, `phase`, and `application` levels. * Settings are resolved by priority, where lower-level layers override higher-level defaults, allowing servers to inherit common settings while maintaining specific overrides. * This structure allows the team to control environment-specific behaviors, such as disabling canary deployments in development environments, from a single centralized directory. * The directory structure maps files 1:1 to their respective layers, ensuring that naming conventions drive the CI/CD application process. ## Solving Duplication with Template Patterns * Standard YAML overlays often fail when dealing with long strings or arrays, such as `JVM_OPTION`, because changing a single value usually requires redefining the entire block. * To prevent the proliferation of nearly identical environment variables, the team introduced a template pattern using placeholders like `{{MAX_HEAP}}`. * Developers can modify specific parameters at the application layer while the core string remains defined at the global layer, significantly reducing the risk of typos. * This approach ensures that critical settings, like G1GC parameters or heap region sizes, remain consistent across the infrastructure unless explicitly changed. ## Dynamic and Conditional Configuration Logic * The system allows for "evolutionary" configurations where Python scripts can be injected to generate dynamic values, such as random JMX ports or data fetched from remote APIs. * Advanced conditional logic was added to handle complex deployment scenarios, enabling environment variables to change their values automatically based on the target cluster name (e.g., different profiles for AWS vs. IDC). * By treating configuration as a living codebase, the team can adapt to new infrastructure requirements without abandoning their core architectural principles. ## Reliable Batch Processing through Simplicity * For batch operations handling massive settlement volumes, the team prioritized "appropriate technology" and simplicity to minimize failure points. * They chose Jenkins for its low learning curve and reliability, despite its lack of native GitOps support. * To address inconsistencies in manual UI entries and varying Java versions across machines, they standardized the batch infrastructure to ensure that high-stakes financial calculations are executed in a controlled, predictable environment. The most effective way to manage large-scale infrastructure is to transition from static, duplicated configuration files to a dynamic, code-centric system. By combining an overlay architecture for hierarchy and a template pattern for granular changes, organizations can achieve the flexibility needed for hybrid clouds while maintaining the strict safety standards required for financial systems.

AWSKubernetesInfrastructure As CodeGitops+3

netflix·Dec 15, 2025Original article

Netflix Live Origin. Xiaomei Liu, Joseph Lynch, Chris Newton | by Netflix Technology Blog | Dec, 2025 | Netflix TechBlog (opens in new tab)

The Netflix Live Origin is a specialized, multi-tenant microservice designed to bridge the gap between cloud-based live streaming pipelines and the Open Connect content delivery network. By operating as an intelligent broker, it manages content selection across redundant regional pipelines to ensure that only valid, high-quality segments are distributed to client devices. This architecture allows Netflix to achieve high resilience and stream integrity through server-side failover and deterministic segment selection. ### Multi-Pipeline and Multi-Region Awareness * The origin server mitigates common live streaming defects, such as missing segments, timing discontinuities, and short segments containing missing video or audio samples. * It leverages independent, redundant streaming pipelines across different AWS regions to ensure high availability; if one pipeline fails or produces a defective segment, the origin selects a valid candidate from an alternate path. * Implementation of epoch locking at the cloud encoder level allows the origin to interchangeably select segments from various pipelines. * The system uses lightweight media inspection at the packager level to generate metadata, which the origin then uses to perform deterministic candidate selection. ### Stream Distribution and Protocol Integration * The service operates on AWS EC2 instances and utilizes standard HTTP protocol features for communication. * Upstream packagers use HTTP PUT requests to push segments into storage at specific URLs, while the downstream Open Connect network retrieves them via GET requests. * The architecture is optimized for a manifest design that uses segment templates and constant segment durations, which reduces the need for frequent manifest refreshes. ### Open Connect Streaming Optimization * While Netflix’s Open Connect Appliances (OCAs) were originally optimized for VOD, the Live Origin extends nginx proxy-caching functionality to meet live-specific requirements. * OCAs are provided with Live Event Configuration data, including Availability Start Times and initial segment numbers, to determine the legitimate range of segments for an event. * This predictive modeling allows the CDN to reject requests for objects outside the valid range immediately, reducing unnecessary traffic and load on the origin. By decoupling the live streaming pipeline from the distribution network through this specialized origin layer, Netflix can maintain a high level of fault tolerance and stream stability. This approach minimizes client-side complexity by handling failovers and segment selection on the server side, ensuring a seamless experience for viewers of live events.

AWSMicroservicesLive StreamingNginx+4

aws·Dec 8, 2025Original article

AWS Weekly Roundup: AWS re:Invent keynote recap, on-demand videos, and more (December 8, 2025) (opens in new tab)

The December 8, 2025, AWS Weekly Roundup recaps the major themes from AWS re:Invent, signaling a significant industry transition from AI assistants to autonomous AI agents. While technical innovation in infrastructure remains a priority, the event underscored that developers remain at the heart of the AWS mission, empowered by new tools to automate complex tasks using natural language. This shift represents a "renaissance" in cloud computing, where purpose-built infrastructure is now designed to support the non-deterministic nature of agentic workloads. ## Community Recognition and the Now Go Build Award * Raphael Francis Quisumbing (Rafi) from the Philippines was honored with the Now Go Build Award, presented by Werner Vogels. * A veteran of the ecosystem, Quisumbing has served as an AWS Hero since 2015 and has co-led the AWS User Group Philippines for over a decade. * The recognition emphasizes AWS's continued focus on community dedication and the role of individual builders in empowering regional developer ecosystems. ## The Evolution from AI Assistants to Agents * AWS CEO Matt Garman identified AI agents as the next major inflection point for the industry, moving beyond simple chat interfaces to systems that perform tasks and automate workflows. * Dr. Swami Sivasubramanian highlighted a paradigm shift where natural language serves as the primary interface for describing complex goals. * These agents are designed to autonomously generate plans, write necessary code, and call various tools to execute complete solutions without constant human intervention. * AWS is prioritizing the development of production-ready infrastructure that is secure and scalable specifically to handle the "non-deterministic" behavior of these AI agents. ## Core Infrastructure and the Developer Renaissance * Despite the focus on AI, AWS reaffirmed that its core mission remains the "freedom to invent," keeping developers central to its 20-year strategy. * Leaders Peter DeSantis and Dave Brown reinforced that foundational attributes—security, availability, and performance—remain the non-negotiable pillars of the AWS cloud. * The integration of AI agents is framed as a way to finally realize material business returns on AI investments by moving from experimental use cases to automated business logic. To maximize the value of these updates, organizations should begin evaluating how to transition from simple LLM implementations to agentic frameworks that can execute end-to-end business processes. Reviewing the on-demand keynote sessions from re:Invent 2025 is recommended for technical teams looking to implement the latest secure, agent-ready infrastructure.

AWSAIAI AgentsMachine Learning+3

aws·Dec 3, 2025Original article

Introducing checkpointless and elastic training on Amazon SageMaker HyperPod (opens in new tab)

Amazon SageMaker HyperPod has introduced checkpointless and elastic training features to accelerate AI model development by minimizing infrastructure-related downtime. These advancements replace traditional, slow checkpoint-restart cycles with peer-to-peer state recovery and enable training workloads to scale dynamically based on available compute capacity. By decoupling training progress from static hardware configurations, organizations can significantly reduce model time-to-market while maximizing cluster utilization. **Checkpointless Training and Rapid State Recovery** * Replaces the traditional five-stage recovery process—including job termination, network setup, and checkpoint retrieval—which can often take up to an hour on self-managed clusters. * Utilizes peer-to-peer state replication and in-process recovery to allow healthy nodes to restore the model state instantly without restarting the entire job. * Incorporates technical optimizations such as collective communications initialization and memory-mapped data loading to enable efficient data caching. * Reduces recovery downtime by over 80% based on internal studies of clusters with up to 2,000 GPUs, and was a core technology used in the development of Amazon Nova models. **Elastic Training and Automated Cluster Scaling** * Allows AI workloads to automatically expand to use idle cluster capacity as it becomes available and contract when resources are needed for higher-priority tasks. * Reduces the need for manual intervention, saving hours of engineering time previously spent reconfiguring training jobs to match fluctuating compute availability. * Optimizes total cost of ownership by ensuring that training momentum continues even as inference volumes peak and pull resources away from the training pool. * Orchestrates these transitions seamlessly through the HyperPod training operator, ensuring that model development is not disrupted by infrastructure changes. For teams managing large-scale AI workloads, adopting these features can reclaim significant development time and lower operational costs by preventing idle cluster periods. Organizations scaling to thousands of accelerators should prioritize checkpointless training to mitigate the impact of hardware faults and maintain continuous training momentum.

AWSAIMachine LearningCloud Computing+5

aws·Dec 2, 2025Original article

Introducing Database Savings Plans for AWS Databases (opens in new tab)

AWS has expanded its flexible pricing model to include managed database services with the launch of Database Savings Plans, offering up to 35% cost reduction for consistent usage. By committing to a specific hourly spend over a one-year term, customers can maintain cost efficiency across multiple accounts, resource types, and AWS Regions. This initiative simplifies financial management for organizations running diverse data-driven and AI applications while providing the agility to modernize architectures without losing discounted rates. ### Flexibility and Modernization Support * The plan allows customers to switch between different database engines and deployment types, such as moving from provisioned instances to serverless options, without affecting their savings. * Usage is portable across AWS Regions, enabling global organizations to shift workloads as business needs evolve while retaining their commitment benefits. * The model supports ongoing cost optimization by automatically applying discounts to new instance types, sizes, or eligible database offerings as they become available. ### Service Coverage and Tiered Discounts * Database Savings Plans cover a wide array of services, including Amazon Aurora, RDS, DynamoDB, ElastiCache, DocumentDB, Neptune, Keyspaces, Timestream, and AWS DMS. * Serverless deployments offer the most significant savings, providing up to 35% off standard on-demand rates. * Provisioned instances across supported services deliver discounts of up to 20%. * Specific workloads for Amazon DynamoDB and Amazon Keyspaces receive tailored rates, with up to 18% savings for on-demand throughput and up to 12% for provisioned capacity. ### Implementation and Cost Management * Customers can purchase and manage these plans through the AWS Billing and Cost Management Console or via the AWS CLI. * Discounts are applied automatically on an hourly basis to all eligible usage; any consumption exceeding the hourly commitment is billed at the standard on-demand rate. * Integrated cost management tools allow users to analyze their coverage and utilization, ensuring spend remains predictable even as application usage patterns fluctuate. For organizations with stable or growing database requirements, Database Savings Plans offer a low-risk path to reducing operational expenses. Customers should utilize the AWS Cost Explorer to analyze their historical usage and determine an appropriate hourly commitment level to maximize their return on investment over a one-year term.

AWSServerlessAmazon RdsAmazon Dynamodb+4

aws·Dec 2, 2025Original article

New and enhanced AWS Support plans add AI capabilities to expert guidance (opens in new tab)

AWS has announced a major transformation of its support plans, moving from a reactive model to a proactive, AI-driven approach for issue prevention and workload optimization. By integrating AI-powered capabilities with deep technical expertise, these enhanced plans aim to help organizations identify potential operational risks before they impact business performance. This new tier-based structure provides businesses with varying levels of contextual assistance, ranging from intelligent automated recommendations to direct access to specialized engineering teams. ### Business Support+ * Introduces intelligent, AI-powered assistance designed to provide contextual recommendations for developers, startups, and small businesses. * Features a seamless transition from AI tools to human experts, with critical case response times reduced to 30 minutes—twice as fast as previous standards. * Provides personalized workload optimization suggestions based on the user's specific environment via a low-cost monthly subscription. ### Enterprise Support * Assigns a designated Technical Account Manager (TAM) who utilizes data-driven insights and AI tools to mitigate risks and identify optimization opportunities. * Grants access to the AWS Security Incident Response service at no additional fee, centralizing the tracking, monitoring, and investigation of security events. * Guarantees a 15-minute response time for production-critical issues, with support engineers receiving AI-generated context to ensure faster, more personalized resolution. * Includes access to hands-on workshops and interactive programs to foster continuous technical growth within the organization. ### Unified Operations Support * Provides the highest level of context-aware assistance through a dedicated core team including a TAM, a Domain Engineer, and a Senior Billing and Account Specialist. * Delivers industry-leading 5-minute response times for critical incidents, supported by around-the-clock monitoring and AI-powered proactive risk identification. * Offers on-demand access to specialized experts in migration, incident management, and security through the customer’s preferred collaboration channels. These updates reflect AWS’s commitment to using generative AI to shorten resolution times and provide more personalized architectural guidance. Organizations should evaluate their operational complexity and required response times to select the plan that best aligns with their mission-critical cloud needs.

AWSAIIncident ManagementCloud Monitoring+3

aws·Dec 2, 2025Original article

Amazon OpenSearch Service improves vector database performance and cost with GPU acceleration and auto-optimization (opens in new tab)

Amazon OpenSearch Service has introduced serverless GPU acceleration and auto-optimization features designed to enhance the performance and cost-efficiency of large-scale vector databases. These updates allow users to build vector indexes up to ten times faster at a quarter of the traditional indexing cost, enabling the creation of billion-scale databases in under an hour. By automating complex tuning processes, OpenSearch Service simplifies the deployment of generative AI and high-speed search applications. ### GPU Acceleration for Rapid Indexing The new serverless GPU acceleration streamlines the creation of vector data structures by offloading intensive workloads to specialized hardware. * **Performance Gains:** Indexing speed is increased by 10x compared to non-GPU configurations, significantly reducing the time-to-market for data-heavy applications. * **Cost Efficiency:** Indexing costs are reduced to approximately 25% of standard costs, and users only pay for active processing through OpenSearch Compute Units (OCU) rather than idle instance time. * **Serverless Management:** There is no need to provision or manage GPU instances manually; OpenSearch Service automatically detects acceleration opportunities and isolates workloads within the user's Amazon VPC. * **Operational Scope:** Acceleration is automatically applied to both initial indexing and subsequent force-merge operations. ### Automated Vector Index Optimization Auto-optimization removes the requirement for deep vector expertise by automatically balancing competing performance metrics. * **Simplified Tuning:** The system replaces manual index tuning—which can traditionally take weeks—with automated configurations. * **Resource Balancing:** The tool finds the optimal trade-off between search latency, search quality (recall rates), and memory requirements. * **Improved Accuracy:** Users can achieve higher recall rates and better cost savings compared to using default, unoptimized index configurations. ### Configuration and Integration These features can be integrated into new or existing OpenSearch Service domains and Serverless collections through the AWS Console or CLI. * **CLI Activation:** Users can enable acceleration on existing domains using the `update-domain-config` command with the `--aiml-options` flag set to enable `ServerlessVectorAcceleration`. * **Index Settings:** To leverage GPU processing, users must create a vector index with specific settings, notably setting `index.knn.remote_index_build.enabled` to `true`. * **Supported Workloads:** The service supports standard OpenSearch operations, including the Bulk API for adding vector data and text embeddings. For organizations managing large-scale vector workloads for RAG (Retrieval-Augmented Generation) or semantic search, enabling GPU acceleration is a highly recommended step to reduce operational overhead. Developers should transition existing indexes to include the `remote_index_build` setting to take immediate advantage of the improved speed and reduced OCU pricing.

AWSAIGenerative AIVector Db+4

coupang·Nov 15, 2024Original article

Optimizing Operating Costs through Cloud Service (opens in new tab)

Coupang’s Finance and Engineering teams collaborated to optimize cloud expenditures by focusing on resource efficiency and the company's "Hate Waste" leadership principle. Through a dedicated optimization project team and the implementation of data-driven analytics, the company successfully reduced on-demand costs by millions of dollars without compromising business growth. This initiative transformed cloud management from a reactive expense into a proactive engineering culture centered on financial accountability and technical efficiency. ### Forming the Optimization Project Team * A specialized team consisting of Cloud Infrastructure Engineers and Technical Program Managers (TPMs) was established to bridge the gap between finance and engineering. * The project team focused on educating domain teams about the variable cost model of cloud services, moving away from a fixed-cost mindset. * Technical experts helped domain teams identify opportunities to use cost-efficient technologies, such as ARM-based AWS Graviton processors and AWS Spot Instances for data processing. * The initiative established clear ownership, ensuring that each domain team understood and managed their specific cloud resource usage. ### Analytics and Dashboards for Visibility * Engineers developed custom dashboards using Amazon Athena to process Amazon CloudWatch data, providing deep insights into resource performance. * The team utilized AWS Cost & Usage Reports (CUR) within internal Business Intelligence (BI) tools to provide granular visibility into spending patterns. * Finance teams worked alongside engineers to align technical roadmaps with monthly and quarterly budget goals, making cost management a shared responsibility. ### Strategies for Usage and Cost Reduction * **Spend Less (Usage Reduction):** Coupang implemented automation to ensure that non-production environment resources were only active when needed, resulting in a 25% cost saving for those environments. * **Pay Less (Right-sizing):** The team analyzed usage patterns to manually identify and decommission unused EC2 resources across all domain teams. * **Instance and Storage Optimization:** The project prioritized migrating workloads to the latest instance generations and optimizing Amazon S3 storage structures to reduce costs for data at rest. To achieve sustainable cloud efficiency, organizations should move beyond simple monitoring and foster an engineering culture where resource management is a core technical discipline. Prioritizing automated resource scheduling and adopting modern, high-efficiency hardware like Graviton instances are essential steps for any large-scale cloud operation looking to maximize its return on investment.

AWSAmazon Ec2Amazon S3Finops+3

coupang·Nov 14, 2024Original article

Cloud expenditure optimization for cost efficiency (opens in new tab)

Coupang addressed rising cloud costs by establishing a cross-functional Central team to bridge the gap between engineering usage and financial accountability. Through a data-driven approach involving custom analytics and automated resource management, the company successfully reduced on-demand expenditure by millions of dollars. This initiative demonstrates that aligning technical infrastructure with financial governance is essential for maintaining growth without unnecessary waste. **The Central Team and Data-Driven Governance** * Coupang formed a specialized Central team consisting of infrastructure engineers and technical program managers to identify efficiency opportunities across the organization. * The team developed custom BI dashboards utilizing Amazon CloudWatch, AWS Cost and Usage Reports (CUR), and Amazon Athena to provide domain teams with actionable insights into their spending. * The finance department partnered with engineering to enforce strict budget compliance, ensuring that domain teams managed their resources within assigned monthly and quarterly limits. **Strategies for Spending and Paying Less** * The company implemented "Spending Less" strategies by automating the launch of resources in non-production environments only when needed, resulting in a 25% cost reduction for those areas. * "Paying Less" initiatives focused on rightsizing, where the Central team worked with domain owners to manually identify and eliminate unutilized or underutilized EC2 resources. * Workloads were migrated to more efficient hardware and pricing models, specifically leveraging ARM-based AWS Graviton processors and AWS Spot Instances for data processing and storage. **Targeted Infrastructure Optimization** * Engineering teams focused on instance generation alignment, ensuring that services were running on the most cost-effective hardware generations available. * Storage costs were reduced by optimizing Amazon S3 structures at rest, improving how data is organized and stored. * The team refined Amazon EMR (Elastic MapReduce) configurations to enhance processing efficiency, significantly lowering the cost of large-scale data analysis. To achieve sustainable cloud efficiency, engineering organizations should move beyond viewing cloud costs as a purely financial concern and instead treat resource management as a core technical metric. By integrating financial accountability directly into the engineering workflow through shared analytics and automated resource controls, companies can foster a culture of efficiency that supports long-term scalability.

AWSAmazon S3FinopsAmazon Athena+4

figma·Oct 9, 2024·2 min readCurated summary

The Search for Speed in Figma | Figma Blog

Figma’s search investigation revealed that OpenSearch itself was responsible for less than 30% of total search latency. The larger costs came from query construction and especially permission checks before and after searches. By measuring the correct end-to-end metrics, Figma identified misleading monitoring data and established a foundation for improving search performance at scale. ## Diagnosing the Latency Gap - Figma migrated from an older Elasticsearch version to AWS-managed OpenSearch, a fork created after Elasticsearch’s 2021 license change. - OpenSearch reported an average search time of roughly **8 ms**, while Figma’s API showed: - About **150 ms average latency** - **200–400 ms** latency at the 99th percentile - Minimum latency above **40 ms** - Search performance also varied significantly depending on traffic levels, with peak periods much slower than weekends. - Additional instrumentation showed that substantial time was spent both before and after the OpenSearch request. ## Understanding OpenSearch’s Metrics - OpenSearch distributes a query through a coordinator node to worker nodes, typically sending one request per index shard. - It then gathers, sorts, and fetches results during the query and fetch phases. - The reported 8 ms metric measured only the average time for individual shard queries—not the total time required to coordinate hundreds of shard requests. - Figma’s queries could involve as many as **500 shard-level requests**, many of which ran in parallel but not all. - OpenSearch did not provide built-in metrics or logs for overall query duration. - Figma instead extracted the `took` value from each search response, producing a backend latency measure that aligned more closely with application-level timing. ## Permission Processing as the Main Bottleneck - Less than 30% of total query API time was spent waiting for OpenSearch. - Pre-processing: - Retrieved information about files the user could access. - Built an OpenSearch filter intended to exclude inaccessible files. - Post-processing: - Performed additional permission checks on every returned file. - Was especially slow and consumed more time than the search itself. - The investigation demonstrated that optimizing the search engine alone would not solve Figma’s overall latency problem. Figma’s experience highlights the importance of measuring end-to-end request latency rather than relying on subsystem metrics. Accurate coordinator-level and application-level instrumentation is essential, particularly when distributed searches involve many shards and expensive authorization work.

Read original(opens in new tab)

AWSObservabilityElasticsearchOpensearch+2

figma·Sep 24, 2023·2 min readCurated summary

EU Hosting for Figma and FigJam Files | Figma Blog

Figma announced EU hosting for Figma and FigJam file data, giving European organizations more control over data location while retaining the benefits of its cloud platform. The service is intended to address regional privacy and regulatory concerns and is available exclusively to Enterprise customers. File contents are hosted primarily in Frankfurt, with backups in Dublin, though some metadata, login information, and temporary in-transit data remain outside the EU. ## EU Data Hosting Availability - Customers can host the document contents of Figma and FigJam files within the EU. - The option is available only on Figma’s Enterprise plan. - Existing customers can migrate files from the US after upgrading to Enterprise. - Account managers coordinate migration dates, and eligible customers can join a migration waitlist. ## Hosting Locations and Infrastructure - Primary file storage is located in Frankfurt, Germany. - Backup data is stored in Dublin, Republic of Ireland. - Figma uses AWS as its cloud infrastructure provider. ## Data Included and Excluded - EU hosting initially covers the contents of supported Figma and FigJam files. - User metadata and login data are not hosted within the EU. - Some data moving through Figma’s systems may be temporarily stored outside the EU. - Customers should consult Figma’s file-type breakdown to determine which data is covered. ## Security and Compliance Figma positions EU hosting as an addition to its existing security and privacy measures, including: - SOC 2 Type II reporting - ISO 27018 and ISO 27001 certifications - Level 2 adherence to the EU Cloud Code of Conduct The feature is particularly relevant to organizations such as Volkswagen Group, which require stronger control over data residency for regulatory and data-protection reasons. Organizations with strict EU residency requirements should evaluate which data types are covered and consider Figma Enterprise for localized file hosting.

Read original(opens in new tab)

AWSFigmaFigjamData Residency+3

datadog·Apr 13, 2022·3 min readCurated summary

It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking

A routine update to a critical metrics query service caused intermittent errors and increased latency. Although logs initially pointed to DNS failures, the investigation revealed a deeper networking problem involving dropped packets and saturated AWS VPC connection tracking. The incident highlighted how Kubernetes, Cilium, AWS networking, and DNS behavior can interact in ways that obscure the true cause. ## Initial Symptoms and Apparent DNS Failures - Errors increased whenever the metrics query service was rolled out. - The service retrieves data from metric stores for dashboards and monitor evaluations. - Automatic retries reduced user-facing failures but increased latency. - Service logs showed DNS errors when connecting to dependencies inside Kubernetes. - The investigation therefore began with the cluster’s DNS infrastructure. ## NodeLocal DNSCache Reaches Its Limits - NodeLocal DNSCache runs as a `node-local-dns` DaemonSet on every Kubernetes node. - DNS pods had: - A 64 MB memory limit - A `max_concurrent` limit of 1,000 requests - The pods experienced out-of-memory errors and rejected requests during rollouts. - Increasing memory to 256 MB stopped the OOM errors, but DNS failures continued. - Request volume was far below the expected capacity: - Normally about 400 queries per second - Nearly 2,000 queries per second during rollouts - Expected capacity of at least 200,000 queries per second - Upstream resolvers were marked unhealthy, suggesting that NodeLocal DNSCache could not establish or maintain connections. - Because upstream requests could wait up to five seconds, connection failures consumed concurrency slots and made the cache appear overloaded. ## Evidence of a Network Problem - The instances were below their 5-Gbps sustained throughput limits. - TCP retransmits increased in correlation with service rollouts. - Engineers suspected brief traffic spikes, or microbursts, that were not visible in aggregate throughput metrics. - This shifted the investigation from DNS configuration toward lower-level AWS networking behavior. ## AWS VPC Connection Tracking - ENA metrics revealed a significant increase in `conntrack_allowance_exceeded`. - This metric counts packets dropped when VPC connection tracking becomes saturated. - Connection tracking maintains state for network flows and supports features such as stateful EC2 security groups. - The infrastructure used two tracking layers: - VPC conntrack maintained at the hypervisor level - Linux conntrack inside each instance - VPC conntrack appeared saturated even though Linux conntrack contained fewer than 60,000 entries—well within the observed capacity of similar instances. - AWS Support confirmed that conntrack capacity varies by instance type and that VPC conntrack limits could differ substantially from Linux conntrack limits. - Scaling to larger instances resolved the symptoms, but the engineers wanted to understand the traffic pattern and find a more efficient long-term solution. ## VPC Flow Logs as the Next Investigation Tool - The team turned to Amazon VPC Flow Logs to examine the service’s low-level network behavior. - These logs were expected to clarify why connection tracking filled up and how rollout traffic contributed to the saturation. - The investigation was still ongoing at the point where the provided article excerpt ends.

Read original(opens in new tab)

AWSDatabase DesignKubernetesPostgreSQL+4

figma·Nov 20, 2019·3 min readCurated summary

Figma’s infrastructure: What goes into powering a web-based design tool | Figma Blog

Figma’s infrastructure team is focused on making a web-based design tool feel as fast and reliable as a desktop application. Its cloud-based, collaborative model creates demanding networking, storage, and scaling challenges, especially as organizations grow. The company is therefore evolving from a simple but operationally intensive backend toward a more mature, horizontally scalable architecture. ## Figma’s Collaborative Design Model - Figma stores designs in the cloud, giving each file a unique URL and making it a shared source of truth. - Users can collaborate, comment, prototype, and support developer handoff without exporting files or managing separate versions. - Its multiplayer functionality allows multiple people to view and edit files simultaneously. - Files may contain complex shapes and large images, creating significant data-transfer demands. - Users accustomed to desktop design tools expect fast interaction despite Figma running in a browser. ## Limits of the Original Infrastructure - Figma’s early backend favored simple strategies that worked well for smaller teams. - The application initially loaded all shared components available to a user when opening a file. - This approach became increasingly inefficient as organizations accumulated nearly 10,000 shared design elements. - Growth among large customers and the broader user base exposed the limits of the original design. ## Reducing Interaction Latency - Figma still preloads more data than users immediately need, increasing backend load and slowing startup. - Fixing this requires more than backend optimization: the client-server interaction model must be redesigned. - The client should request information incrementally, only when it is needed. - This change also requires coordination with product teams because existing user experiences depend on preloaded data. ## Building a Horizontally Scalable Database - At the time of the article, Figma relied on a single powerful AWS database instance. - The simple architecture reflected the company’s preference for the KISS principle and had supported substantial growth. - As usage increased, the database approached its capacity limits. - Figma planned to replace it with a database layer capable of scaling horizontally, a difficult transition because nearly every system depends on the database. ## Improving International Performance - More than 80% of Figma’s weekly active users were located outside the United States. - Their latency was affected by the round-trip distance to Figma’s Oregon datacenter. - Figma planned to move selected infrastructure components closer to users around the world. - The initial step was to deploy strategically placed remote proxies globally. Figma’s broader infrastructure strategy is to preserve the simplicity that helped it move quickly while introducing selective caching, better data loading, geographic distribution, and horizontal scaling where growth demands it.

Read original(opens in new tab)

AWSReal Time CollaborationDistributed SystemsPerformance Optimization+2

datadog·Apr 2, 2019·3 min readCurated summary

Building highly reliable data pipelines at Datadog

Datadog’s approach to reliable data pipelines focuses on delivering correct data on time, even when individual jobs fail. Reliability therefore requires fault tolerance, monitoring, and fast recovery rather than eliminating every failure. The company achieves this through isolated, short-lived clusters and pipelines designed to limit the impact of failures. ## Reliability Means Timely, Correct Results - A reliable pipeline is one that consistently produces correct outputs within the required time window. - Occasional crashes do not necessarily make a pipeline unreliable if automatic recovery still delivers the data on schedule. - Pipelines should be designed with the expectation that failures will eventually occur. - Monitoring must detect unexpected failures early, while operational processes should support rapid recovery. ## Architecture for Batch Pipelines - Datadog streams and analyzes live data in real time but uses batch pipelines for features such as optimized long-term storage. - Historical data is stored in object storage. - Cloud Hadoop/Spark services launch and configure processing clusters. - Luigi workers manage tasks and workflows, while Spark workers compile code and submit jobs. - Jobs can be launched through a web interface, command line, or scheduler. ## One Cluster per Pipeline Instead of placing all workloads on one large Hadoop cluster, Datadog gives each pipeline its own cluster. - **Isolation:** Jobs do not compete for resources or interfere with one another, simplifying monitoring and diagnosis. - **Workload-specific hardware:** Clusters can use CPU-optimized or memory-optimized instances depending on the job. - **Elastic scaling:** Clusters can be expanded to catch up with delays or handle growing data volumes without waiting for a shared cluster. - **Safer upgrades:** Hadoop and Spark versions can be upgraded gradually across separate clusters. - Clusters are typically short-lived, averaging about three hours, although dozens may run simultaneously. ## Using Spot Instances to Encourage Fault Tolerance - AWS spot instances can reduce infrastructure costs by as much as 80%, but their nodes may be terminated whenever capacity or demand changes. - Rather than avoiding this failure mode, Datadog designs pipelines to tolerate disappearing clusters. - Long-running jobs are risky because failures discard more work and make recovery slower. - Pipelines are split into smaller jobs: - **Vertically:** Separate transformations into multiple stages, persisting intermediate results in S3. - **Horizontally:** Partition input data so multiple jobs process different portions concurrently. ## Breaking Up the Rollup Pipeline - Datadog’s rollup pipeline generates aggregated time-series data for historical metrics queries. - A single job would take more than 14 hours, making failures costly and difficult to recover from. - The pipeline is divided into two stages: - Aggregate high-resolution data and checkpoint it to S3 as Parquet files. - Convert the intermediate data into a custom format optimized for queries. - As these jobs grew, they were partitioned further using Kafka’s partitioning scheme. - Kafka partitions are grouped into shards, allowing Datadog to: - Adjust how much data each job processes. - Run more or fewer jobs as needed. - Isolate unusually large or sensitive shards. - This decomposition adds overhead because launching jobs and checkpointing to S3 take extra time, but it substantially limits the work lost during failures. ## Practical Recommendation Design pipelines around failure rather than assuming uninterrupted execution. Use isolated, scalable clusters, short jobs, intermediate checkpoints, and partitioned processing so that failures affect only a small portion of the workload and recovery remains fast.

Read original(opens in new tab)

AWSDatabase DesignPostgreSQLMonitoring+4