Cloud Infrastructure | Techlist.io

awsApr 28, 20263 min readCurated summary

Top announcements of the What’s Next with AWS, 2026 | Amazon Web Services

The 2026 “What’s Next with AWS” event focused on how AI agents are reshaping business operations. Major announcements included Amazon Quick, an AI work assistant; four specialized Amazon Connect solutions; and an expanded AWS–OpenAI partnership bringing OpenAI models and Codex to Amazon Bedrock. Together, these offerings emphasize integrated agents that can connect to existing systems, make decisions, and execute tasks within enterprise-controlled infrastructure. ## Amazon Quick Becomes a Broader AI Work Assistant - Amazon Quick connects to workplace information, learns user preferences, and takes action on users’ behalf. - A new desktop app, currently in preview, can access local files, calendars, and communications without requiring a browser. - Free and Plus plans are available without an AWS account. Users can register with a personal email or Google, Apple, GitHub, or Amazon credentials. - Quick can generate documents, presentations, infographics, and images directly within chat. - New native integrations include Google Workspace, Zoom, Airtable, Dropbox, and Microsoft Teams. ## Amazon Connect Expands into Four Agentic AI Products AWS is repositioning Amazon Connect as a portfolio of solutions for specific business workflows: - **Amazon Connect Decisions:** A supply-chain planning and intelligence platform using AI teammates, Amazon’s operational expertise, and more than 25 specialized tools to support proactive planning. - **Amazon Connect Talent:** A hiring solution in preview that provides AI-led interviews, science-backed assessments, and standardized evaluations for large-scale recruiting. - **Amazon Connect Customer:** The renamed customer-experience product, supporting voice, chat, and digital channels. New configuration tools aim to let organizations deploy conversational AI in weeks rather than months. - **Amazon Connect Health:** Automates patient verification, appointments, patient insights, ambient documentation, and medical coding to improve access to care and reduce administrative workloads. ## AWS and OpenAI Expand Their Partnership The companies announced several limited-preview offerings that bring OpenAI capabilities into AWS environments: - **OpenAI models on Amazon Bedrock:** Models including GPT-5.5 and GPT-5.4 will be accessible through existing Bedrock APIs, with AWS security, governance, and cost controls. - **Codex on Amazon Bedrock:** Organizations can run OpenAI’s coding agent using AWS credentials and infrastructure, with usage counting toward AWS cloud commitments. Initial access includes the Codex CLI, desktop app, and Visual Studio Code extension. - **Bedrock Managed Agents powered by OpenAI:** This service combines OpenAI models with AWS-managed infrastructure and the OpenAI harness for building production-ready agents capable of reasoning through long-running tasks. AWS’s announcements point toward a future in which AI agents are embedded directly into workplace tools, operational systems, customer-service platforms, and cloud development environments. Organizations looking to adopt these capabilities should evaluate the available previews, integrations, governance controls, and workflow fit before moving to production.

Read original

Cloud Infrastructure AWS Openai Codex+4

lineMar 24, 20264 min readCurated summary

How We Built an SRE Bot That Reduced Our Team’s Repetitive Work by 90%

LINE Home DevOps created an SRE bot to reduce the repetitive work caused by growing services, Flava cloud migration, and increasing developer requests. By making Slack the central interface and automating Jira, Confluence, and workflow updates, the team reduced deployment-request handling from roughly 30 minutes to under one minute. The bot also improved tracking, consistency, and response speed, helping SREs move away from constant firefighting. ## Repetitive SRE Work and Its Costs - Developers frequently asked how to inspect Flava pod logs, request permissions, interpret errors, and access staging environments. - Deployment requests required manual movement between Slack, Confluence, and Jira: - Finding release checklists - Copying information into Jira - Creating missing Fix Versions - Linking Epics and active sprints - Sharing ticket links and deployment documentation - Each deployment request previously took about 30 minutes to an hour. - Manual processing caused omissions and mistakes, especially during urgent releases. - General requests were buried in Slack mentions, making ownership and completion status difficult to track. - Measurement showed that each SRE spent nearly half a day per week on repetitive work. ## Slack-Centered Automation The team adopted the principle that developers should only need Slack, while SREs should be able to manage work with a few clicks. - **Slack as the single source of truth:** Requests begin and remain trackable in Slack. - **Zero manual work:** Rule-based Jira and documentation tasks are automated. - **Immediate visibility:** Status changes and results are posted to Slack in real time. - **Permission control:** Only authorized SRE members can claim or complete requests. ## Key Technical Decisions ### Slack Workflows Instead of Slash Commands - Slash commands are easy to implement but depend on users entering correctly formatted text. - Slack Workflows provide structured forms with required-field validation. - Because Workflows are native Slack functionality, the team avoided building a separate user interface. - The lower usage barrier made adoption more likely. ### Asynchronous Processing - Slack requires event responses within three seconds. - Sequential calls to Jira, Confluence, and other APIs could exceed that limit. - The bot immediately acknowledges the request, then performs external work in the background. - Successes and failures are reported in the Slack thread, keeping processing transparent. ### Redis-Based State Management - In-memory state would be lost whenever the bot restarted. - Slack metadata APIs were considered too slow for real-time interactions such as emoji clicks. - Redis was selected for sub-100-millisecond lookups and persistent state. - A 30-day TTL limits stale data. - Redis transactions using `WATCH/MULTI/EXEC` ensure consistent updates when multiple SREs interact simultaneously. ### Hexagonal Architecture - The bot uses ports and adapters to isolate business logic from external systems. - The architecture separates: - Inbound Slack event adapters - Application use cases and business logic - Outbound Jira, Confluence, and Redis adapters - External API or SDK changes can be handled without modifying core business logic. - This structure also makes testing and future feature development easier. ## Automated Request Scenarios ### Deployment Requests - Developers submit required project, release-version, checklist, and other details through a Slack Workflow. - The bot automatically: - Creates a missing Jira Fix Version - Creates and configures the Jira ticket - Links the Epic - Adds the ticket to the active sprint - Finds the relevant deployment manual - Posts the result to the Slack thread - An SRE can click 👀 to claim the work. - Clicking ✅ completes the Jira ticket and posts a completion notification. - SRE effort falls from about 30 minutes to under one minute, with minimal risk of missing required fields. ### Emergency Deployments - Selecting an urgent request automatically sets Jira Priority to `Highest`. - The bot immediately announces the request in Slack. - An SRE can claim it with 👀, perform the deployment, and complete it with ✅. - The process reduces delays from roughly 30–40 minutes to about one minute. ### General SRE Requests - Requests such as production-access permissions are submitted through a structured Slack Workflow. - The bot creates a Jira ticket, links the Epic, assigns the active sprint, and sets an appropriate priority. - Slack retains the ticket link and status, eliminating the need to search through message history later. - SREs claim and complete the request using the same emoji-based workflow. The main recommendation is to automate repetitive, rule-based operations at the point where requests already occur. A Slack-centered, asynchronous bot with durable state and clean system boundaries can reduce manual effort while making ownership, progress, and completion visible to everyone.

Read original

Cloud Infrastructure Database Design Slack Jira+4

figmaMar 17, 20262 min readCurated summary

Design’s Influence Is Expanding, and Here’s Why That Feels Hard | Figma Blog

Design is expanding into more products, interactions, and strategic decisions, especially as AI introduces new software categories and interfaces. Although AI makes design work faster, it also increases output, expectations, and workload rather than reducing effort. This leaves designers divided: the field is growing, but many are unsure whether it is improving. ## Design’s Expanding Influence - Each technological shift—from graphical interfaces to the web and mobile apps—has increased design’s scope. - AI is creating new categories such as agent orchestration systems and answer engines. - Existing products are gaining generative, conversational, and predictive features. - Users now interact through prompts, speech, and image uploads, creating new design challenges: - Translating ambiguous input into clear intent - Making automated experiences understandable and human - Designing beyond traditional screen-by-screen navigation - Survey results show mixed sentiment: - 36% of designers think the profession has improved - 35% think it has worsened - 29% see no change - Meanwhile, 82% of hiring managers say demand for designers has increased or remained steady, though only 20% believe the industry itself is improving. ## AI Expands the Work - AI helps teams address new design problems more quickly, but it does not necessarily reduce the amount of work. - Product builders reported a 17.5% year-over-year increase in the number of tasks they perform. - Research from UC Berkeley found that AI users work faster while also taking on more tasks and working longer hours. - Workers often feel more productive without feeling less busy. ## The Jevons Paradox in Design - As AI makes creation cheaper and easier, teams produce more designs, explore more options, and iterate more deeply. - This follows the Jevons Paradox: efficiency increases can lead to greater overall consumption rather than reduced consumption. - Software development experienced a similar pattern when cloud infrastructure made releases easier, resulting in more frequent releases and redesigns. - AI has changed the rhythm and volume of design work rather than eliminating it. Designers should view AI as a force multiplier, not a shortcut to less work. Its benefits will depend on managing rising expectations and workload while developing clearer approaches to complex, automated interactions.

Read original

Cloud Infrastructure AI Generative AI User Experience Design+3

airbnbFeb 11, 20263 min readCurated summary

My Journey to Airbnb — Anna Sulkina

Anna Sulkina’s career journey moved from hardware diagnostics and frontend development into backend infrastructure and engineering leadership. Her experiences at Twitter taught her to design distributed systems for failure and to build consensus around transformative technologies like GraphQL. She joined Airbnb in 2022 because it aligned her passion for travel with an opportunity to strengthen developer infrastructure, organizational strategy, and engineering collaboration. ## Discovering Technology in Post-Soviet Ukraine - Sulkina grew up in Eastern Ukraine as the Soviet Union collapsed. - Her older brother introduced her to computers by bringing home hardware components and assembling a machine that loaded programs from a cassette player. - Seeing how individual components formed a working system inspired her to pursue technology. ## Learning English While Building Technical Skills - She studied programming at a Ukrainian university before immigrating to the United States. - Although she understood written English and knew how to program, communicating in English was initially more difficult than learning programming languages. - She took ESL classes while studying C++ and Java through Berkeley Extension. - Her first job was in hardware diagnostics at a five-person company. - A language barrier caused her to run out of time on a technical interview, but an interviewer familiar with her Berkeley class gave her another opportunity. - She eventually transitioned from C++ to Java, which became her primary language for many years. ## Moving Down the Stack and Into Leadership - Sulkina’s career progressed from hardware diagnostics to frontend, backend, and infrastructure engineering. - At the same time, she increasingly took on leadership responsibilities. - At Caymas Systems, her manager recognized her leadership potential and showed her the difference effective leadership makes. - At Comcast, she moved from individual contributor to engineering manager. - Coaching engineers, building software collaboratively, and developing high-performing teams convinced her that leadership was the right path. ## Lessons from Twitter’s Distributed Systems - During nearly nine years at Twitter, Sulkina advanced from first-line manager to director. - She worked through major operational events, including the “fail whale” period and the tweetstorm surrounding Ellen DeGeneres’s viral selfie. - Twitter’s transition from a monolith to microservices taught her that failure is inevitable in complex systems. - Resilient distributed systems must be designed to handle failures rather than assuming failures can be prevented. - Her cultural lesson involved turning promising ideas into adopted technologies. - She helped bootstrap Twitter’s GraphQL API, replacing legacy REST services. - The effort required leadership support, cross-team consensus, and stakeholder alignment, but ultimately improved product teams’ development velocity. ## Choosing Airbnb - Airbnb contacted Sulkina in 2022, when she felt ready to move beyond a well-established organization at Twitter. - The company appealed to her because it combined her professional interests with her personal passion for travel; she had been an Airbnb guest since 2013. - Airbnb’s Developer Platform organization had strong work happening in separate silos but needed clearer strategy, direction, and trust across engineering. - Sulkina began by clarifying the organization’s purpose and future direction. - Her early priorities included strengthening the organization, coaching leaders, and creating alignment within the team and with the teams it supported. - Over the following years, this work produced a high-performing organization with clearer strategy, stronger execution, and a focus on delivering business value. Sulkina’s story emphasizes that technical growth, organizational leadership, and personal motivation can reinforce one another. Her experience suggests that successful engineering leaders design for failure, invest in alignment, and use clear strategy to turn fragmented efforts into meaningful platform-wide impact.

Read original

Cloud Infrastructure Distributed Systems Microservices Java+3

gitlabFeb 10, 2026Original article

DevSecOps-as-a-Service on Oracle Cloud Infrastructure by Data Intensity (opens in new tab)

Data Intensity’s DevSecOps-as-a-Service provides a solution for organizations that require the granular control of GitLab Self-Managed but wish to eliminate the operational burden of infrastructure maintenance. By hosting dedicated GitLab instances on Oracle Cloud Infrastructure (OCI), the service combines the security and customization of a self-managed environment with the convenience of a fully managed platform. This partnership enables teams to focus on software delivery while leveraging expert management for high availability and disaster recovery. ### The Benefits of GitLab Self-Managed * Offers complete ownership of data residency and instance configuration to meet strict regulatory and compliance requirements. * Enables deep customization and integration possibilities that are often restricted in standard SaaS environments. * Addresses the challenges of manual server management, upgrades, and high-availability scaling by offloading these tasks to a managed provider. ### Managed Service Features and Support * Provides 24/7 monitoring, alarming, and expert technical support for standalone GitLab instances. * Includes scheduled quarterly patching performed during customer-specified maintenance windows to minimize disruption. * Ensures business continuity through automated backups and professional disaster recovery protection. * Utilizes tiered architectures designed to scale based on specific user capacities and recovery time objectives. ### Infrastructure Optimization via OCI * Delivers significant cost efficiency, with organizations typically realizing 40-50% reductions in infrastructure spending compared to other hyperscalers. * Supports diverse deployment models, including Public Cloud, Government Cloud, EU Sovereign Clouds, and dedicated infrastructure behind a corporate firewall. * Maintains consistent pricing and operational tooling across hybrid, global, and regulated environments. ### Implementation and Migration * Data Intensity offers optional migration services to transition existing code repositories and configurations to the OCI environment seamlessly. * The service is specifically designed for organizations with predictable cost requirements and those lacking in-house infrastructure expertise. * Deployment planning involves tailored consultations to match specific compliance and data residency needs with OCI’s global region availability. This managed service is a recommended path for enterprise teams that need to prioritize data sovereignty and flexibility without sacrificing the speed of a turnkey solution. Organizations currently using or planning to adopt OCI can leverage this service to standardize their DevSecOps workflows while achieving significant infrastructure savings.

Cloud Infrastructure Gitlab DevSecOps Disaster Recovery+4

awsFeb 4, 20262 min readCurated summary

Amazon EC2 C8id, M8id, and R8id instances with up to 22.8 TB local NVMe storage are generally available | Amazon Web Services

Amazon has generally released the EC2 C8id, M8id, and R8id instances, combining custom Intel Xeon 6 processors with up to 22.8 TB of local NVMe SSD storage. Compared with prior sixth-generation instances, they provide up to 43% more compute performance, 3.3× higher memory bandwidth, and significant gains for I/O-intensive databases and analytics. The instances target compute-heavy, balanced, and memory-intensive workloads respectively. ## Instance Families and Workloads - **C8id:** Designed for compute-intensive applications such as video encoding, image processing, and media workloads requiring fast local storage. - **M8id:** Balances compute and memory for data logging, media processing, and medium-sized data stores. - **R8id:** Targets memory-intensive workloads, including large SQL/NoSQL databases, in-memory databases, analytics, and AI inference. ## Capacity and Performance - Scale up to **96xlarge**, compared with 32xlarge in the previous generation. - Offer up to: - **384 vCPUs** - **3 TiB memory** - **22.8 TB local NVMe storage** - Available in **metal-48xl** and **metal-96xl** configurations for workloads needing direct physical-resource access. - Deliver up to: - **43% higher compute performance** - **3.3× greater memory bandwidth** - **46% better I/O-intensive database performance** - **30% faster I/O-intensive real-time analytics queries** ## Networking, Storage, and Compatibility - Support **Instance Bandwidth Configuration**, allowing network or EBS bandwidth to be increased by up to 25% depending on workload needs. - Use sixth-generation AWS Nitro cards to offload virtualization, storage, and networking operations. - Require AMIs with **ENA and NVMe drivers**; current AWS Windows and Linux AMIs include the NVMe driver by default. - Local NVMe devices appear automatically after boot and do not require block device mappings. - Storage is hardware-encrypted with **XTS-AES-256** and unique keys. - Local NVMe storage is temporary: it is destroyed when the instance is stopped or terminated. ## Availability and Purchasing - Available in US East (N. Virginia), US East (Ohio), and US West (Oregon). - R8id instances are also available in Europe (Frankfurt). - Offered as On-Demand, Savings Plans, Spot Instances, Dedicated Instances, and Dedicated Hosts. These instances are best suited to applications that can exploit high-performance, ephemeral local NVMe storage; persistent data should remain on services such as Amazon EBS.

Read original

Cloud Infrastructure Amazon Ec2 Ai Inference Intel Xeon 6+4

awsJan 15, 2026Original article

Opening the AWS European Sovereign Cloud (opens in new tab)

AWS has officially launched the AWS European Sovereign Cloud, a specialized infrastructure designed to meet the rigorous data residency and operational autonomy requirements of European public sector organizations and highly regulated industries. This new offering provides a fully featured cloud environment that is physically and logically separate from existing AWS Regions, ensuring all data and metadata remain entirely within the European Union. By bridging the gap between legacy on-premises security and modern cloud innovation, AWS enables sensitive workloads to operate under strict European jurisdiction and independent governance. **Strategic Independence and Operational Control** Organizations in the EU often face complex regulatory hurdles that prevent them from using standard public cloud offerings, frequently forcing them to remain on aging on-premises hardware. The AWS European Sovereign Cloud addresses these challenges through: * **Independent Operations:** The infrastructure is operated independently from other AWS Regions, providing a distinct management layer specific to the EU. * **Enhanced Sovereignty Controls:** Robust technical controls and legal protections are integrated to ensure that data remains under European jurisdiction. * **Governance Autonomy:** The cloud is built to provide European entities with full control over their data residency and operational transparency. **Independent Infrastructure and Regional Presence** The architecture is designed for high availability and resilience, ensuring that mission-critical services remain functional regardless of external connectivity. * **Initial Region:** The first region is now generally available in Brandenburg, Germany, serving as the primary hub for the sovereign infrastructure. * **Redundancy:** The infrastructure utilizes multiple Availability Zones with redundant power and networking to maintain continuous operation. * **Isolated Connectivity:** The design allows the cloud to continue operating even if connectivity to the rest of the global AWS network is interrupted. **Expansion and Hybrid Deployment Options** To support the diverse needs of EU member states, AWS is expanding the footprint of this sovereign infrastructure through localized hardware and edge services. * **Sovereign Local Zones:** Future expansion plans include new Local Zones in Belgium, the Netherlands, and Portugal to provide low-latency access within specific borders. * **Hybrid Integration:** Customers can extend sovereign infrastructure to their own data centers using AWS Outposts or AWS Dedicated Local Zones. * **Advanced Capabilities:** The platform supports specialized workloads through AWS AI Factories, allowing regulated industries to leverage artificial intelligence within a sovereign boundary. For European organizations navigating strict compliance landscapes, the AWS European Sovereign Cloud provides a viable path to digital transformation. Decision-makers should evaluate their current on-premises or restricted cloud environments to determine how these new sovereign regions and local zones can fulfill upcoming data residency mandates while providing access to advanced cloud-native services.

Cloud Infrastructure Data Residency Aws European Sovereign Cloud Digital Sovereignty+4

stripeDec 18, 20254 min readCurated summary

Stripe Atlas startups in 2025: Year in review

In 2025, early-stage startups launched faster, reached revenue sooner, and sold internationally from the beginning. Stripe Atlas data shows that these gains occurred despite a smaller share of startups raising funding, suggesting improved infrastructure and execution—not just venture capital—are driving growth. Founders are also increasingly building AI companies, particularly AI-agent businesses, although the post ends before fully explaining that shift. ## A More Global Startup Ecosystem - Atlas incorporations represented 169 countries in 2025, up from 158 in 2024. - European incorporations grew 48%, especially in the UK, France, and Germany, as founders sought access to US capital markets. - While 56% of Atlas startups are US-based, founding teams are increasingly distributed: - 24% of teams with multiple founders span more than one country. - This is a 79% increase since 2017. - Common pairings include Canada–US, UK–US, and India–US. - Distributed teams form both through prior in-person relationships and entirely online professional connections. ## International Sales from Launch - The typical startup sold to customers in two countries during its first six months in 2025, compared with one country in previous years. - Startups at the 90th percentile reached 15 countries, up from 12 in 2024. - Examples include: - Rork, which reached 69 countries in its first month and generated $100,000 in five days. - Zeabur, which served developers in 46 countries. - Payment infrastructure, compliance tools, cloud services, translation APIs, and globally distributed founders have reduced the barriers to international expansion. - Selling globally is increasingly a default launch strategy rather than a post-product-market-fit phase. ## Revenue Arrives Faster - The share of Atlas startups gaining a first paying customer within 30 days rose from 8% in 2020 to 20% in 2025. - Among startups that began accepting payments within three months, median time to first payment fell from 38 to 34 days. - Atlas’s 2025 payment changes allowed founders to accept payments immediately after incorporation, avoiding lengthy EIN delays for non-US founders. - Median first-six-month revenue increased 39% year over year, indicating that faster monetization reflects stronger product shipping and customer acquisition as well as better infrastructure. ## More Startups Reach Significant Revenue - The number of startups reaching $100,000 in their first six months rose 56% from 2024. - These companies reached that milestone in 108 days, compared with 121 days previously. - The average startup acquired 242 customers in its first six months, more than 50% above the prior year. - Growth was strongest among top performers: - 10th-percentile startups generated 18% more revenue than comparable 2024 companies. - 90th-percentile startups generated 52% more. - The overall market improved, but the gap between breakout companies and average performers widened. ## AI Becomes a Dominant Startup Focus - The share of Atlas founders identifying their companies as AI startups grew from 15% in 2023 to 33% in 2024 and 42% in 2025. - AI adoption also expanded among LLCs, rising from 5% in 2023 to 22% in 2025. - This growth occurred alongside weaker early fundraising: - Only 2.2% of Atlas startups at least six months old raised funding within three months of incorporation, down from 3.1% in 2024. - Pre-seed deal volume was largely unchanged even as Delaware C-corp formations increased. - The supplied article indicates that founders are shifting attention toward AI agents rather than AI infrastructure or copilots, but the provided text ends before detailing that trend. ## Practical Conclusion Modern payments, cloud infrastructure, compliance tools, and developer platforms are enabling founders to launch, monetize, and expand globally within weeks. Early startups increasingly need to assume international reach from day one while focusing on rapid customer validation and efficient execution, particularly as funding becomes less automatic.

Read original

Cloud Infrastructure AI Agents Fintech No Code+4

awsDec 8, 2025Original article

AWS Weekly Roundup: AWS re:Invent keynote recap, on-demand videos, and more (December 8, 2025) (opens in new tab)

The December 8, 2025, AWS Weekly Roundup recaps the major themes from AWS re:Invent, signaling a significant industry transition from AI assistants to autonomous AI agents. While technical innovation in infrastructure remains a priority, the event underscored that developers remain at the heart of the AWS mission, empowered by new tools to automate complex tasks using natural language. This shift represents a "renaissance" in cloud computing, where purpose-built infrastructure is now designed to support the non-deterministic nature of agentic workloads. ## Community Recognition and the Now Go Build Award * Raphael Francis Quisumbing (Rafi) from the Philippines was honored with the Now Go Build Award, presented by Werner Vogels. * A veteran of the ecosystem, Quisumbing has served as an AWS Hero since 2015 and has co-led the AWS User Group Philippines for over a decade. * The recognition emphasizes AWS's continued focus on community dedication and the role of individual builders in empowering regional developer ecosystems. ## The Evolution from AI Assistants to Agents * AWS CEO Matt Garman identified AI agents as the next major inflection point for the industry, moving beyond simple chat interfaces to systems that perform tasks and automate workflows. * Dr. Swami Sivasubramanian highlighted a paradigm shift where natural language serves as the primary interface for describing complex goals. * These agents are designed to autonomously generate plans, write necessary code, and call various tools to execute complete solutions without constant human intervention. * AWS is prioritizing the development of production-ready infrastructure that is secure and scalable specifically to handle the "non-deterministic" behavior of these AI agents. ## Core Infrastructure and the Developer Renaissance * Despite the focus on AI, AWS reaffirmed that its core mission remains the "freedom to invent," keeping developers central to its 20-year strategy. * Leaders Peter DeSantis and Dave Brown reinforced that foundational attributes—security, availability, and performance—remain the non-negotiable pillars of the AWS cloud. * The integration of AI agents is framed as a way to finally realize material business returns on AI investments by moving from experimental use cases to automated business logic. To maximize the value of these updates, organizations should begin evaluating how to transition from simple LLM implementations to agentic frameworks that can execute end-to-end business processes. Reviewing the on-demand keynote sessions from re:Invent 2025 is recommended for technical teams looking to implement the latest secure, agent-ready infrastructure.

Cloud Infrastructure AI AI Agents Machine Learning+3

microsoftJul 18, 20243 min readCurated summary

Managed DevOps Pools – The Origin Story

Microsoft’s vast, diverse engineering organization had accumulated more than 5,000 self-hosted Azure DevOps pools, creating duplicated tooling, inconsistent reliability, security gaps, and compliance challenges. Its One Engineering System (1ES) team addressed this with 1ES Hosted Pools, a standardized service for flexible, secure, and scalable CI/CD infrastructure. Adoption reduced costs by more than 60%, cut remaining self-hosted pools to a few dozen, and eventually led to the external Managed DevOps Pools offering. ## The Scale and Challenges of Self-Hosted Infrastructure - Microsoft supports over 100,000 engineers across many businesses, programming languages, operating systems, hardware platforms, build engines, and test frameworks. - By 2021, teams had created: - More than 5,000 self-hosted Azure DevOps pools - Hundreds of thousands of agents - Teams needed capabilities unavailable from Microsoft-hosted agents, including: - Larger compute sizes - Private-network connectivity - Custom images - Stateful agents - Long-running tests - The decentralized approach caused: - Duplicate engineering effort - Uneven support and reliability - Poor resource utilization and higher costs - Inconsistent patching and security practices - Difficult and time-consuming compliance audits ## 1ES Hosted Pools - 1ES developed a standardized internal service for custom Azure DevOps infrastructure. - Teams could connect agents to private resources such as package registries, secret managers, and on-premises services. - They could bring custom images, using centrally maintained images as their base. - Business continuity features allowed backup pools and failover to other Azure regions. - Agents were stateless by default, but teams could reuse stateful agents for better performance through local caches. - Stateful agents were automatically recycled based on age or available disk space. - Teams could select Azure VM families and sizes suited to their workload. - Standby agents could be pre-warmed on schedules or automatically provisioned using historical demand. ## Operational and Business Benefits - **Lower costs:** Infrastructure bills fell by more than 60% through improved utilization, better SKU selection, and selective use of Azure Spot VMs. - **Faster development:** Teams spent less time maintaining CI/CD infrastructure and more time building products. - **Simpler compliance:** Standardized telemetry made audits easier and allowed security and compliance improvements to be deployed centrally. - **Greater mobility:** Developers changing teams no longer had to learn different infrastructure-management systems. - **Improved security:** Features such as Azure Confidential VMs, Trusted Launch, and Secure TPM became available across pools. - **Reduced fragmentation:** By 2024, Microsoft had reduced its remaining self-hosted pools from more than 5,000 to only a few dozen. ## From Internal Platform to Managed DevOps Pools - 1ES first built Hosted Pools as an internal “Host On Behalf Of” service to validate whether centralized management could reduce self-hosting. - Success inside Microsoft, combined with customer demand, led to the external **Managed DevOps Pools (MDP)** service. - Organizations using VM Scale Set agents or self-hosted agents can migrate to MDP to gain standardized scaling, security, compliance, and operational support. - The external offering initially does not include every feature available in 1ES Hosted Pools, though additional capabilities may be added later. Centralizing CI/CD infrastructure can eliminate redundant platform work while improving cost efficiency, security, compliance, and developer productivity. Managed DevOps Pools extends Microsoft’s internal solution to organizations facing similar self-hosting challenges.

Read original

Cloud Infrastructure CI/CD DevOps Azure+3

datadogJun 16, 20233 min readCurated summary

2023-03-08 incident: A deep dive into the platform-level recovery

Datadog’s March 8, 2023 outage removed 60% of its compute capacity, forcing teams to restore infrastructure in stages while accounting for regional and cloud-provider differences. In EU1, recovery depended on rebooting affected nodes, restoring Kubernetes control planes in a strict hierarchy, and gradually bringing application capacity back online. Scaling afterward exposed infrastructure limits that had not been considered during normal operations. ## EU1 Platform Recovery - A system patch disconnected affected EU1 nodes from the network, but the nodes could be recovered through reboots. - Recovery was initially slowed by the lack of observability and unavailable Kubernetes APIs. - Datadog operates: - **Parent clusters**, which host the control-plane pods for other clusters. - **Child clusters**, where Datadog applications run. - This hierarchy allows Datadog to use Kubernetes deployment, replacement, rolling-update, and autoscaling capabilities for child-cluster control planes. - Parent-cluster control planes run on VMs and are managed with `systemd`. ## Restoring Kubernetes Clusters Because both parent and child environments were affected by the Ubuntu 22.04 issue, recovery had to follow a strict sequence: - **Parent control planes:** Nodes running Cilium were rebooted to restore network connectivity. This finished by 08:45 UTC. - **Child control planes:** All parent-cluster nodes hosting child control-plane pods were rebooted. This finished by 09:30 UTC. - **Application nodes:** Thousands of instances across dozens of child clusters were restarted. - Recovery reached 60% by 10:20 UTC. - All application nodes were restored by 12:05 UTC. - Restarts were prioritized by workload importance and paced to avoid overwhelming Kubernetes control planes. ## Scaling Capacity and Recovering Backlogs After restoring the clusters, Datadog needed substantial additional capacity to process data buffered during the outage. - EU1 hit a Google Cloud mesh limit of **15,500 VM instances** at 14:18 UTC. - Instance creation failures became apparent around 15:00 UTC. - Datadog had not checked this documented limit before the incident, but Google Cloud quickly raised it after Datadog submitted a high-priority request. - Autoscaling also exhausted the IP capacity of subnets used by three log- and trace-processing clusters. - These clusters normally used about 35–45% of their IP capacity, but the backlog caused autoscaling to request more than twice their usual replica counts, filling the subnets. ## Practical Lessons The recovery demonstrated that restoring compute capacity is not enough: teams must also understand dependency order, control-plane architecture, cloud-provider quotas, and network-address limits. Capacity planning should account for severe backlog-driven scaling, not just normal operating utilization, and documented infrastructure limits should be validated before emergencies occur.

Read original

Cloud Infrastructure Kubernetes Observability Cloud Computing+4

datadogJun 1, 2023Original article

2023-03-08 incident: A deep dive into our incident response | Datadog (opens in new tab)

Datadog’s first global outage on March 8, 2023, served as a rigorous stress test for their established incident response framework and "you build it, you own it" philosophy. While the outage was triggered by a systemic failure during a routine systemd upgrade, the company's commitment to blameless culture and decentralized engineering autonomy allowed hundreds of responders to coordinate a complex recovery across multiple regions. Ultimately, the event validated their investment in out-of-band monitoring and rigorous, bi-annual incident training as essential components for managing high-scale system disasters. ## Incident Response Structure and Philosophy * Datadog employs a decentralized "you build it, you own it" model where individual engineering teams are responsible for the 24/7 health and monitoring of the services they build. * For high-severity incidents, a specialized rotation is paged, consisting of an Incident Commander to lead the response, a communications lead, and a customer liaison to manage external messaging. * The organization prioritizes "people over process," empowering engineers to use their judgment to find creative solutions rather than following rigid, pre-written playbooks that may not apply to unprecedented failures. * A blameless culture is strictly maintained across all levels of the company, ensuring that post-incident investigations focus on systemic improvements rather than assigning fault to individuals. ## Multi-Layered Monitoring Strategy * Standard telemetry provides internal visibility, but Datadog also maintains "out-of-band" monitoring that operates completely outside its own infrastructure. * This out-of-band system interacts with Datadog APIs exactly like a customer would, ensuring that engineers are alerted even if the internal monitoring platform itself becomes unavailable. * Communication is streamlined through a dedicated Slack incident app that automatically generates coordination channels, providing situational awareness to any engineer who joins the effort. ## Anatomy of the March 8 Outage * The outage began at 06:00 UTC, triggered by a systemd upgrade that caused widespread Kubernetes failures and prevented pods from restarting correctly. * The global nature of the outage was diagnosed within 32 minutes of the initial monitoring alerts, leading to the activation of executive on-calls and the customer support management team. * Responders identified "unattended upgrades" as the incident trigger approximately five and a half hours after the initial failure. * Recovery was executed in stages: compute capacity was restored first in the EU1 region, followed by the US1 region, with full infrastructure restoration completed by 19:00 UTC. Organizations should treat incident response as a perishable skill that requires constant practice through a low threshold for declaring incidents and regular training. By combining out-of-band monitoring with a culture that empowers individual engineers to act autonomously during a crisis, teams can more effectively navigate the "not if, but when" reality of large-scale system failures.

Cloud Infrastructure Kubernetes Datadog Automation+4

datadogMay 24, 20233 min readCurated summary

2023-03-08 incident: A deep dive into the platform-level impact

Datadog’s March 8, 2023 outage was caused by an unexpected interaction between Ubuntu 22.04, systemd-networkd, and an automated security patch. A systemd change introduced behavior that flushed unfamiliar IP routing rules whenever systemd-networkd restarted; a CVE patch triggered that restart across many hosts. Because the patch was installed automatically and outside Datadog’s carefully staged deployment process, infrastructure across regions and cloud providers was affected simultaneously. ## A Systemd Behavior Change - systemd v248 introduced a systemd-networkd startup behavior that removed IP rules it did not recognize. - systemd v249 added the `ManageForeignRoutingPolicyRules` setting, which could disable this behavior, but the default configuration continued managing foreign rules. - These changes were backported to older systemd releases. - Ubuntu 20.04 used systemd v245, which did not flush IP rules during a systemd-networkd restart. - Ubuntu 22.04, adopted progressively by Datadog beginning in November 2022, used systemd v249 with the behavior enabled. Initially, the change caused no visible problems because systemd-networkd generally started only when new hosts were created, before Datadog’s custom routing rules existed. ## The Security Patch That Triggered the Problem - On March 7, 2023, Ubuntu released systemd patch `249.11-0ubuntu3.7` for a CVE. - Installing the patch restarted all systemd components, including systemd-networkd. - That restart caused systemd-networkd to flush routing policy rules on affected Ubuntu 22.04 hosts. - Ubuntu 20.04 hosts received a similar patch but were not affected because systemd v245 did not exhibit the problematic restart behavior. ## Unattended Upgrades Created Broad Exposure Datadog used Ubuntu’s default unattended-upgrade configuration: - Package metadata was downloaded twice daily using `apt-daily.timer`, with randomized delays of up to 12 hours. - Upgrades ran daily using `apt-daily-upgrade.timer`, between 06:00 and 07:00 UTC. - Only security updates and required dependencies were automatically installed. - Regular updates from the `-updates` repository were excluded. This meant many hosts automatically installed the systemd security patch during the same daily upgrade window. Not every host was affected: more than 90% of the fleet used Ubuntu 22.04, and some nodes had not yet downloaded the patch when their upgrade ran. ## Conflict with Datadog’s Deployment Process - Datadog normally updates nodes by replacing them automatically rather than relying on unattended upgrades. - Its standard process validates changes on experimental clusters, then progressively deploys them through staging and production. - Deployments are normally limited to selected clusters, availability zones, and regions before expanding. - The unattended systemd patch bypassed this process because it was installed directly on existing hosts. - As a result, a low-level networking change propagated across otherwise isolated regions and cloud providers at nearly the same time. Datadog’s experience demonstrates that even security-only automated updates can introduce coordinated infrastructure risk. Critical system packages should be tested and rolled out through the same staged process as other production changes, or their automated upgrades should be carefully constrained and monitored.

Read original

Cloud Infrastructure Kubernetes Systemd Ubuntu+4

datadogJan 22, 20193 min readCurated summary

Rethinking UX for AI-driven alerting

Alerting UX is shifting from manually configured static thresholds toward statistical systems that understand trends, seasonality, and group behavior. Forecasting, anomaly detection, and outlier detection reduce maintenance and nuisance alerts, while algorithmic feeds can discover problems users never explicitly configured. The post argues that supervised feeds—trained by user feedback—could become the most significant evolution in monitoring. ## Traditional Alerting UX Most alerts are built from four dimensions: - **Scope:** The hosts, containers, services, or other targets being monitored. - **Metric:** The value tracked, such as free disk space. - **Thresholds:** Static warning or critical values that trigger alerts. - **Time:** A duration or time window during which the condition must occur. Static threshold alerts remain common, with many taking the form: “If free disk space equals zero, alert.” ## Problems with Static Thresholds - Static thresholds do not adapt to system growth, changing workloads, or temporary events such as holiday traffic. - They require regular review and maintenance to remain useful. - Warning thresholds often serve as manual “heads-ups” so engineers can inspect a graph and estimate whether intervention is necessary. - Large numbers of warning alerts create false positives and alert fatigue. - Monitoring systems must be explicitly told which scopes and metrics to watch, leading to duplicated configuration and ongoing maintenance. ## Algorithmic Alerting Statistical alerting introduces three primary methods: - **Forecasting** - Uses historical data to predict when a metric will cross a threshold. - Changes “alert when disk reaches zero” into “alert if disk will reach zero within 24 hours.” - Lets teams specify how much remediation time they need. - Can eliminate separate warning thresholds because the forecast provides advance notice. - **Anomaly detection** - Predicts what should be happening now based on historical behavior. - Considers configurable confidence intervals and seasonality, such as daily or weekly patterns. - Alerts when current behavior deviates significantly from the expected range. - **Outlier detection** - Compares members of a group that should behave similarly. - Flags an individual server or service whose behavior differs from its peers. - Does not depend on historical behavior. These methods make thresholds and time behavior more flexible, but they still require users to define the metrics and scopes in advance. ## Algorithmic Feeds Algorithmic feeds apply similar statistical techniques without requiring detailed alert configuration. - They can monitor systems without predefined individual scopes or metrics. - They are especially useful for unpredictable anomalies and outliers. - Examples include Slack Highlights and Datadog Watchdog. - Feeds shift monitoring from **opt-in alerting**—where users specify what to watch—to discovering noteworthy activity automatically. - The post presents this shift as potentially the largest change in alerting UX, while noting that algorithmic feeds are still immature. ## Supervised Algorithmic Feeds Once a monitoring system generates a stream of events, user feedback can help train it to surface more relevant information. - The model is compared to social media feeds, where actions such as “likes” guide future recommendations. - This suggests a future in which engineers can teach monitoring systems which anomalies and events matter to them. - The provided excerpt ends while introducing this concept, so it does not describe the specific feedback mechanisms or implementation details. Monitoring is likely moving toward adaptive systems that combine statistical detection with user-guided prioritization. Teams should use forecasting and anomaly-based alerts where appropriate, while treating algorithmic feeds as a complementary way to discover issues outside manually configured monitoring.

Read original

Cloud Infrastructure Database Design Datadog Anomaly Detection+4