awsApr 14, 2026

AWS Interconnect is now generally available, with a new option to simplify last-mile connectivity | Amazon Web Services (opens in new tab)

AWS Interconnect is a managed service for private, high-speed connectivity between AWS and other clouds or on-premises networks. Its multicloud capability, now generally available, initially connects AWS with Google Cloud while Microsoft Azure support is planned for later in 2026. The service aims to replace complex VPN, colocation, and third-party networking setups with a turnkey, resilient configuration managed through AWS. ## AWS Interconnect Capabilities - **Interconnect – multicloud** connects an AWS VPC privately to VPCs on other cloud providers. - **Interconnect – last mile** simplifies connectivity from branch offices, data centers, and remote sites through existing network providers. - Both capabilities provide: - Dedicated bandwidth - Private connectivity - Managed provisioning - Reduced infrastructure and configuration overhead - Connections can be configured through the AWS Console by selecting the location or provider, AWS Region, and bandwidth. ## Multicloud Connectivity - The service provides a managed **Layer 3 connection** between AWS and another cloud provider. - Traffic uses the AWS global backbone and the partner’s private network rather than the public internet. - This improves: - Latency predictability - Throughput consistency - Isolation from internet congestion - Google Cloud is supported at launch; Microsoft Azure is expected later in 2026. ## Security, Resilience, and Monitoring - Physical links between AWS and partner routers use **IEEE 802.1AE MACsec encryption** by default. - Each cloud provider handles encryption on its own backbone, so customers must verify that the resulting deployment satisfies compliance requirements. - Connections use multiple logical links across at least two physical facilities to protect against device or facility failures. - Amazon CloudWatch integration includes: - A Network Synthetic Monitor for round-trip latency and packet loss - Bandwidth utilization metrics for capacity planning ## Open Partner Specification - AWS has published the underlying Interconnect specification on GitHub under the **Apache 2.0 license**. - Other cloud providers can become partners by implementing the specification and meeting AWS requirements for: - Resiliency - Support - Service-level agreements - Operational readiness ## Provisioning an AWS–Google Cloud Connection - The demonstration connects a single AWS VPC to a Google Cloud VPC using a Direct Connect Gateway. - In the AWS Direct Connect console, the user: - Selects Google Cloud as the provider - Chooses AWS Region `eu-central-1` - Chooses Google Cloud Region `europe-west3` - Specifies bandwidth - Selects a Direct Connect Gateway - Enters the Google Cloud project ID - AWS then generates an activation key for use on the Google Cloud side. ## Configuring Google Cloud - Because a Google Cloud web console option was unavailable at the time, the example uses the `gcloud` CLI. - The user creates a transport resource with: - The AWS activation key - The Google Cloud region - The target VPC network - Advertised AWS routes - After the transport reaches the appropriate state, the user creates a VPC peering connection between the Google Cloud VPC and the generated transport network. - Custom routes are imported and exported through the peering configuration. ## Completing the AWS Configuration - Once the Google Cloud transport and peering are configured: - The AWS Interconnect status can be checked in the Interconnect console. - The Direct Connect Gateway shows the new attachment. - The final AWS-side step is associating the gateway with the appropriate Virtual Private Gateway. - The Virtual Private Gateway must be in the same AWS Region as the Interconnect. - AWS routing still requires a final route entry so workloads can reach the remote Google Cloud network. AWS Interconnect is best suited to organizations operating hybrid or multicloud environments that want private, resilient connectivity without managing physical links or complex third-party networking. The managed provisioning process can reduce setup time to minutes, but teams should still validate routing, encryption responsibilities, regional constraints, and compliance requirements.

network-configurationgoogle-cloudcloud-monitoringaws-interconnect+4

datadogMay 24, 2023

2023-03-08 incident: A deep dive into the platform-level impact (opens in new tab)

Datadog’s March 8, 2023 outage was caused by an unexpected interaction between Ubuntu 22.04, systemd-networkd, and an automated security patch. A systemd change introduced behavior that flushed unfamiliar IP routing rules whenever systemd-networkd restarted; a CVE patch triggered that restart across many hosts. Because the patch was installed automatically and outside Datadog’s carefully staged deployment process, infrastructure across regions and cloud providers was affected simultaneously. ## A Systemd Behavior Change - systemd v248 introduced a systemd-networkd startup behavior that removed IP rules it did not recognize. - systemd v249 added the `ManageForeignRoutingPolicyRules` setting, which could disable this behavior, but the default configuration continued managing foreign rules. - These changes were backported to older systemd releases. - Ubuntu 20.04 used systemd v245, which did not flush IP rules during a systemd-networkd restart. - Ubuntu 22.04, adopted progressively by Datadog beginning in November 2022, used systemd v249 with the behavior enabled. Initially, the change caused no visible problems because systemd-networkd generally started only when new hosts were created, before Datadog’s custom routing rules existed. ## The Security Patch That Triggered the Problem - On March 7, 2023, Ubuntu released systemd patch `249.11-0ubuntu3.7` for a CVE. - Installing the patch restarted all systemd components, including systemd-networkd. - That restart caused systemd-networkd to flush routing policy rules on affected Ubuntu 22.04 hosts. - Ubuntu 20.04 hosts received a similar patch but were not affected because systemd v245 did not exhibit the problematic restart behavior. ## Unattended Upgrades Created Broad Exposure Datadog used Ubuntu’s default unattended-upgrade configuration: - Package metadata was downloaded twice daily using `apt-daily.timer`, with randomized delays of up to 12 hours. - Upgrades ran daily using `apt-daily-upgrade.timer`, between 06:00 and 07:00 UTC. - Only security updates and required dependencies were automatically installed. - Regular updates from the `-updates` repository were excluded. This meant many hosts automatically installed the systemd security patch during the same daily upgrade window. Not every host was affected: more than 90% of the fleet used Ubuntu 22.04, and some nodes had not yet downloaded the patch when their upgrade ran. ## Conflict with Datadog’s Deployment Process - Datadog normally updates nodes by replacing them automatically rather than relying on unattended upgrades. - Its standard process validates changes on experimental clusters, then progressively deploys them through staging and production. - Deployments are normally limited to selected clusters, availability zones, and regions before expanding. - The unattended systemd patch bypassed this process because it was installed directly on existing hosts. - As a result, a low-level networking change propagated across otherwise isolated regions and cloud providers at nearly the same time. Datadog’s experience demonstrates that even security-only automated updates can introduce coordinated infrastructure risk. Critical system packages should be tested and rolled out through the same staged process as other production changes, or their automated upgrades should be carefully constrained and monitored.

network-configurationk8scloud-infrastructuresystemd+4

datadogMay 24, 2023

2023-03-08 incident: A deep dive into the platform-level impact | Datadog (opens in new tab)

The March 2023 Datadog outage was triggered by a simultaneous, global failure across multiple cloud providers and regions, caused by an unexpected interaction between a systemd security patch and Ubuntu 22.04’s default networking behavior. While Datadog typically employs rigorous, staged rollouts for infrastructure changes, the automated nature of OS-level security updates bypassed these controls. The incident highlights the hidden risks in system-level defaults and the potential for "unattended upgrades" to create synchronized failures across supposedly isolated environments. ## The systemd-networkd Routing Change * In December 2020, systemd version 248 introduced a change where `systemd-networkd` flushes all IP routing rules it does not recognize upon startup. * Version 249 introduced the `ManageForeignRoutingPolicyRules` setting, which defaults to "yes," confirming this management behavior for any rules not explicitly defined in systemd configuration files. * These changes were backported to earlier versions (v247 and v248) but were notably absent from v245, the version used in Ubuntu 20.04. ## Dormant Risks in the Ubuntu 22.04 Migration * Datadog began migrating its fleet from Ubuntu 20.04 to 22.04 in late 2022, eventually reaching 90% coverage across its infrastructure. * Ubuntu 22.04 utilizes systemd v249, meaning the majority of the fleet was susceptible to the routing rule flushing behavior. * The risk remained dormant during the initial rollout because `systemd-networkd` typically only starts during the initial boot sequence when no complex routing rules have been established yet. ## The Trigger: Unattended Upgrades and the CVE Patch * On March 7, 2023, a security patch for a systemd CVE was released to the Ubuntu security repositories. * Datadog’s fleet used the Ubuntu default configuration for `unattended-upgrades`, which automatically installs security-labeled patches once a day, typically between 06:00 and 07:00 UTC. * The installation of the patch forced a restart of the `systemd-networkd` service on active, running nodes. * Upon restarting, the service identified existing IP routing rules (crucial for container networking) as "foreign" and deleted them, effectively severing network connectivity for the nodes. ## Failure of Regional Isolation * Because the security patch was released globally and the automated upgrade window was synchronized across regions, the failure occurred nearly simultaneously worldwide. * This automation bypassed Datadog’s standard practice of "baking" changes in staging and experimental clusters for weeks before proceeding to production. * Nodes on the older Ubuntu 20.04 (systemd v245) were unaffected by the patch, as that version of systemd does not flush IP rules upon a service restart. To mitigate similar risks, infrastructure teams should consider explicitly disabling the management of foreign routing rules in systemd-networkd configuration when using third-party networking plugins. Furthermore, while automated security patching is a best practice, organizations must balance the speed of patching with the need for controlled, staged rollouts to prevent global configuration drift or synchronized failures.

network-configurationk8sdockersystemd+4