Kubernetes

144 posts

gitlab3 min readCurated summary

Shai-Hulud copycat campaign targets Python developers through PyPI typosquatting

GitLab researchers uncovered a coordinated PyPI supply-chain campaign distributing a copy of the Shai-Hulud worm. Five packages—four typosquats and one compromised legitimate project—execute malware during Python startup, steal credentials from CI/CD and cloud environments, and propagate through developers’ repositories and package registries. The campaign demonstrates that Python packages can be weaponized without imports or explicit function calls. ## Malicious PyPI Packages - All packages were published by the `elitexp` account: - `rlask` and `tlask`, typosquats of Flask - `rsquests`, a typosquat of Requests - `nhmpy`, a typosquat of NumPy - `mflux-streamlit`, a legitimate project later weaponized in versions `0.0.3` and `0.0.4` - The attacker first uploaded clean probe versions matching current upstream version numbers, then replaced them with payload-bearing releases. - The activity followed the public release of Shai-Hulud’s source code, suggesting an independent copycat operation targeting Python users. ## Python Startup-Based Infection - The malware uses Python `.pth` files, which Python processes automatically at startup. - The dropper: - Checks for a `.bun_ran` marker in the temporary directory. - Downloads the Bun JavaScript runtime from GitHub. - Executes a roughly 5 MB obfuscated JavaScript payload. - Early `rlask` versions also included `sitecustomize.py`, which searched `sys.path` for and executed a hidden `_index.js` file. - This approach requires no explicit package import or function invocation. ## Payload Obfuscation - The JavaScript is protected by multiple layers: - Package-specific ROT-N encoding - AES-128-GCM encryption - Variable-name mangling using `_0x` identifiers - Researchers identified: - A small encrypted Bun downloader - A 772 KB Shai-Hulud credential stealer - Approximately 2,538 hardcoded strings ## Credential Theft The worm targets credentials and secrets from: - GitHub Actions tokens, repository secrets, OIDC tokens, artifacts, and runner memory - AWS IAM credentials, instance metadata, Secrets Manager, SSM, and STS tokens - Azure managed identities, Key Vault, and Microsoft Graph tokens - GCP service-account keys and application credentials - HashiCorp Vault tokens and Kubernetes authentication - npm, JFrog, PyPI, and RubyGems publishing credentials - SSH private keys and Kubernetes service-account tokens - Sigstore credentials and Fulcio signing certificates - MongoDB, MySQL, PostgreSQL, and Redis connection strings ## Self-Propagation Using stolen credentials, the worm can: - Add `.github/setup.js` and workflow files to repositories so it runs in other CI pipelines. - Insert `.github/copilot-instructions.md` to influence AI coding assistants. - Publish poisoned packages to PyPI, npm, and RubyGems. - Attempt privilege escalation on self-hosted runners through `sudoers` modifications. - Detect StepSecurity’s harden-runner and alter its behavior. ## Attacker Infrastructure and Weaponized Project - The PyPI account was created in 2024 and was associated with the legitimate `mflux-streamlit` project. - Package uploads used `Bun/1.3.14`, matching the runtime downloaded by the malware. - Unlike a pure typosquatting campaign, the compromise of a real project could affect existing users through normal dependency updates. Developers should audit environments for the affected packages, review CI/CD and cloud credentials, rotate exposed secrets, and enforce dependency pinning and package provenance checks. CI runners and publishing tokens should be treated as potentially compromised if any affected version was installed.

Read original(opens in new tab)
aws3 min readCurated summary

AWS Weekly Roundup: BYOM for Amazon RDS for SQL Server, AWS IoT Device SDK for Swift, and more (June 8, 2026) | Amazon Web Services

The AWS roundup highlights the general availability of the AWS IoT Device SDK for Swift, bringing MQTT 5, Device Shadow, Jobs, and fleet provisioning to Apple platforms and Linux. It also covers major AWS releases involving SQL Server licensing, Cognito resilience, OpenAI models on Bedrock, Kubernetes, AI agents, cost reporting, and location services. Together, the announcements show AWS expanding support for Swift edge computing, enterprise AI, multi-Region architectures, and specialized infrastructure. ## AWS IoT Device SDK for Swift - Now generally available for macOS, iOS, tvOS, and Linux. - Provides production-ready support for: - MQTT 5 connectivity - Device Shadow - IoT Jobs - Fleet provisioning - Reflects Swift’s growing use across server-side development, IoT, and edge computing. - Projects such as WendyOS are also bringing Swift to NVIDIA Jetson and Raspberry Pi hardware. ## Major AWS Headlines ### Amazon RDS for SQL Server BYOM - Amazon RDS for SQL Server now supports Bring Your Own Media. - Customers migrating from on-premises SQL Server can reuse existing licenses, including Software Assurance. - Support is provided through Microsoft’s License Mobility program. - AWS License Manager tracks license usage and compliance. ### Multi-Region Amazon Cognito - Cognito can replicate user and machine identity data to a standby Region in near real time. - Replicated data includes credentials, user pool settings, and federation configurations. - Users can continue using applications without re-authentication after a primary-Region disruption. - Available as an add-on for Essentials and Plus user pools across 16 Regions. ### OpenAI Models on Amazon Bedrock - GPT-5.5, GPT-5.4, and Codex are generally available for production use. - GPT-5.5 targets agentic coding, data analysis, and complex autonomous tasks. - Codex supports the Codex App, CLI, and integrations with VS Code, JetBrains, and Xcode. - AWS governance and security controls remain available, pricing follows OpenAI rates, and usage counts toward existing AWS commitments. ## Recent AWS Launches - **Amazon Bedrock observability:** CloudWatch metrics now cover inference counts, token usage, and client errors for OpenAI- and Anthropic-compatible APIs. - **Redesigned Bedrock console:** Adds model catalogs, side-by-side comparisons, project organization, and pre-filled code examples. - **AgentCore Identity secrets:** Credential providers can reference existing AWS Secrets Manager secret ARNs, supporting custom KMS keys, tagging, and rotation. - **Step Functions agentic reasoning:** Workflows can invoke AgentCore-powered agents sequentially or in parallel, include human approval, and trace decisions. - **Kubernetes 1.36 on EKS:** Adds User Namespaces GA, Mutating Admission Policies, in-place pod resource scaling, and resource health reporting. - **ECS Managed Instances accelerators:** Supports Trainium1, Trainium2, and Inferentia2 instances with automatic accelerator allocation. - **Amazon Quick VPC connectivity:** Enables private connections to MCP servers without exposing internal tools to the public internet. - **Cost and Usage Report 2.0:** Adds Athena and Redshift integrations with generated infrastructure templates, table definitions, and loading guidance. - **Amazon Location Service:** Routes API now supports transit and intermodal journeys across 13 Regions. AWS also directs readers to its What’s New page, Builder Center, and upcoming events for further announcements and community resources.

Read original(opens in new tab)
datadog3 min readCurated summary

When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes

Datadog’s gameday testing exposed a PostgreSQL failure mode in which network latency caused replication lag to grow until no standby could be safely promoted. Although the clusters remained writable, they could not fail over without risking data loss, forcing operators to wait for connectivity and replicas to recover. Datadog’s solution was to redesign failover candidates around synchronous replication coordinated by Patroni, balancing stronger durability with acceptable write latency. ## The Zonal Failure That Exposed the Weakness - A simulated availability-zone failure introduced network latency in a staging environment. - Several Kubernetes-based PostgreSQL clusters had primary nodes in the affected zone. - Communication between primaries and replicas degraded, causing: - Rapidly increasing replication lag - Stalled writes - Applications serving stale data - No replica being current enough for safe promotion - The clusters prioritized continued writes over durability, leaving them writable but unable to fail over safely. ## Baseline PostgreSQL Architecture - Each cluster uses a single-writer design: - One active leader handles writes. - Two standby nodes are reserved for failover and do not serve application traffic. - A separate read-replica pool handles read-only traffic and scales independently. - Read replicas are intentionally excluded from failover candidates. - Patroni manages replication, leader elections, and failover. - ZooKeeper acts as Patroni’s distributed configuration store, tracking: - The current leader lock - Cluster configuration - Member replication state and latest LSN - ZooKeeper’s ephemeral leader key ensures that only one node can become primary. - During partitions, Patroni favors safety by pausing or demoting nodes that cannot verify cluster state. ## Why Failover Was Not Safe - Patroni checks replication lag before promoting a standby using `maximum_lag_on_failover`. - During the gameday, all eligible standbys exceeded that threshold. - Patroni correctly rejected promotion because each candidate could have been missing committed transactions. - The cluster therefore had no safe writable primary, even though the original leader was impaired. - The failure was a consequence of asynchronous replication and network latency, not a failure in Patroni’s safety mechanisms. ## Asynchronous Versus Synchronous Replication - **Asynchronous replication**, used originally: - Lets the leader commit and respond without waiting for replicas. - Provides low write latency and high throughput. - Can lose transactions committed on the leader but not yet copied to a standby. - **Synchronous replication**: - Requires the leader to receive acknowledgment from at least one replica before confirming a transaction. - Reduces the chance that a failover candidate is significantly behind. - Provides stronger durability, but may increase write latency when replicas experience network or availability problems. ## The Redesigned Approach - Datadog reworked its PostgreSQL deployment so failover candidates use synchronous replication. - Patroni coordinates these replicas and continues to enforce safe leader election. - The design aims to make failover both automatic and safe while limiting performance impact. - Benchmarking and failure testing were used to evaluate the trade-off between durability and latency. Datadog’s experience demonstrates that asynchronous replication can leave a system operational but unable to fail over during network disruption. For clusters where data durability and automatic recovery are critical, synchronous replication for designated failover candidates offers a safer architecture, provided its latency and availability costs are measured carefully.

Read original(opens in new tab)
line4 min readCurated summary

ODW #7: Reduce Token Consumption by 40% in Three Ways! Context Engineering with ADK

The post explains how LY Corporation’s Orchestration Development Workshop uses context engineering to reduce AI-agent costs and improve accuracy. As internal adoption of tools such as Claude Code, Cline, and ADK grows, excessive token usage, missed instructions, and declining performance in long conversations have become common. The recommended solution is to deliberately select and manage the context sent to an LLM, demonstrated through an ADK-based Jira weekly-report agent. ## Problems Caused by Expanding AI Use - Increased AI adoption has led to unexpectedly high token consumption. - Users often receive incomplete or incorrect results despite providing detailed prompts. - Long-running conversations can cause the model to produce irrelevant answers. - Major causes include: - Trial-and-error prompting - More complex and long-running agents - Expansion from single-agent to multi-agent systems - Tool integrations such as MCP, whose definitions also consume context - Limited awareness of context optimization techniques ## Context Rot and Context Engineering - **Context rot** occurs when long-running agents accumulate conversation history, intermediate results, and irrelevant information. - As the context grows: - The context window becomes pressured. - Relevant information becomes harder to identify. - Noise overwhelms important signals, reducing accuracy. - Context engineering is the deliberate design and management of all information provided during inference, including: - **Static context:** System prompts and tool definitions - **Dynamic context:** User messages, conversation history, and retrieved external data - **Long-term context:** Persistent session state and accumulated information - The core principles are: - Treat tokens as a limited resource and retain the smallest set of high-signal information. - Provide neither too little information, which forces guesswork, nor too much, which wastes tokens and reduces clarity. ## Why Use ADK Google’s open-source Agent Development Kit (ADK) is presented as a practical platform for applying context engineering. - Agents can be designed and shared using team knowledge rather than relying on individual CLI expertise. - ADK includes UI, API-server, evaluation, and multi-agent capabilities. - Its multi-agent architecture naturally supports separating and controlling context. ## ADK Context-Engineering Components The workshop introduces nine key components, including: - **Structured input and output:** JSON or schema-based formats reduce unnecessary text and make agent processing more reliable. - **AgentTool:** Embeds one agent inside another as a tool. The calling agent receives only the final result, preventing internal tools and intermediate context from accumulating. - **MCP Toolset filtering:** The `tool_filter` parameter exposes only required MCP tools, reducing tool-definition tokens and improving model decisions. - The remaining components can be combined with these techniques to control context throughout an agent workflow. ## Jira Weekly Report Example The workshop builds `jira_weekly_report`, an agent that analyzes team Jira tickets and generates a weekly Markdown report. ### Version 1: Single Agent Without Context Engineering - A single agent retrieves the ticket list, fetches each ticket, analyzes it, and builds the report. - All Jira tools are exposed through one MCP toolset. - As the number of tickets increases, detailed ticket contents accumulate in the agent’s context. - This leads to context rot, higher token usage, and declining reliability. ### Version 2: Context-Aware Multi-Agent Design - The workflow is split into: - A root agent that searches Jira tickets and aggregates the final report. - A sub-agent dedicated to analyzing one ticket at a time. - `input_schema` requires a structured `issue_key`. - `output_schema` requires a structured report containing ticket content and progress, including comments. - The sub-agent receives only the `jira_get_issue` MCP tool. - The root agent receives only the `jira_search` tool. - `AgentTool` hides the sub-agent’s internal context and returns only its final report. - The sub-agent is instructed to include facts only and avoid speculation. This design limits each agent’s responsibilities, removes unnecessary tool definitions, and prevents individual ticket details from polluting the root agent’s context. ## Practical Recommendation For production AI agents, treat context as a constrained resource. Use structured schemas, narrowly filtered tools, and specialized sub-agents to pass only the information needed for each step.

Read original(opens in new tab)
spotify3 min readCurated summary

Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify | Spotify Engineering

Spotify argues that AI has shifted software development’s main constraint from writing code to coordinating people, systems, and decisions. Years of investment in standardized platforms, automation, and developer experience enabled Spotify to adopt AI coding tools at extraordinary scale. The company’s experience suggests that consistent infrastructure and strong feedback loops are essential for making both human developers and coding agents effective. ## Rapid AI Adoption - More than 99% of Spotify engineers use AI coding tools weekly. - 94% report improved productivity. - Pull request frequency has increased by 76%, with most PRs created by developers working alongside AI agents. - Adoption accelerated sharply after the release of Claude Opus 4.5. ## Fleet Management Before AI Agents - Spotify’s codebase was growing seven times faster than its engineering workforce. - Developers increasingly spent time on dependency upgrades, API migrations, and vulnerability fixes. - Fleet Management automated changes across hundreds or thousands of components. - Its orchestration system, Fleetshift, has merged more than 2.5 million maintenance PRs, most without human intervention. - This approach reduced migrations from work taking weeks or months across many teams to centrally managed operations. ## Honk: A Background Coding Agent - Deterministic scripts struggled with complex refactoring and the edge cases found across large codebases. - Spotify created Honk, a background coding agent powered by Claude through the Agent SDK. - Honk runs in Kubernetes pods, allowing many coding sessions to execute concurrently. - It can use trusted tools and run builds in CI across multiple operating systems. - Fleetshift identifies targets, schedules work, and tracks PRs, while Honk performs the code changes. - A recent Java migration across Spotify’s backend services took three days. - Engineers can invoke Honk through Slack, where it uses conversation context to create and return PRs. - Honk v2 adds shared sessions, team projects, and agent orchestration through Chirp. ## Standardization Improves Agent Performance - Spotify’s principle of limiting the number of technologies it supports reduces decisions and improves collaboration. - Consistent service architectures and design patterns also give AI agents better reference material. - Agents perform worse in fragmented codebases with inconsistent conventions. - Backstage provides a unified internal developer portal and catalog for software components. - Spotify exposes Backstage capabilities to agents through MCP integrations and command-line tools. - Agents can discover component ownership, read documentation, and contact responsible teams. ## Guardrails Through Backstage - Backstage’s Soundcheck and “golden state” define recommended technologies and practices. - Teams can assess their components against these standards. - Static analysis and linting provide immediate feedback when developers or agents use unsuitable patterns. - This creates a feedback loop that helps agents correct their work and drives consistency across the organization. Spotify’s experience indicates that scaling AI development requires more than giving engineers access to models. Organizations should invest in standardized platforms, searchable component metadata, automated fleet-wide workflows, and strong validation systems so agents can operate reliably at team scale.

Read original(opens in new tab)
gitlab3 min readCurated summary

Consolidate your GitLab stack with Gitaly on Kubernetes

Gitaly on Kubernetes is now generally available with GitLab 18.11, allowing teams to run their entire GitLab stack in Kubernetes instead of maintaining Gitaly on separate virtual machines. GitLab addressed Kubernetes-specific challenges involving cgroup isolation, pod restarts, and request reliability. The result is a more unified deployment model, though full high availability still depends on Gitaly Cluster support for Kubernetes. ### Challenges of Running Gitaly on Kubernetes - Git operations can consume unpredictable amounts of memory. - Gitaly isolates individual Git processes in dedicated cgroups so an out-of-memory failure does not bring down the main Gitaly process. - Kubernetes deployments required special handling because containerd traditionally restricted cgroupfs writes to privileged containers. - GitLab solved this by using an init container to mount `/sys/fs/cgroup` and make it writable. ### Handling Pod Restarts - Virtual-machine deployments can upgrade Gitaly in place and reload gracefully while preserving the socket. - Kubernetes StatefulSet replacements cause pods to stop and restart abruptly during upgrades, node drains, or configuration changes. - This could cause downtime, particularly for Gitaly Sharded deployments without built-in high availability. - GitLab made Gitaly client retries configurable, allowing clients such as Rails to retry requests until Gitaly becomes available again. - Users may experience slightly higher latency during restarts, but requests generally succeed without visible downtime. ### Benchmark Results and High Availability - GitLab tested common Git operations against VM-based and Kubernetes-based Gitaly installations during upgrades. - Success rates were nearly identical in both environments despite Kubernetes abruptly terminating pods and closing sockets. - Achieving complete success across every operation still requires Gitaly Cluster with Praefect. - Praefect does not yet support Kubernetes, but Kubernetes support is being developed. ### Benefits for GitLab Deployments - Teams with hybrid infrastructure can move Gitaly from virtual machines into their existing Kubernetes cluster. - This removes the need to maintain and monitor a separate VM fleet. - Organizations adopting GitLab on Kubernetes can use a fully Kubernetes-native deployment through the official Helm chart. - Gitaly can run as part of a complete GitLab installation or as an external component. ### Installation - The recommended deployment method is the GitLab Helm chart. - Users should review the Gitaly on Kubernetes documentation before installation. - The documentation covers configuration guidance, common pitfalls, full installations, and external Gitaly deployments. Gitaly on Kubernetes is a practical option for consolidating GitLab infrastructure and simplifying operations. Teams should use the Helm chart and configure client retries carefully, while recognizing that Kubernetes-based high availability through Praefect is still forthcoming.

Read original(opens in new tab)
gitlab3 min readCurated summary

Automate deployment processes with GitLab Duo Agent Platform

GitLab Duo Agent Platform can automate the complex, repetitive work of onboarding a microservice into an established GitOps workflow. By analyzing an application’s repositories and configuration, a custom agent can generate manifests, update pipelines, configure image automation, and follow organization-specific conventions. The approach combines AI-driven speed with GitLab-managed versioning, governance, and enterprise security. ## TanukiBank’s GitOps Use Case - The fictional TanukiBank application needs a new `intra-account-transfers` microservice for its Quick Transfer feature. - Its deployment architecture includes: - Individual service projects with container registries and build pipelines. - **Tanuki Bank - Delivery**, which stores deployment manifests and delivery pipelines. - **Flux Config**, which contains Flux manifests for Kubernetes. - Flux Image Automation watches service registries and updates corresponding delivery manifests. - A delivery pipeline then builds and signs the image, while Flux CD synchronizes it to the Kubernetes cluster. - Adding a service manually requires coordinated changes across all these components. ## Generating the Custom Agent’s System Prompt - GitLab Duo Agentic Chat examines the TanukiBank group, subgroups, source files, Dockerfiles, manifests, configuration, and dependencies. - It generates a detailed system prompt describing: - The existing GitOps workflow. - Required operating rules. - Reporting instructions. - Recommended tools. - The prompt is specific to the workflow at the time it is generated. - If the application’s GitOps process changes, the prompt should be regenerated. ## Creating and Configuring the Agent - A new `application-agents` project manages custom agents, their administrators, and where they can run. - A managed agent named **TanukiBank Microservice Onboarder** is created with: - A description. - The generated system prompt. - Tools recommended by GitLab Duo. - The agent is enabled in both **Tanuki Bank - Delivery** and **Flux Config**. - Its presence in each project’s Agentic Chat agent selector confirms that it is available. ## Creating the Microservice - A new `services/intra-account-transfers` project is created. - GitLab Duo’s **Developer** foundational flow implements the service from an issue specification. - The flow: - Reads the requirements. - Writes the implementation. - Creates a branch and merge request. - Links the merge request to the issue. - After local verification with `curl`, the merge request is merged and the project pipeline publishes container images. - At this stage, the service exists, but the GitOps system has not been updated: - `manifests/dev` has no service manifests. - The delivery pipeline does not reference the service. - `Flux Config` lacks an `image-update-automation.yaml` entry. ## Using the Custom Onboarding Agent - The custom agent is enabled in the new service project. - From **Tanuki Bank - Delivery**, the user selects **TanukiBank Microservice Onboarder** in Agentic Chat and provides the service name and hostname. - The agent begins onboarding by: - Finding and reading the service’s Dockerfile. - Determining the application port. - Generating the required Kubernetes manifests. - Updating the relevant delivery pipelines. - This automates the coordinated repository changes normally required for a new microservice. ## Practical Takeaway A custom GitLab Duo agent is most valuable when it is grounded in an organization’s real repositories and deployment conventions. Generate its prompt from the current system, keep the agent centrally governed, and regenerate the prompt whenever the GitOps workflow changes.

Read original(opens in new tab)
slack3 min readCurated summary

From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

Slack had more than 700 SSH-based operators running critical EMR workloads, creating security risks, operational failures, and barriers to infrastructure modernization. The company replaced these connections with REST-based job submission across eight data regions without downtime. YARN Distributed Shell was the key enabler for migrating arbitrary command-line jobs that lacked dedicated REST APIs. ## How Slack’s SSH Architecture Developed - Airflow originally connected directly to EMR master nodes using `SSHOperator`. - Over time, teams created more than 700 SSH-based jobs for: - Spark and MapReduce workloads - AWS CLI commands - Custom Python scripts - Data-transfer operations such as `hadoop distcp` - The approach was simple but tightly coupled orchestration workers to production clusters. ## Security and Operational Costs of SSH - Direct SSH access expanded the attack surface. - SSH keys had to be distributed and rotated across orchestration workers. - Auditing required correlating activity across multiple systems. - Permissions became complicated, often involving custom security groups and configurations. - Jobs ran on EMR master nodes, causing resource contention. - Restarted Kubernetes pods could break SSH connections. - Long-running processes could become orphaned “zombie” jobs. - Connection failures made job success or failure difficult to determine. - SSH dependencies blocked Spark-on-Kubernetes, EMR on EKS, AWS child-account migration, and better observability. - Slack’s search-indexing pipeline was especially sensitive because it processed terabytes of data daily and supported search for millions of users. ## REST-Based Job Submission - SSH creates a stateful connection whose failure can leave job status ambiguous. - REST APIs provide a durable, server-managed lifecycle: - `POST` submits a job and returns an ID. - `GET` retrieves its status. - `DELETE` cancels it cleanly. - Clients can crash or restart without terminating the underlying job. - Existing systems such as YARN, Trino, and Snowflake use this model. - YARN provides REST submission for Hadoop, Spark, Hive, and MapReduce workloads, but not arbitrary shell commands. ## YARN Distributed Shell - Spark and Hive already had REST-compatible options through Livy and HiveServer2. - The difficult cases were MapReduce and more than 300 CLI-based jobs. - Slack considered custom wrapper services, Ansible or Salt, and creating a new YARN job type. - These alternatives added complexity, security work, or long-term maintenance. - YARN Distributed Shell—implemented through `ApplicationMaster`—could execute arbitrary scripts inside YARN containers. - It used existing YARN APIs and authentication mechanisms, avoiding a custom security layer. ## The Distributed Shell Workflow - Upload a command script to S3, such as an `aws s3 sync` operation. - Submit a YARN application specifying: - The Distributed Shell application master - The S3 script location - Script metadata such as length and timestamp - YARN then: - Allocates a resource-managed container - Downloads and executes the script - Enforces memory and vCore limits - Provides isolation, retries, cancellation, and centralized logging By using REST submission and YARN Distributed Shell, Slack could remove SSH from its EMR data pipelines while preserving support for both standard data-processing jobs and arbitrary command-line workloads.

Read original(opens in new tab)
discord3 min readCurated summary

You’ve Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage

Discord’s March 25, 2026 voice outage began when a Kubernetes configuration change abruptly terminated 17% of session processes. The resulting reconnection storm propagated through Discord’s realtime systems and overloaded voice-routing infrastructure, preventing many users from starting or joining calls. The incident exposed how failures in one distributed subsystem can create cascading load several services away. ## The Infrastructure Background - Discord is migrating stateful Elixir services to Kubernetes. - Each host runs thousands of in-memory processes for guilds, presence, messaging, and calls. - Deployments normally wait for a server’s entity count to reach zero before shutting it down, allowing processes to hand off their state safely. - The sessions service maintains one process for every connected device and carries websocket traffic, messages, presence updates, and other realtime events. - To reduce weekend CPU utilization, Discord planned to increase pod CPU and memory while proportionally reducing the number of pods. ## The Session Loss - The resource change was deployed to the first availability zone at 12:13 PDT. - Kubernetes terminated half of that zone’s pods because of the reduced replica count. - A safety check delayed process handoffs until other events completed, but the Kubernetes termination grace period expired first. - Because the service operated across three balanced zones, approximately 17% of Discord’s sessions stopped without a graceful handoff. - The outage lasted from 12:13 to 15:30 PDT, with users commonly seeing “Awaiting Endpoint.” ## How Elixir Monitoring Amplified the Failure - Discord relies heavily on Elixir `GenServer` processes, which process one mailbox message at a time. - Process monitors notify dependent processes whenever a monitored process exits. - The sudden loss of sessions therefore generated a large number of `{:DOWN, …}` notifications throughout the realtime infrastructure. - Guild and other processes stopped attempting to deliver updates to disconnected users, while the gateway began driving those users to reconnect. ## Reconnecting Users - The gateway handles websocket ingress and egress, creating sessions and maintaining client connections. - Session disconnections are normally expected and recoverable, whether caused by hardware, network problems, software bugs, or temporary connectivity loss. - When a session disappears, the gateway immediately instructs the client to reconnect. - It optimistically tries to resume the session through a gateway instance in the same zone, but the mass failure created a much larger reconnection surge than the system was designed to absorb. The incident demonstrates that reducing pod count can be dangerous in stateful distributed systems: an apparently routine capacity adjustment can cause abrupt process loss, trigger widespread retries, and overload unrelated downstream services. Changes to stateful workloads should be evaluated not only for steady-state resource usage but also for graceful shutdown behavior and synchronized failure scenarios.

Read original(opens in new tab)
gitlab3 min readCurated summary

How to build CI/CD observability at scale

CI/CD observability is essential for improving pipeline performance at enterprise scale, particularly in self-managed GitLab environments. The post presents a containerized solution built with `gitlab-ci-pipelines-exporter`, Prometheus, Grafana, and Node Exporter to turn pipeline and infrastructure data into actionable insights. Its conclusion is that centralized dashboards help teams identify bottlenecks, plan runner capacity, and measure delivery performance. ## Defining CI/CD Performance - Teams should first determine: - Which metrics matter, such as pipeline duration, job success rates, queue times, and runner utilization. - Who needs access, including developers, DevOps engineers, platform teams, and leadership. - Which decisions the data will support, such as infrastructure investment, bottleneck remediation, and capacity planning. ## Observability Architecture - The solution uses two exporters: - **Pipeline Exporter:** Collects pipeline duration, job status, and deployment metrics through the GitLab API. - **Node Exporter:** Collects host CPU, memory, and disk metrics for infrastructure correlation. - Prometheus gathers and stores the metrics. - Grafana provides real-time and historical dashboards. - Dashboards are provisioned automatically through Grafana’s file-based provisioning and can be filtered by project, branch, or time range. ## Grafana Dashboards - **Pipeline Overview:** Displays pipeline volume, success and failure rates, cancelled runs, and average duration trends. - **Job Performance:** Shows job-duration histograms, the ten slowest jobs, and failure heatmaps by project and stage. - **Runner & Infrastructure:** Correlates runner queue times with CPU, memory, and disk usage to support capacity planning. - **Deployment Frequency:** Tracks deployment counts and durations by environment, supporting DORA-style delivery analysis and detection of environment drift. ## Kubernetes Deployment - The recommended enterprise deployment runs each component as a separate workload in a dedicated `gitlab-observability` namespace. - A Kubernetes secret stores the GitLab personal access token, which requires the `read_api` scope. - The Pipeline Exporter runs as a Deployment with a service on port `8080`. - Node Exporter runs as a DaemonSet so each node can expose host metrics on port `9100`. - Prometheus and Grafana are deployed alongside the exporters and configured to scrape and visualize their metrics. - Kubernetes deployment supports existing cluster infrastructure, secrets managers, network policies, and scalable operations. ## Prerequisites - GitLab Self-Managed 18.1 or later. - Kubernetes for enterprise deployments, or Docker/Podman for smaller environments and proof-of-concept testing. - A GitLab personal access token with `read_api` permissions. - Secure secret-management practices, preferably using external secret operators in production. The practical recommendation is to begin with clearly defined performance questions, then deploy the exporter–Prometheus–Grafana stack in a controlled namespace. Combining pipeline data with host metrics provides the context needed to distinguish inefficient jobs from infrastructure capacity problems.

Read original(opens in new tab)
aws3 min readCurated summary

AWS Weekly Roundup: Anthropic & Meta partnership, AWS Lambda S3 Files, Amazon Bedrock AgentCore CLI, and more (April 27, 2026) | Amazon Web Services

This week’s AWS news centers on deeper AI infrastructure partnerships and tools for building production-ready agents. AWS and Anthropic are expanding Claude’s integration with AWS hardware and Amazon Bedrock, while Meta is adopting Graviton for large-scale agentic AI workloads. New services for Lambda, EKS, Aurora, and Bedrock also emphasize simpler data access, hybrid networking, serverless scaling, and faster agent development. ## Anthropic and Meta Expand AWS AI Partnerships - Anthropic is training advanced foundation models on AWS Trainium and Graviton processors. - Anthropic and AWS’s Annapurna Labs are co-engineering at the silicon level to improve efficiency across the stack. - Claude Cowork is now available through Amazon Bedrock, allowing enterprise teams to collaborate with Claude while keeping data within AWS. - A unified Claude Platform on AWS is planned, offering a single experience for building, deploying, and scaling Claude applications. - Meta signed an agreement to deploy tens of millions of AWS Graviton cores for CPU-intensive agentic AI tasks, including reasoning, code generation, search, and orchestration. ## New Lambda and Kubernetes Infrastructure - AWS Lambda can mount Amazon S3 buckets as file systems using S3 Files. - Functions can perform standard file operations without downloading data first. - Built on Amazon EFS, S3 Files combines file-system access with S3’s scalability, durability, and cost model. - Multiple Lambda functions can share the same workspace, supporting AI agents that need persistent memory or shared state. - The Amazon EKS Hybrid Nodes gateway simplifies networking between cloud-based EKS resources and on-premises Kubernetes Pods. - It enables pod-to-pod traffic, control-plane webhook communication, and access to AWS services without making on-premises pod networks routable. - The gateway is available at no additional charge. ## Aurora Serverless and Bedrock Agent Development - Aurora Serverless now offers up to 30% better performance on platform version 4. - Its scaling algorithm better handles competing workloads, including busy APIs and bursty agentic AI applications. - The service continues to scale to zero during idle periods, with no additional charge for the improvements. - Amazon Bedrock AgentCore adds a managed harness in preview, allowing developers to define a model, system prompt, and tools without writing orchestration code. - Harnesses can later be exported as Strands-based code for greater control. - The AgentCore CLI supports governed, auditable deployments through AWS CDK, with Terraform support planned. - The CLI is available in 14 AWS Regions at no additional charge, and AgentCore skills support coding assistants. ## Cost Management, Operations, and Machine Learning - Granular cost attribution for Amazon Bedrock enables teams to track usage by project or organization and support detailed chargeback. - AWS DevOps Agent can work with the Salesforce MCP Server to investigate incidents, diagnose causes, and notify customers through Salesforce Service Cloud. - AWS microcredentials are now free through AWS Skill Builder in supported countries. - These hands-on assessments use simulated business scenarios and live AWS environments rather than traditional multiple-choice testing. - Amazon SageMaker AI can recommend optimized generative AI inference configurations, including instance types, containers, and inference parameters, helping reduce latency and deployment costs. ## Upcoming AWS Events - “What’s Next with AWS” is scheduled as a virtual event on April 28. - AWS Summits continue in May across cities including Singapore, Tel Aviv, Warsaw, Stockholm, Sydney, Hamburg, Seoul, Amsterdam, Bangkok, and Milan. AWS’s latest releases point toward a more integrated AI platform: specialized hardware for model execution, managed agent tooling, shared state through serverless storage, and stronger cost and operational controls. Builders should evaluate S3 Files and AgentCore for AI workflows, while teams running production inference can benefit from SageMaker recommendations and Bedrock’s improved cost attribution.

Read original(opens in new tab)
aws3 min readCurated summary

AWS Weekly Roundup: Claude Opus 4.7 in Amazon Bedrock, AWS Interconnect GA, and more (April 20, 2026) | Amazon Web Services

The roundup highlights major AWS advances in AI, networking, developer tooling, and security. Claude Opus 4.7 is now available through Amazon Bedrock with stronger agentic coding and research capabilities, while AWS Interconnect simplifies private connectivity across clouds and remote locations. Additional launches improve container supply-chain security, application modernization, database access, cost attribution, and quantum-resistant encryption. ## Anthropic Claude Opus 4.7 in Amazon Bedrock - Anthropic’s latest Opus model improves: - Agentic coding and long-running tasks - Complex code reasoning - Document creation, financial analysis, and multi-step research - It scores: - 64.3% on SWE-bench Pro - 87.6% on SWE-bench Verified - Bedrock features include: - Dynamic capacity allocation - Adaptive thinking and request-specific token budgets - A 1-million-token context window - High-resolution image support for charts, documents, and screen interfaces - The model launched in US East, Tokyo, Ireland, and Stockholm, supporting up to 10,000 requests per minute per account and Region. ## AWS Interconnect Reaches General Availability - **AWS Interconnect – Multicloud** provides Layer 3 private connectivity between AWS VPCs and other clouds. - Google Cloud is supported initially; Azure and OCI are planned. - Traffic uses private networks and the AWS global backbone rather than the public internet. - Includes MACsec encryption, multi-facility resilience, and CloudWatch monitoring. - The underlying specification is open source under Apache 2.0. - **AWS Interconnect – Last Mile** connects branches, data centers, and remote sites to AWS through network providers. - Automatically provisions four redundant connections across two physical locations. - Configures BGP, MACsec, and Jumbo Frames. - Supports adjustable bandwidth from 1 to 100 Gbps. - Launches in US East with Lumen. ## Developer, Database, and Modernization Updates - Amazon ECR pull-through cache now discovers and synchronizes OCI referrers such as signatures, SBOMs, and attestations. - AWS Transform is available directly in Kiro and VS Code for migrations such as language-version upgrades and AWS SDK updates. - Aurora DSQL’s PHP connector supports IAM authentication, SSL, connection pooling, and optional optimistic-concurrency retries. - AWS Transform Custom can modernize VB6 applications into C# ASP.NET Core applications, including COM, ADO, and UI migration challenges. ## Security, Access Control, and Cost Management - Amazon Q for Google Drive now enforces document-level permissions using indexed ACLs and real-time access checks. - AWS Secrets Manager supports hybrid post-quantum TLS using ML-KEM through updated agents, Lambda extensions, and CSI drivers. - Amazon Bedrock can attribute inference costs to individual IAM principals, with reporting through CUR 2.0 and aggregation by teams, projects, or cost centers. ## Compute, Kubernetes, and Storage - EC2 C8in and C8ib instances use sixth-generation Intel Xeon processors and AWS Nitro cards. - C8in offers up to 600 Gbps networking. - C8ib provides up to 300 Gbps EBS bandwidth. - Both scale to 384 vCPUs. - EKS Auto Mode automates networking components such as VPC CNI, load balancers, and DNS while retaining enterprise security controls. - EBS Volume Clones provide immediately usable point-in-time copies for development, disaster recovery testing, and CI/CD workflows. ## Additional AWS Guidance - CloudFront Functions and CloudFront KeyValueStore can support zero-downtime API decomposition using user-aware routing and the Strangler Fig pattern. - The roundup also points readers to AWS events, weekly Power Hour training, and Community.aws meetups. The most significant developments are Bedrock’s expanded AI capabilities and Interconnect’s managed private networking. Teams should evaluate Claude Opus 4.7 for complex AI workflows, use Interconnect where multicloud or resilient connectivity is required, and consider the new security and cost-attribution features for stronger governance.

Read original(opens in new tab)
line4 min readCurated summary

ODW #2: Developing Single/Multi-Agents with ADK and Integrating with Internal Systems

AI adoption can improve productivity, but relying on individual developers to create and refine local AI agents leads to knowledge silos, duplicated effort, and uneven results. LY Corporation’s Orchestration Development Workshop addresses this by teaching engineers to build shared single- and multi-agent systems with Google’s Agent Development Kit (ADK). The workshop combines theory with hands-on integration of agents and internal tools such as Jira and Confluence through MCP. ## Organizational Need for AI - Potential applications include pull request reviews, customer support, and internal document search. - Information is difficult to find because company knowledge is distributed across systems such as Jira and Confluence. - LY Corporation aims to double work productivity within three years through AI and continuous innovation. - As tools such as Cline and Claude Code spread, usage remains concentrated among individuals. - This creates: - Productivity gaps between employees - AI knowledge silos - Repeated prompt-development work across teams - Limited awareness of multi-agent approaches - Abandonment of AI when single agents cannot handle complex tasks ## Why a Hands-On Workshop The organizers concluded that organization-wide adoption required practical understanding of three areas: - The strengths and limitations of single-agent and multi-agent systems - A team-based model for building and sharing centralized agents - Integration between AI agents and internal systems through the Model Context Protocol (MCP) Rather than teaching only concepts, the workshop required participants to build working agents with ADK. ## Single-Agent and Multi-Agent Systems - **Single agents** - Use one LLM and are relatively inexpensive and simple to develop. - Work well for straightforward tasks. - Struggle with complex problems requiring multiple specialties. - **Multi-agent systems** - Divide work among multiple specialized LLM-based agents. - Can handle more complex workflows and optimize tasks more effectively. - Require more development effort and token usage. - Must be designed carefully to avoid usage limits and excessive costs. ## Introducing Google ADK - ADK is open-source software for defining agent behavior and building multi-agent systems. - It supports Python, Java, and Go. - Python functions can be exposed as tools that agents invoke. - Teams can build and host shared agents, reducing the need for every employee to independently optimize prompts. ## Building a Single Agent Participants practiced: - Running an ADK web UI and interacting with an agent in a browser - Modifying instructions to change agent behavior - Connecting a prepared Python function as an executable tool The exercises demonstrated that prompts can flexibly control responses and that ordinary Python code can be integrated into an agent with relatively little effort. ## Connecting Agents to Internal Systems with MCP - MCP is an open standard for connecting LLMs to external systems. - It enables agents to actively search sources such as previous inquiries, documentation, Jira, and Confluence. - Participants learned that merely exposing tools is insufficient; the agent also needs clear instructions to use them effectively. - Giving one agent too many tools can enlarge its context, slow responses, and reduce accuracy. - Splitting responsibilities across multiple agents can help isolate context and mitigate these problems. ## Building a Sequential Project Tracker The main exercise created a project-tracking system that analyzes Jira projects and produces translated progress reports. - Four agents execute sequentially: 1. Analyze in-progress tasks 2. Analyze todo or unstarted tasks 3. Generate a consolidated Markdown report 4. Translate the report into the configured language - The first two agents use Jira through MCP. - The report generator synthesizes the preceding analyses. - The translator preserves the report’s formatting and structure. - ADK’s `SequentialAgent` coordinates the workflow and passes results between specialized agents. ## Practical Recommendation Organizations seeking broader AI adoption should move beyond individual experimentation. Shared agents built with ADK, connected to internal systems through MCP, can consolidate expertise, reduce duplicated prompt work, and make multi-agent workflows accessible to entire teams.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Welcome to Agents Week

Cloudflare argues that AI agents require a fundamental shift in Internet and cloud infrastructure. Unlike traditional one-to-many applications, agents create unique, ephemeral execution environments for individual users and tasks, making current container-based economics and scaling inadequate. The company positions lightweight V8 isolates, alongside containers and browser support, as the foundation for making agents practical at global scale. ## The Internet Was Built for Applications, Not Agents - Cloud infrastructure evolved during the smartphone era to serve many users through a finite number of application instances. - Microservices, containers, Kubernetes, load balancing, and replication all support this one-to-many model. - Agents differ because an LLM dynamically determines code paths, tool usage, and task duration. ## One User, One Agent, One Task - Each agent may need its own execution environment, filesystem, tools, and state. - Coding agents currently use containers with access to Git, Bash, filesystems, and arbitrary binaries. - As agents spread to assistants, analysts, customer service, and planning tasks, the number of simultaneous environments could grow dramatically. ## The Scale Challenge - If 100 million US knowledge workers used agents at 15% concurrency, infrastructure would need about 24 million simultaneous sessions. - At 25–50 users per CPU, that implies roughly 500,000 to 1 million server CPUs in the US alone. - Multiple agents per person and global adoption would increase demand by orders of magnitude. ## Isolates as Agent Infrastructure - Cloudflare’s Workers platform uses V8 isolates instead of containers. - Isolates start in milliseconds, use only a few megabytes of memory, and provide secure sandboxing. - They can be up to 100 times faster to start and up to 100 times more memory-efficient than containers. - Dynamic Workers can create execution environments on demand, run code, and discard them at a scale of millions per second. - This efficiency could make one-agent-per-user economics viable beyond expensive coding assistants. ## The “Horseless Carriage” Phase - Early agent infrastructure often adapts existing systems instead of using designs built specifically for agents. - Agents use headless browsers to navigate human-oriented websites, though structured protocols such as MCP could provide direct service access. - Many MCP servers simply wrap REST APIs, despite LLMs often being better at writing and executing code than making long sequences of tool calls. - CAPTCHAs and behavioral fingerprinting ask whether a requester is human, while agent systems need identity, authorization, and permission controls. - Full containers are frequently used for tasks that require only a few API calls and a response. ## Supporting Both Old and New Models - Infrastructure transitions rarely happen all at once; technologies such as IPv4/IPv6, HTTP/2/HTTP/3, and TLS 1.2/1.3 coexist. - Cloudflare plans to support existing agent workloads while developing more efficient primitives. - Containers remain important for coding agents that need filesystems, Git, Bash, and arbitrary binaries. - Cloudflare is also expanding container-based sandbox environments and browser-rendering capabilities for services that do not yet support agent-native protocols. Cloudflare’s broader recommendation is to build infrastructure that can serve today’s container-based agents while moving toward lightweight, ephemeral isolates designed for billions of specialized agent sessions.

Read original(opens in new tab)
datadog1 min readCurated summary

How we built a real-world evaluation platform for autonomous SRE agents at scale

The provided content does not include the blog post itself. It contains Datadog navigation links and a page title announcing that Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, but no substantive discussion of the evaluation platform or its conclusions. ## Available Information - Datadog’s page promotes its recognition as a Gartner Magic Quadrant Leader. - The navigation lists products across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI capabilities, including Bits AI Agents and Bits Investigation - The referenced URL path suggests the intended article may concern Datadog’s “Bits AI eval platform,” but the article text is not included. ## Conclusion Please provide the full blog post content for a meaningful section-by-section summary.

Read original(opens in new tab)