Prometheus | Techlist.io

awsAug 3, 20262 min readCurated summary

AWS Weekly Roundup: Price reduction of GPT models in Bedrock, CloudWatch managed collectors for Prometheus metrics, and more (August 3, 2026) | Amazon Web Services

The AWS Weekly Roundup highlights major updates in AI pricing, observability, multicloud networking, identity resilience, and data lakes. The biggest change is an up to 80% price reduction for OpenAI GPT‑5.6 Luna models in Amazon Bedrock, alongside several managed services that reduce infrastructure and operational overhead. ## Lower Bedrock Pricing for GPT‑5.6 - Effective July 30, GPT‑5.6 Luna inference prices dropped by 80%. - New pricing is: - $0.20 per million input tokens - $1.20 per million output tokens - GPT‑5.6 Terra prices decreased by 20%. - The reductions apply automatically and require no customer action. ## Managed Prometheus Monitoring in CloudWatch - Amazon CloudWatch now provides fully managed Prometheus collectors. - Customers can collect metrics from: - Amazon EKS - Amazon EC2 - Amazon ECS - Amazon MSK - Amazon OpenSearch Service - This removes the need to deploy and maintain custom Prometheus scraping agents. ## Private Multicloud Connectivity with OCI - AWS Interconnect for Oracle Cloud Infrastructure is now generally available. - It enables resilient, scalable private connections between AWS and OCI. - Traffic avoids the public internet, improving security, performance, and reliability for multicloud workloads. ## Multi-Region IAM Identity Center - IAM Identity Center can now replicate its built-in Identity Center directory across Regions. - During a primary-Region disruption, users can continue accessing AWS accounts through provisioned entitlements in additional Regions. - Previously, multi-Region support was limited to deployments using external identity providers. ## Variant Support in S3 Tables - Amazon S3 Tables now supports Apache Iceberg V3’s Variant data type. - Variant provides native, high-performance support for semi-structured data. - Suitable use cases include IoT sensor data, application logs, and schema-flexible payloads without storing everything as JSON blobs. ## Additional AWS Resources - New AWS CLI single-line commands simplify installation and upgrades across platforms and CI environments. - A deployment guide covers running Moonshot AI’s Kimi K3 on SageMaker HyperPod and Amazon EKS. - Amazon MSK Express brokers can deliver Kafka data to Apache Iceberg streaming tables on S3 Tables, with throughput of up to 10 GB/s. - AWS Summits and AWS Community Days offer upcoming opportunities for cloud and AI learning and networking. AWS users should review the new Bedrock pricing, consider managed CloudWatch collectors to reduce monitoring maintenance, and evaluate the multicloud, identity, and Iceberg updates for architectures requiring greater resilience and scalability.

Read original

Prometheus Amazon Bedrock Apache Kafka Apache Iceberg+4

discordMay 8, 20263 min readCurated summary

How Discord Automates ScyllaDB Clusters at Scale

Discord’s Persistence Infrastructure team replaced fragile, manually sequenced scripts with the Scylla Control Plane (SCP), a framework for safely automating large-scale database operations. The effort was driven by the difficulty of creating shadow clusters and managing hundreds of ScyllaDB nodes with a seven-person team. SCP emphasizes resumability, safety checks, configurable parallelism, and incremental development. ## The Scale of Discord’s Database Operations - Discord operates Elasticsearch, Postgres, and ScyllaDB infrastructure across dozens of clusters and hundreds of nodes. - ScyllaDB stores critical data, including messages, channels, servers, and much of Discord’s user data. - Routine work includes: - Rolling restarts after configuration changes - Cluster expansion as traffic grows - Operating-system upgrades without downtime - Creating test clusters for validating ScyllaDB releases - These operations require careful sequencing and continuous validation rather than simple, fire-and-forget automation. ## From Scripts to the Scylla Control Plane - Discord initially accumulated Python, Bash, and other scripts incrementally. - The scripts were useful but fragile and dependent on institutional knowledge. - As operational demands grew, Discord created the Scylla Control Plane, or SCP, to provide a more structured automation system. ## Shadow Clusters for Safer Upgrades - Shadow clusters are temporary, full replicas of production that receive the same reads and writes as live traffic. - They allow Discord to detect upgrade problems under realistic load before changing production. - Building one manually requires: - Provisioning and configuring nodes - Joining nodes to the cluster - Validating replication - Establishing dual-write pipelines - Eventually tearing the environment down - Repeating this process across every ScyllaDB cluster made automation essential, especially for testing operating-system, hardware, and ScyllaDB version changes. ## Lessons from the Previous Automation Discord identified three major weaknesses in its old scripts: - **Unsafe:** Scripts could be run against the wrong nodes or in the wrong order, often without precondition checks. - **Unrecoverable:** A failure late in a multi-step process required restarting from the beginning. - **Difficult to extend:** New operations often required copying and modifying existing scripts instead of composing reusable components. SCP was designed around four goals: - Provide an extensible task framework that hides orchestration complexity. - Support configurable parallelism, including constraints such as avoiding simultaneous work in different availability zones. - Make safety the default through preconditions, retries, and persisted state. - Deliver functionality incrementally and refine it through real-world use. ## SCP’s Task-Based Architecture - SCP is organized around **tasks, workflows, and jobs**. - A task represents one unit of work, such as draining a node, checking repair status, or running cleanup. - **Node tasks** operate on individual nodes. - **Cluster tasks** coordinate operations across an entire cluster and may run node tasks across many nodes. - SCP also uses **conditions**, which pause execution until a required state is reached. - Conditions poll ScyllaDB APIs or Prometheus metrics. - They either succeed when the criterion is met or fail after a timeout. - For example, after restarting a node, SCP can wait for compactions to settle before continuing. - This avoids unreliable fixed-duration sleeps and reduces the risk of creating cascading pressure during rolling operations. ## Practical Recommendation For large-scale database operations, automation should be built as a reusable, stateful orchestration framework rather than a collection of scripts. Explicit preconditions, observable conditions, retries, controlled parallelism, and resumable state make complex infrastructure changes safer and more repeatable.

Read original

Prometheus PostgreSQL Rust Elasticsearch+3

gitlabApr 28, 20263 min readCurated summary

How to build CI/CD observability at scale

CI/CD observability is essential for improving pipeline performance at enterprise scale, particularly in self-managed GitLab environments. The post presents a containerized solution built with `gitlab-ci-pipelines-exporter`, Prometheus, Grafana, and Node Exporter to turn pipeline and infrastructure data into actionable insights. Its conclusion is that centralized dashboards help teams identify bottlenecks, plan runner capacity, and measure delivery performance. ## Defining CI/CD Performance - Teams should first determine: - Which metrics matter, such as pipeline duration, job success rates, queue times, and runner utilization. - Who needs access, including developers, DevOps engineers, platform teams, and leadership. - Which decisions the data will support, such as infrastructure investment, bottleneck remediation, and capacity planning. ## Observability Architecture - The solution uses two exporters: - **Pipeline Exporter:** Collects pipeline duration, job status, and deployment metrics through the GitLab API. - **Node Exporter:** Collects host CPU, memory, and disk metrics for infrastructure correlation. - Prometheus gathers and stores the metrics. - Grafana provides real-time and historical dashboards. - Dashboards are provisioned automatically through Grafana’s file-based provisioning and can be filtered by project, branch, or time range. ## Grafana Dashboards - **Pipeline Overview:** Displays pipeline volume, success and failure rates, cancelled runs, and average duration trends. - **Job Performance:** Shows job-duration histograms, the ten slowest jobs, and failure heatmaps by project and stage. - **Runner & Infrastructure:** Correlates runner queue times with CPU, memory, and disk usage to support capacity planning. - **Deployment Frequency:** Tracks deployment counts and durations by environment, supporting DORA-style delivery analysis and detection of environment drift. ## Kubernetes Deployment - The recommended enterprise deployment runs each component as a separate workload in a dedicated `gitlab-observability` namespace. - A Kubernetes secret stores the GitLab personal access token, which requires the `read_api` scope. - The Pipeline Exporter runs as a Deployment with a service on port `8080`. - Node Exporter runs as a DaemonSet so each node can expose host metrics on port `9100`. - Prometheus and Grafana are deployed alongside the exporters and configured to scrape and visualize their metrics. - Kubernetes deployment supports existing cluster infrastructure, secrets managers, network policies, and scalable operations. ## Prerequisites - GitLab Self-Managed 18.1 or later. - Kubernetes for enterprise deployments, or Docker/Podman for smaller environments and proof-of-concept testing. - A GitLab personal access token with `read_api` permissions. - Secure secret-management practices, preferably using external secret operators in production. The practical recommendation is to begin with clearly defined performance questions, then deploy the exporter–Prometheus–Grafana stack in a controlled namespace. Combining pipeline data with host metrics provides the context needed to distinguish inefficient jobs from infrastructure capacity problems.

Read original

Prometheus Kubernetes Observability CI/CD+4

slackMar 31, 20262 min readCurated summary

From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus

Slack needed better client-side observability while migrating edge services to HTTP/3, which uses QUIC over UDP rather than TCP. Existing SaaS tools and Prometheus Blackbox Exporter could not probe HTTP/3 endpoints, so an intern added QUIC support using Go’s `quic-go` library and open-sourced it. The result unified HTTP/1.1, HTTP/2, and HTTP/3 monitoring while making the capability available to the broader Prometheus community. ## Limitations of Legacy Monitoring - Slack used a mix of commercial monitoring services and internal tools for network measurements. - HTTP/3 introduced a major observability gap because it runs over QUIC/UDP. - Existing SaaS solutions lacked built-in HTTP/3 probing. - Prometheus Blackbox Exporter had no native QUIC support. - Without probing at scale, Slack could not reliably measure round-trip times, detect regressions to HTTP/2, or monitor hundreds of thousands of HTTP/3 endpoints. ## Adding QUIC Support to Blackbox Exporter - Intern Sebastian Feliciano selected `quic-go` because of its adoption and first-class Go HTTP client support. - The implementation used an `http3.Transport` with TLS and QUIC configuration: ```go http3Transport := &http3.Transport{ TLSClientConfig: tlsConfig, QUICConfig: &quic.Config{}, } ``` - The new transport was attached to a standard Go `http.Client`. - The implementation preserved Blackbox Exporter’s existing configuration and composability patterns. - Sebastian open-sourced the feature and eventually got it accepted upstream. ## In-House Integration and Operational Benefits - Because upstream review could take longer than the internship timeline, Slack built an internal system around the new functionality. - Grafana now provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics. - Operators can compare protocol performance and correlate it with other telemetry. - Improved visibility supports more accurate alerts and faster debugging of HTTP/3 issues. ## Future Enhancements - **SNI routing tests:** Verify that shared edge infrastructure routes hostnames to the correct backend and presents the correct TLS certificate. - **End-to-end path visualization:** Map network hops between monitoring agents and endpoints to identify latency spikes or packet loss more precisely. ## Broader Lessons - Observability should be established before a major protocol or infrastructure migration. - Filling gaps through open source can benefit both the organization and the wider engineering community. - Supporting emerging protocols such as QUIC early helps future-proof monitoring systems. Slack recommends trying the new QUIC functionality in Prometheus Blackbox Exporter and contributing to its continued development.

Read original

Prometheus Observability Go Quic+4

lineMar 19, 20264 min readCurated summary

Utilizing SLI/SLO to Improve Reliability Part 1: SLI/SLO Framework and the Development Story of Service Status Check Tool LINE Status

Repeated SLI/SLO adoption revealed a common process that could be standardized across services. The team turned that process into a reusable framework and built “LINE Status,” an internal tool that automatically presents service health according to user experience rather than raw alerts. Together, these initiatives create a shared organizational language for understanding reliability and its impact on users. ## A Reusable SLI/SLO Framework After applying SLI/SLOs to several platforms and services, the SRE team identified recurring patterns independent of service type. They organized these patterns into a five-stage framework: - **Select critical user journeys (CUJs) and define SLIs** - Identify the experiences most important to users. - Define measurable SLIs that represent those experiences. - **Design instrumentation and metrics** - Build or adapt metrics suitable for each CUJ. - Use standardized naming based on Prometheus or OpenTelemetry. - **Create dashboards and recording rules** - Provide Grafana dashboards for quickly assessing SLO achievement. - Precompute complex PromQL operations to improve query performance. - **Set SLOs and alerts** - Begin with flexible targets, such as 99.9% availability over a 28-day rolling window, allowing roughly 40 minutes of downtime. - Define runbooks for responding to alerts. - Refine targets after operational data and experience accumulate. - **Establish error-budget governance** - Balance release speed against reliability. - Review objectives monthly or quarterly. - Adjust SLOs and processes as needed. The framework is currently distributed as a Confluence template containing guidance and FAQs, reducing the communication effort required from SREs during initial adoption. ## Moving from Alerts to User-Centered Service Status As more services adopted SLI/SLOs, the team wanted a consistent way to understand the health of services they did not directly operate. - The existing public LINE Status API page focused on external users and was updated manually during major incidents. - The new internal tool was intended to: - Represent the status of individual service components. - Update automatically from SLI/SLO alerts and outage data. - Show whether user experience was being affected. - Rather than simply reflecting whether an alert or outage existed, status was based on CUJ-related SLI performance and SLO achievement. - Only representative, high-value CUJs were exposed, avoiding unnecessary technical detail. ## LINE Status Architecture and Interface LINE Status was designed as more than an alert list. It collects events through webhooks, stores them in a separate database, and uses that data to track both current status and historical changes. - Technical SLI/SLO terms are translated into user-facing functions such as “Message Sending” or “Read Receipts.” - Status colors provide an immediate overview: - Green: normal - Yellow: event detected - Red: outage - The main page provides: - An overview of all services. - CUJ status within each service card. - AI-generated one-line summaries. - Service detail pages provide: - Recently affected items near the top. - Timeline-based event displays. - Monthly historical events. - The history page shows: - The scope of impact for each service during an event. - Past events organized by month. The initial implementation took about a month and was refined through colleague feedback. The author also used AI-assisted “vibe coding” for the frontend, emphasizing that clear, detailed requirements were more important than the development tool itself. ## Connecting the Framework and LINE Status Once a service adopts SLI/SLOs through the framework, it can be registered in LINE Status. This connects the definition of reliability objectives with an organization-wide view of service health. - Developers and operators can use the same CUJ-based standards. - Teams can focus on whether users are affected instead of interpreting isolated alerts. - During incidents, the tool helps identify impacted experiences quickly. - Over time, the approach may improve decision-making speed and cross-team communication. The team plans to refine CUJs, SLIs, and status-transition rules through continued operational experience. The practical goal is to make SLI/SLOs a common language for describing service health, enabling reliability practices to scale without depending heavily on individual teams or specialists.

Read original

Prometheus Database Design API Opentelemetry+3

tossJan 20, 2026Original article

The story of how I destroyed (opens in new tab)

Toss Payments modernized its inherited legacy infrastructure by building an OpenStack-based private cloud to operate alongside public cloud providers in an Active-Active hybrid configuration. By overcoming extreme technical debt—including servers burdened with nearly 2,000 manual routing entries—the team achieved a cloud-agnostic deployment environment that ensures high availability and cost efficiency. The transformation demonstrates how a small team can successfully implement complex open-source infrastructure through automation and the rigorous technical internalization of Cluster API and OpenStack. ### The Challenge of Legacy Networking - The inherited infrastructure relied on server-side routing rather than network equipment, meaning every server carried its own routing table. - Some legacy servers contained 1,997 individual routing entries, making manual management nearly impossible and preventing efficient scaling. - Initial attempts to solve this via public cloud (AWS) faced limitations, including rising costs due to exchange rates, lack of deep visibility for troubleshooting, and difficulties in disaster recovery (DR) configuration between public and on-premise environments. ### Scaling OpenStack with a Two-Person Team - Despite having only two engineers with no prior OpenStack experience, the team chose the open-source platform to maintain 100% control over the infrastructure. - The team internalized the technology by installing three different versions of OpenStack dozens of times and simulating various failure scenarios. - Automation was prioritized using Ansible and Terraform to manage the lifecycle of VMs and load balancers, enabling new instance creation in under 10 seconds. - Deep technical tuning was applied, such as modifying the source code of the Octavia load balancer to output custom log formats required for their specific monitoring needs. ### High Availability and Monitoring Strategy - To ensure reliability, the team built three independent OpenStack clusters operating in an Active-Active configuration. - This architecture allows for immediate traffic redirection if a specific cluster fails, minimizing the impact on service availability. - A comprehensive monitoring stack was implemented using Zabbix, Prometheus, Mimir, and Grafana to collect and visualize every essential metric across the private cloud. ### Managing Kubernetes with Cluster API - To replicate the convenience of Public Cloud PaaS (like EKS), the team implemented Cluster API to manage the Kubernetes lifecycle. - Cluster API treats Kubernetes clusters themselves as resources within a management cluster, allowing for standardized and rapid deployment across the private environment. - This approach ensures that developers can deploy applications without needing to distinguish between the underlying cloud providers, fulfilling the goal of "cloud-agnostic" infrastructure. ### Practical Recommendation For organizations dealing with massive technical debt or high public cloud costs, the Toss Payments model suggests that a "Private-First" hybrid approach is viable even with limited headcount. The key is to avoid proprietary black-box solutions and instead invest in the technical internalization of open-source tools like OpenStack and Cluster API, backed by a "code-as-infrastructure" philosophy to ensure scalability and reliability.

Prometheus Kubernetes DevOps Infrastructure As Code+4

lineAug 1, 2025Original article

Replacing the Payment System DB Handling (opens in new tab)

The LINE Billing Platform successfully migrated its large-scale payment database from Nbase-T to Vitess to handle high-traffic global transactions. While initially exploring gRPC for its performance reputation, the team transitioned to the MySQL protocol to ensure stability and reduce CPU overhead within their Java-based environment. This implementation demonstrates how Vitess can manage complex sharding requirements while maintaining high availability through automated recovery tools. ### Protocol Selection and Implementation - The team initially attempted to use the gRPC protocol but encountered `http2: frame too large` errors and significant CPU overhead during performance testing. - Manual mapping of query results to Java objects proved cumbersome with the Vitess gRPC client, leading to a shift toward the more mature and recommended MySQL protocol. - Using the MySQL protocol allowed the team to leverage standard database drivers while benefiting from Vitess's routing capabilities via VTGate. ### Keyspace Architecture and Data Routing - The system utilizes a dual-keyspace strategy: a "Global Keyspace" for unsharded metadata and a "Service Keyspace" for sharded transaction data. - The Global Keyspace manages sharding keys using a "sequence" table type to ensure unique, auto-incrementing identifiers across the platform. - The Service Keyspace is partitioned into $N$ shards using a hash-based Vindex, which distributes coin balances and transaction history. - VTGate automatically routes queries to the correct shard by analyzing the sharding key in the `WHERE` clause or `INSERT` statement, minimizing cross-shard overhead. ### MySQL Compatibility and Transaction Logic - Vitess maintains `REPEATABLE READ` isolation for single-shard transactions, while multi-shard transactions default to `READ COMMITTED`. - Advanced features like Two-Phase Commit (2PC) are available for handling distributed transactions across multiple shards. - Query execution plans are analyzed using `VEXPLAIN` and `VTEXPLAIN`, often managed through the VTAdmin web interface for better visibility. - Certain limitations apply, such as temporary tables only being supported in unsharded keyspaces and specific unsupported SQL cases documented in the Vitess core. ### Automated Operations and Monitoring - The team employs VTOrc (based on Orchestrator) to automatically detect and repair database failures, such as unreachable primaries or replication stops. - Monitoring is centralized via Prometheus, which scrapes metrics from VTOrc, VTGate, and VTTablet components at dedicated ports (e.g., 16000). - Real-time alerts are routed through Slack and email, using `tablet_alias` to specifically identify which MySQL node or VTTablet is experiencing issues. - A web-based recovery dashboard provides a history of automated fixes, allowing operators to track the health of the cluster over time. For organizations migrating high-traffic legacy systems to a cloud-native sharding solution, prioritizing the MySQL protocol over gRPC is recommended for better compatibility with existing application frameworks and reduced operational complexity.

Prometheus Database Design Grpc Mysql+4