Opensearch | Techlist.io

slackJul 14, 20263 min readCurated summary

Shipyard: How We Built Slack’s Next-Generation EC2 Platform

Slack’s Shipyard is a next-generation EC2 platform that replaces continuously modified, long-lived instances with immutable, deployable infrastructure artifacts. It combines layered machine images, service-level deployments, progressive rollouts, automated rollback, and short-lived instances to make EC2 operations more predictable and secure. The platform preserves EC2’s flexibility for workloads that cannot easily move to containers while adopting modern application-delivery practices. ## Why Slack Built Shipyard - Slack previously improved its Chef infrastructure with: - Multiple resilient Chef stacks - Versioned cookbook deployments - Safer promotion workflows - Split production environments and signal-based Chef runs - Despite these improvements, the traditional model of continuously updating instances still caused: - Infrastructure drift - Difficult service-level deployments - Coordination problems across infrastructure layers - Increasing operational complexity - Shipyard shifts infrastructure management from mutable instances and constant configuration enforcement to build pipelines, deployable artifacts, and automated safety mechanisms. ## Shipyard’s Core Capabilities - Supports AMD64 and ARM-based Graviton processors. - Supports Ubuntu, RHEL, and Amazon Linux. - Targets workloads that cannot easily migrate to containers, including: - Infrastructure services - Kubernetes worker nodes - Egress network stacks - Integrates with Slack’s Gondola deployment orchestrator for progressive rollouts. - Uses service health metrics to automatically pause deployments or roll back to a known-good version. - Uses layered images so instances perform less work during startup and provision quickly and consistently across regions. ## Configuration Management Without Continuous Mutation - Under the previous model, scheduled Chef jobs repeatedly checked and reapplied configuration. - Shipyard applies configuration during defined lifecycle stages, primarily: - Image baking - Initial instance provisioning - Service deployment - Configuration management tools no longer continuously modify the entire running system. - This reduces background workload, prevents unexpected overwrites, and makes instance behavior easier to understand. ## Fleet Inventory with Peekaboo - Shipyard introduces Peekaboo, a near-real-time inventory system for EC2. - Peekaboo uses AWS EventBridge, OpenSearch, and Lambda to collect cloud events and instance metadata. - It provides: - A fleet-exploration UI - An API for integrations - A command-line interface - Unlike Chef Server, Peekaboo is not limited to Shipyard-managed instances and can provide visibility across the entire EC2 fleet. ## Short-Lived Immutable Instances - Shipyard regularly rotates instances instead of relying on in-place updates. - Limited instance lifespans: - Reduce the window in which vulnerabilities can persist - Prevent long-term configuration drift - Encourage teams to replace instances rather than repair them manually - This makes infrastructure more closely resemble immutable application artifacts. ## The `slack-zero` Golden Image - `slack-zero` is Slack’s shared foundational AMI, maintained by the Compute Platform Team with security and monitoring teams. - It includes: - Operating-system hardening - Networking and service discovery - Monitoring and security agents - Common tools and foundational configuration - Service-specific images are built on top of `slack-zero`, similar to layering application images on a Docker base image. - When foundational components change, Slack produces a new immutable base image and rebuilds downstream service images to inherit updates. ## AWS Image Builder - Slack uses AWS Image Builder instead of Packer to create `slack-zero`. - Image Builder provides lifecycle policies that automatically remove old AMIs and reduce storage costs. - Each new image publishes its latest AMI identifier through an AWS Systems Manager parameter, helping downstream systems discover the current base image. Shipyard’s central recommendation is to treat EC2 infrastructure as versioned, replaceable artifacts rather than mutable machines. Combining trusted base images, automated provisioning, metric-driven deployment controls, fleet-wide inventory, and scheduled instance replacement gives Slack safer and more predictable EC2 operations at scale.

Read original

Opensearch AWS Amazon Ec2 Chef+4

awsJun 29, 20263 min readCurated summary

AWS Weekly Roundup: Agentic CX designer for Amazon Connect Customer, EC2 AMI Watermarks, Open Governance for MySQL, and more (June 29, 2026) | Amazon Web Services

The AWS Weekly Roundup highlights tools aimed at making AI, infrastructure management, and cloud operations faster and more accessible. The main announcement is Amazon Connect Customer’s no-code Agentic CX designer, which lets business teams create governed AI customer experiences without relying on lengthy engineering backlogs. Other updates cover isolated serverless compute, AMI governance, guided migrations, AI-assisted security investigations, and broader community initiatives. ## Agentic Customer Experience Design - Amazon Connect Customer launched the Agentic CX designer (NLX) in preview. - The no-code canvas enables business teams to design, test, simulate, and deploy voice and digital self-service experiences. - It combines agentic and deterministic AI within a governed workflow. - AWS also introduced Live Sync in preview, allowing web or mobile interfaces to update in real time as customers speak or type. - Customers could, for example, complete forms or open product pages while continuing a voice conversation. ## New AWS Infrastructure and Operations Features - **AWS Lambda MicroVMs** - Provides VM-level isolation with near-instant startup and resume times. - Supports suspending and resuming execution for up to eight hours. - Targets multi-tenant applications running user-generated or AI-generated code. - **Amazon EC2 AMI Watermarks** - Embeds custom identifiers in private AMIs. - Watermarks persist across copies, Regions, and account shares. - Works with Allowed AMIs and Declarative Policies to enforce approved-image usage. - **AWS Outposts lifecycle management** - Adds self-service configuration, quoting, ordering, subscription management, renewal, and decommissioning. - A new quoting tool provides rapid cost estimates and identifies account or regional constraints. ## AI-Assisted Developer and Migration Tools - **Amazon MSK AI Agent Skills** gives coding assistants such as Kiro, Claude Code, and Cursor operational guidance for Amazon MSK. - It supports Kafka sizing, configuration, troubleshooting, monitoring, and migrations to MSK Express. - **Amazon OpenSearch Service Migration Assistant** now offers agent-guided migrations from Solr, Elasticsearch, and OpenSearch to managed clusters or OpenSearch Serverless. - The migration tooling adds live traffic capture and replay for Solr workloads. ## AI-Powered Security Investigations - Amazon GuardDuty’s AI-powered investigations entered preview. - It analyzes findings, account context, related activity from the previous 90 days, knowledge graphs, and threat intelligence. - Investigations produce confidence-scored assessments, MITRE ATT&CK classifications, and recommended actions to help distinguish real threats from benign activity. ## Open Governance and AWS Community Updates - Oracle announced a community governance model for MySQL, including four non-Oracle seats on a new Steering Committee and a public GitHub presence. - AWS supports the initiative and contributes fixes upstream. - AWS Certification holders can renew eligible Associate and Professional certifications for an additional year through selected Skill Builder training and hands-on labs instead of retaking an exam. - The 2026 All Builders Welcome Grant offers selected early-career builders conference admission, airfare, and lodging for AWS re:Invent. AWS’s latest releases broadly point toward more self-service cloud management: business users can design AI experiences, developers can receive operational guidance from coding assistants, and teams can apply stronger controls to infrastructure and security workflows.

Read original

Opensearch AWS Apache Kafka Serverless+4

lineMar 20, 20264 min readCurated summary

Internalizing without specifications: Proving equivalence through validation logic

The article describes how LINE Plus safely internalized black-box e-commerce systems without specifications or source code. The team built an automated equivalence-testing loop using Kafka, CDC, OpenSearch, and ksqlDB to compare legacy and new behavior at massive scale. By repeatedly identifying differences, fixing logic, and rechecking results, they could reduce discrepancies toward zero while also measuring performance and protecting production stability. ## Domain: Products, Catalogs, and Data Ingestion - **Products** are individual seller-listed items, potentially with different prices and shipping conditions. - **Catalogs** group products representing the same model or product type and provide derived value such as: - Real-time lowest prices - Unit-price metrics such as price per 100 ml - **Ingestion** receives large product files from sellers, validates and transforms them into internal formats, and updates product and catalog data. - Because the platform contains tens of millions of catalogs and hundreds of millions of products, small logic differences can affect the entire service. ## The Verification Loop - The goal was not merely to find errors, but to help developers understand and correct them quickly. - Inputs had to be identical for both systems, such as: - The same IDs - The same time-based snapshot - The same product files - Outputs were compared according to system type: - API response objects - Database update values - Final registered product data - The general loop consisted of: - **Trigger:** Database changes, developer requests, or file arrivals - **Execution:** Send identical inputs to legacy and new systems - **Comparison:** Apply logic suited to reads, updates, or end-to-end flows - **Processing:** Store detailed differences and produce real-time statistics - **Action:** Developers inspect dashboards or Slack alerts, fix the implementation, and repeat ## Query Logic Verification - The catalog API was difficult to reproduce because it had over 100 response fields, complex filters, undocumented defaults, and unknown sorting behavior. - CDC streamed database binary-log changes into Kafka, allowing verification to begin from many real catalog states. - The verifier made dual API calls and compared legacy and new responses field by field. - Responses were converted into `Map<String, Object>` structures and compared recursively, avoiding the need to model every response class. - If values differed only because list ordering varied, the verifier sorted serialized values and performed a second comparison. - This helped distinguish real implementation defects from harmless ordering differences. - Kafka isolated verification traffic from production services while handling large event volumes. - Difference events were written to Kafka topics and indexed in OpenSearch for detailed investigation. - ksqlDB aggregated streaming discrepancies and sent Slack notifications when abnormal patterns appeared. - Rate limiting restricted repeated errors, such as those from the same field, to a manageable sample per minute. - Because both APIs were called in parallel, the same pipeline also measured and compared their response times. ## Update Logic Verification - The second case involved recalculating catalog statistics whenever product or catalog data changed. - Unlike read verification, this process tested state transitions and asynchronous updates. - When CDC detected a relevant change: - The new statistics logic calculated an expected result. - The verifier compared it with the result actually written by the legacy logic. - Recursive Map-based comparison checked deeply nested statistics fields. - To avoid wasting resources, verification was triggered only for updates related to the catalog-statistics module. ## Handling Asynchronous Lag - Kafka-based processing caused timing gaps: the verifier could read the database before the legacy update had completed. - The team introduced an **N-attempt retry queue**: - Temporarily inconsistent events were requeued. - Only differences that remained after several retries were treated as genuine defects. - The verifier remained a separate process rather than being embedded in the production statistics stream. - This avoided adding load or latency to the existing processing pipeline while preserving independent verification. ## ETL Batch Verification for Missing Triggers - Real-time comparison could detect incorrect results, but not cases where an update should have happened and never occurred. - During refactoring, a complex combination of product and catalog field changes contained a missing trigger condition. - As a result, some statistics remained stale without generating any comparison event. - To detect these silent omissions, the team designed a separate batch-verification process using ETL data alongside the real-time stream checks. The practical recommendation is to treat system internalization as an evidence-building process: define identical inputs and observable outputs, compare legacy and replacement systems continuously, isolate verification through event streams, and supplement real-time checks with batch validation for silent or missing updates.

Read original

Opensearch Database Design PostgreSQL Apache Kafka+4

naverDec 17, 2025Original article

Adopting an Intelligent Log Pipeline Aim (opens in new tab)

Naver’s Logiss platform, responsible for processing tens of billions of daily logs, evolved its architecture to overcome systemic inefficiencies in resource utilization and deployment stability. By transitioning from a rigid, single-topology structure to an intelligent, multi-topology pipeline, the team achieved zero-downtime deployments and optimized infrastructure costs. These enhancements ensure that critical business data is prioritized during traffic surges while minimizing redundant storage for search-optimized indices. ### Limitations of the Legacy Pipeline * **Deployment Disruptions:** The previous single-topology setup in Apache Storm lacked a "swap" feature, requiring a total shutdown for updates and causing 3–8 minute processing lags during every deployment. * **Resource Inefficiency:** Infrastructure was provisioned based on daytime peak loads, which are five times higher than nighttime traffic, resulting in significant underutilization during off-peak hours. * **Indiscriminate Processing:** During traffic spikes or hardware failures, the system treated all logs equally, causing critical service logs to be delayed alongside low-priority telemetry. * **Storage Redundancy:** Data was stored at 100% volume in both real-time search (OpenSearch) and long-term storage (Landing Zones), even when sampled data would have sufficed for search purposes. ### Transitioning to Multi-Topology and Subscribe Mode * **Custom Storm Client:** The team modified `storm-kafka-client` 2.3.0 to revert from the default `assign` mode back to the `subscribe` mode for Kafka partition management. * **Partition Rebalancing:** While `assign` mode is standard in Storm 2.x, it prevents multiple topologies from sharing a consumer group without duplication; the custom `subscribe` implementation allows Kafka to manage rebalancing across multiple topologies. * **Zero-Downtime Deployments:** This architectural shift enables rolling updates and canary deployments by allowing new topologies to join the consumer group and take over partitions without stopping the entire pipeline. ### Intelligent Traffic Steering and Sampling * **Dynamic Throughput Control:** The "Traffic-Controller" (Storm topology) monitors downstream load and diverts excess non-critical traffic to a secondary "retry" path, protecting the stability of the main pipeline. * **Tiered Log Prioritization:** The system identifies critical business logs to ensure they bypass bottlenecks, while less urgent logs are queued for post-processing during traffic surges. * **Storage Optimization via Sampling:** Logiss now supports per-destination sampling rates, allowing the system to send 100% of data to long-term Landing Zones while only indexing a representative sample in OpenSearch, significantly reducing indexing overhead and storage costs. ### Results and Recommendations The implementation of an intelligent log pipeline demonstrates that modifying core open-source components, such as the Storm-Kafka client, can be a viable path to achieving specific architectural goals like zero-downtime deployment. For high-volume platforms, moving away from a "one-size-fits-all" processing model toward a priority-aware and sampling-capable pipeline is essential for balancing operational costs with system reliability. Organizations should evaluate whether their real-time search requirements truly necessitate 100% data ingestion or if sampling can provide the necessary insights at a fraction of the cost.

Opensearch Apache Kafka Data Engineering Apache Storm+3

lineMay 23, 2025Original article

Implementing a RAG-based (opens in new tab)

To address the operational burden of handling repetitive user inquiries for the AWX automation platform, LY Corporation developed a support bot utilizing Retrieval-Augmented Generation (RAG). By combining internal documentation with historical Slack thread data, the system provides automated, context-aware answers that significantly reduce manual SRE intervention. This approach enhances service reliability by ensuring users receive immediate assistance while allowing engineers to focus on high-priority development tasks. ### Technical Infrastructure and Stack * **Slack Integration**: The bot is built using the **Bolt for Python** framework to handle real-time interactions within the company’s communication channels. * **LLM Orchestration**: **LangChain** is used to manage the RAG pipeline; the developers suggest transitioning to LangGraph for teams requiring more complex multi-agent workflows. * **Embedding Model**: The **paraphrase-multilingual-mpnet-base-v2** (SBERT) model was selected to support multi-language inquiries from LY Corporation’s global workforce. * **Vector Database**: **OpenSearch** serves as the vector store, chosen for its availability as an internal PaaS and its efficiency in handling high-dimensional data. * **Large Language Model**: The system utilizes **OpenAI (ChatGPT) Enterprise**, which ensures business data privacy by preventing the model from training on internal inputs. ### Enhancing LLM Accuracy through RAG and Vector Search * **Overcoming LLM Limits**: Traditional LLMs suffer from "hallucinations," lack of up-to-date info, and opaque sourcing; RAG fixes this by providing the model with specific, trusted context during the prompt phase. * **Embedding and Vectorization**: Textual data from wikis and chats are converted into high-dimensional vectors, where semantically similar phrases (e.g., "Buy" and "Purchase") are stored in close proximity. * **k-NN Retrieval**: When a user asks a question, the bot uses **k-Nearest Neighbors (k-NN)** algorithms to retrieve the top *k* most relevant snippets of information from the vector database. * **Contextual Generation**: Rather than relying on its internal training data, the LLM generates a response based specifically on the retrieved snippets, leading to higher accuracy and domain-specific relevance. ### AWX Support Bot Workflow and Data Sources * **Multi-Source Indexing**: The bot references two main data streams: the official internal AWX guide wiki and historical Slack inquiry threads where previous solutions were discussed. * **Automated First Response**: The workflow begins when a user submits a query via a Slack workflow; the bot immediately processes the request and provides an initial AI-generated answer. * **Human-in-the-Loop Validation**: After receiving an answer, users can click "Issue Resolved" to close the ticket or "Call AWX Admin" if the AI's response was insufficient. * **Efficiency Gains**: This tiered approach filters out "RTFM" (Read The F***ing Manual) style questions, ensuring that human administrators only spend time on unique or complex technical issues. Implementing a RAG-based support bot is a highly effective strategy for SRE teams looking to scale their internal support without increasing headcount. For the best results, organizations should focus on maintaining clean internal documentation and selecting embedding models that reflect the linguistic diversity of their specific workforce.

Opensearch AI Python RAG+5

figmaOct 9, 20242 min readCurated summary

The Search for Speed in Figma | Figma Blog

Figma’s search investigation revealed that OpenSearch itself was responsible for less than 30% of total search latency. The larger costs came from query construction and especially permission checks before and after searches. By measuring the correct end-to-end metrics, Figma identified misleading monitoring data and established a foundation for improving search performance at scale. ## Diagnosing the Latency Gap - Figma migrated from an older Elasticsearch version to AWS-managed OpenSearch, a fork created after Elasticsearch’s 2021 license change. - OpenSearch reported an average search time of roughly **8 ms**, while Figma’s API showed: - About **150 ms average latency** - **200–400 ms** latency at the 99th percentile - Minimum latency above **40 ms** - Search performance also varied significantly depending on traffic levels, with peak periods much slower than weekends. - Additional instrumentation showed that substantial time was spent both before and after the OpenSearch request. ## Understanding OpenSearch’s Metrics - OpenSearch distributes a query through a coordinator node to worker nodes, typically sending one request per index shard. - It then gathers, sorts, and fetches results during the query and fetch phases. - The reported 8 ms metric measured only the average time for individual shard queries—not the total time required to coordinate hundreds of shard requests. - Figma’s queries could involve as many as **500 shard-level requests**, many of which ran in parallel but not all. - OpenSearch did not provide built-in metrics or logs for overall query duration. - Figma instead extracted the `took` value from each search response, producing a backend latency measure that aligned more closely with application-level timing. ## Permission Processing as the Main Bottleneck - Less than 30% of total query API time was spent waiting for OpenSearch. - Pre-processing: - Retrieved information about files the user could access. - Built an OpenSearch filter intended to exclude inaccessible files. - Post-processing: - Performed additional permission checks on every returned file. - Was especially slow and consumed more time than the search itself. - The investigation demonstrated that optimizing the search engine alone would not solve Figma’s overall latency problem. Figma’s experience highlights the importance of measuring end-to-end request latency rather than relying on subsystem metrics. Accurate coordinator-level and application-level instrumentation is essential, particularly when distributed searches involve many shards and expensive authorization work.

Read original

Opensearch Observability AWS Elasticsearch+2