line·Mar 31, 2026·3 min readCurated summary

Applying Spark on Kubernetes to process large-scale advertising data for LINE services

LINE Ads processes tens of billions of advertising events daily and nearly one hundred billion internal data records. As growing numbers of features increased computational demands, its Spark-on-YARN environment suffered from resource contention, inefficient scaling, and Hadoop dependencies. The team migrated to Spark on Kubernetes to achieve infrastructure independence, containerized execution, flexible scaling, and easier operational automation. ## Large-Scale LINE Ads Data Pipelines - The data pipeline supports: - Real-time advertising-event processing - Abuse and validity checks - Machine-learning systems and model training - Analytics and system integration - Advertiser reporting - The platform must handle hundreds of billions of events per day and hundreds of thousands per second. - It must provide low latency, elastic capacity, minimal service impact during failures, and rapid recovery. - The most heavily used table grew to approximately 2.91 times its December 2022 size by December 2025 as more features were added. ## Limitations of Spark on YARN - Hadoop’s storage and compute resources were colocated, causing Spark workloads to compete with HDFS and other Hadoop components. - Scaling compute required adding Hadoop nodes, even when additional storage was unnecessary, increasing cost and wasting capacity. - JVM and Spark versions were difficult to manage independently, limiting access to newer Spark features. - Applications became tightly coupled to the Hadoop infrastructure. ## How Spark on Kubernetes Works - Kubernetes replaces YARN as the cluster manager. - Spark drivers and executors run as separate Kubernetes pods. - In cluster mode: - `spark-submit` requests a driver pod. - Kubernetes schedules the driver on an appropriate node. - The driver creates a `SparkContext`, builds the DAG, and requests executors. - Executors run as independent pods with individually allocated CPU and memory. - The driver divides the DAG into stages and distributes tasks to executors. - Shuffle data is normally tied to executor-pod lifecycles unless an external shuffle service is configured. ## Advantages over YARN - **Containerized execution:** Docker images package application dependencies, improving reproducibility and CI/CD integration. - **Infrastructure independence:** Spark can use HDFS, S3, GCS, or other storage systems without requiring a Hadoop cluster. - **Simpler autoscaling:** Kubernetes can scale pods and integrate with cloud VM autoscalers. - **Unified platform:** Spark, Airflow, machine-learning workloads, and API servers can share a Kubernetes cluster. - **Governance and isolation:** Namespaces, `ResourceQuota`, and RBAC provide flexible team-level controls. - **Operational automation:** Helm, ArgoCD, GitOps, and rolling updates enable more automated application management. ## LINE Ads’ Kubernetes-Based System The platform is organized into four layers: - **Deployment layer** - GitHub Actions runs CI workflows based on repository events. - ArgoCD monitors desired and deployed states and supports easier rollback and synchronization. - **Compute layer** - Kubeflow’s Spark Operator deploys applications through the `SparkApplication` Kubernetes custom resource. - Apache YuniKorn schedules batch jobs and supports resource coordination and gang scheduling. - LogSender forwards pod logs to OpenSearch. - ClusterMonitoring sends Prometheus metrics to the company’s monitoring system. - **Storage layer** - Kafka provides high-throughput, low-latency storage for real-time advertising actions. - Hadoop remains available for large-scale, long-term analysis. - **Monitoring layer** - Kubernetes workers and Spark applications are monitored through exposed Prometheus metrics and centralized logging. The migration to Spark on Kubernetes is recommended for organizations whose Spark workloads are outgrowing tightly coupled Hadoop environments. It separates compute from storage, improves deployment flexibility, and allows data applications to be managed as cloud-native workloads.

Read original(opens in new tab)

Database DesignMachine LearningKubernetesApache Kafka+4

aws·Mar 30, 2026·3 min readCurated summary

AWS Weekly Roundup: AWS AI/ML Scholars program, Agent Plugin for AWS Serverless, and more (March 30, 2026) | Amazon Web Services

The March 30, 2026 AWS Weekly Roundup highlights the new AWS AI & ML Scholars program, which will offer free generative AI education to up to 100,000 people and fully funded Udacity Nanodegrees to 4,500 top participants. It also emphasizes new tools for serverless development, SageMaker IDE integration, expanded Lambda Managed Instance capacity, and streaming speech synthesis. AWS Summit and Community Day events are also scheduled worldwide. ## AWS AI & ML Scholars Program - Open to anyone aged 18 or older, with no prior AI or machine learning experience required. - Includes: - A foundational generative AI Challenge phase. - A fully funded three-month Udacity Nanodegree for the top 4,500 performers. - Applications close June 24, 2026. ## Serverless and Database Improvements - **Aurora PostgreSQL express configuration** enables serverless databases to be created and connected in seconds using preconfigured defaults. - **Aurora PostgreSQL is now part of the AWS Free Tier**, with eligible new customers receiving AWS credits. - The **Agent Plugin for AWS Serverless** adds skills, sub-agents, and Model Context Protocol servers to AI coding assistants such as Kiro, Claude Code, and Cursor. - It supports building, deploying, troubleshooting, and managing production-ready serverless applications. - The **Aurora DSQL Connector for Ruby** automatically generates authentication tokens for each connection while remaining compatible with the `pg` gem. ## SageMaker and AWS Console Updates - **SageMaker Studio** now supports remote connections from Kiro and Cursor, combining those IDEs’ coding workflows with SageMaker’s scalable compute. - The AWS Management Console now supports visual customization, including account colors and hiding unused regions or services to reduce interface clutter. ## Expanded Lambda Managed Instance Capacity - The file descriptor limit has increased from 1,024 to 4,096, supporting higher-concurrency and file-intensive workloads. - Functions can now use up to: - 32 GB of memory - 16 vCPUs - Users can select memory-to-vCPU ratios of 2:1, 4:1, or 8:1 for workloads such as data processing, media transcoding, and scientific simulations. ## Conversational Speech with Amazon Polly - Polly’s new Bidirectional Streaming API supports incremental text-to-speech generation. - Audio synthesis can begin before an LLM or other application has produced the complete response, making it better suited to conversational AI. ## Upcoming AWS Events - AWS Summits are free, in-person events covering cloud, AI, best practices, and networking. - Upcoming locations include Paris, London, Bengaluru, Singapore, Tel Aviv, and Stockholm. - AWS Community Days in San Francisco and Romania will feature community-led talks, workshops, and hands-on labs. AWS developers can follow the AWS News Blog and “What’s New with AWS” for additional announcements, while the AWS Builder Center and Events and Webinars pages provide opportunities for learning and community participation.

Read original(opens in new tab)

Database DesignAIPostgreSQLAWS+4

line·Mar 30, 2026·4 min readCurated summary

Image Content Moderation in Large-Scale Service Environments (feat. Multimodal LLM)

Image content moderation has evolved from simple rule-based filtering into an AI-powered decision system capable of handling visual context, text, and policy complexity. At large platforms, the challenge is not only accuracy but also latency, cost, scalability, and adaptability to changing policies. LY Corporation addresses these demands through optimized traditional ML models, a hybrid ML–multimodal LLM pipeline, and modular decision-making that combines OCR, visual analysis, and contextual reasoning. ## The Evolution of Content Moderation - Early systems relied on keyword matching, rule-based filters, and predefined patterns. - Machine learning enabled broader pattern recognition and detection of modified or less explicit violations. - Modern systems combine: - Deep learning for text and image classification - Multimodal models for joint image–text understanding - LLMs for context-sensitive judgments - Separate prediction and policy layers for operational flexibility - Despite these advances, image moderation remains difficult because images lack explicit structure and their meaning often depends on context. ## Why Image Moderation Is Difficult - **Visual complexity:** Backgrounds, objects, people, colors, and composition interact in ways that simple object detection cannot fully interpret. - **Context dependency:** Symbols, gestures, and imagery may have different meanings across cultures; embedded text can also determine whether an image is harmful. - **Evasion and variation:** Memes, composites, partially obscured images, and AI-generated edits continually challenge existing detectors. - **Scale requirements:** Platforms may receive millions or tens of millions of images daily, requiring high accuracy alongside low latency, reliability, and cost efficiency. ## LY Corporation’s Moderation API - LY Corporation operates a monitoring platform designed to process large-scale traffic and enforce diverse content policies. - Its image moderation API detects: - Adult content - Violent or graphic scenes - Offensive or disturbing imagery - Identity documents containing personal information - Social media screenshots and other policy-sensitive images - The system is designed to apply service-specific policies consistently while maintaining high throughput. ## Improving Accuracy, Speed, and Cost ### Traditional ML Model Optimization - A PyTorch-based image classification model was selected with latency, cost, and throughput in mind. - The model was converted to ONNX and optimized with FP16 precision. - ONNX Runtime improved execution efficiency, while FP16 reduced memory usage and inference time. - These changes increased throughput by up to **4.3 times**. ### Hybrid ML and Multimodal LLM Architecture - The traditional classifier acts as a fast first-stage filter. - Clear cases are resolved immediately by the image model. - Ambiguous cases are sent to a multimodal LLM for deeper analysis. - More than 90% of production data could be classified by the traditional model alone. - Since multimodal LLM throughput was over 100 times lower than that of the traditional model, routing every image to the LLM would have significantly increased GPU usage and cost. - The hybrid approach preserves high-quality reasoning where necessary while avoiding unnecessary LLM calls. ### vLLM-Based LLM Optimization The team optimized multimodal LLM serving with vLLM, using characteristics such as repeated prompts, predictable token lengths, and prefill-heavy workloads. - **`enable_prefix_caching`:** Reuses KV-cache blocks for repeated system prompts and templates, reducing prefill computation. - **`max_model_len`:** Limits the maximum input-plus-output length to avoid excessive KV-cache allocation. - **`max_num_seqs`:** Controls concurrent requests, balancing throughput against per-request latency and resource contention. - **`max_num_batched_tokens`:** Sets the token budget per scheduling step; larger values can improve throughput for prefill-heavy workloads. - Regularly updating vLLM is recommended because new releases add improvements such as asynchronous scheduling, CUDA graph support, and broader quantization options. ## Moving Beyond Single-Model Policy Prediction - Earlier end-to-end vision models directly predicted final policy categories from images. - This worked for visually obvious violations, such as detecting smoking, but struggled with complex behaviors such as tobacco sales. - Sales-related judgments may require combining: - Product presence - Prices - Sales language - Contact information - Encouragement to purchase - Directly learning every combination of national regulations, service policies, and exceptions created overly complex output classes. - It also made the model harder to extend and maintain, while limiting the use of text embedded in images. ## Hybrid Decision-Making with OCR and Multimodal Reasoning - The redesigned system separates visual and textual information rather than forcing one model to learn every policy combination. - OCR extracts text from images when relevant. - Extracted text helps identify policy-violating behavior or intent. - Visual signals and textual evidence are then combined with a multimodal LLM. - This allows the system to reason about context and intent beyond simple object detection, while making policy logic more modular and adaptable. The practical recommendation is to avoid routing all traffic through expensive general-purpose models. Use fast specialized models for clear cases, reserve multimodal LLMs for ambiguity, optimize serving according to workload characteristics, and separate content understanding from policy decisions so the system can evolve as requirements change.

Read original(opens in new tab)

Database DesignMachine LearningDeep LearningModel Optimization+4

grammarly·Mar 27, 2026·3 min readCurated summary

How to Create a Chatbot Step by Step: A Beginner’s Guide

Chatbot development begins with a focused purpose, not technology. By choosing the right interaction method, chatbot type, and platform, organizations can automate routine tasks and improve user experiences without necessarily needing developers. Successful chatbots also require deliberate conversation design, testing, monitoring, and continuous improvement. ## What Chatbots Can Do - Simulate conversations through text or voice. - Answer frequently asked questions and provide information. - Handle structured tasks such as: - Checking order status - Booking appointments - Explaining policies - Guiding users through onboarding - AI-powered and hybrid chatbots can manage follow-up questions and more complex, multistep interactions. - They can reduce repetitive work, improve response consistency, and help users reach solutions faster. ## Define the Chatbot’s Goal - Identify two or three specific tasks the chatbot should handle. - Define the target audience, such as customers, employees, or students. - Establish success metrics, including: - Fewer support tickets - Faster response times - Higher task-completion rates - A narrow, well-defined purpose makes the chatbot easier to design, test, and refine. ## Choose the Interaction Method - Decide whether the chatbot will be text-based or voice-based. - Text is generally simpler to build. - Voice requires additional technical setup. - Choose where it will operate, such as: - A website - Mobile application - Messaging platform - Internal company tool - Determine how conversations begin, whether through typed messages, preset options, or proactive prompts. - The access point and interaction style directly affect development and maintenance requirements. ## Select the Chatbot Type - **Rule-based chatbots** use predefined flows, menus, and decision trees for predictable requests. - **Keyword-based chatbots** respond to specific words or short phrases, such as “pricing” or “hours.” - **AI chatbots** use artificial intelligence and natural language processing to handle varied questions and contextual follow-ups, but require more testing and oversight. - **Hybrid chatbots** combine structured rules for common tasks with AI for open-ended questions. - The choice determines the chatbot’s flexibility, behavior, complexity, and ongoing management effort. ## Choose a Building Platform - **No-code platforms** such as Chatling, Voiceflow, Zapier, and Landbot use visual interfaces and are suitable for beginners and simple chatbot tasks. - **Low-code or full-code approaches** using technologies such as Python, Node.js, or AI frameworks provide greater customization and integration capabilities. - Platform selection should account for: - Cost - Integrations - Analytics - Scalability - Data protection - Required technical expertise ## Design the Conversation Flow - Map typical conversations before implementing the chatbot. - Planning helps identify missing responses, avoid dead ends, and create a smoother user experience. - Traditional chatbots generally use structured decision paths, while AI chatbots support more flexible conversations. - The flow should reflect the chatbot’s purpose and provide a clear route for completing tasks or escalating complex issues. ## Ongoing Improvement - Building and launching the chatbot is only the beginning. - Chatbots should be tested before release and monitored afterward. - Regular refinement, accurate training data, configuration updates, and performance reviews help maintain quality over time. A practical approach is to start with a narrow use case and a simple platform, then expand as user needs and performance data become clearer. Choose AI or custom development only when the chatbot requires more flexibility, deeper integrations, or complex conversational capabilities.

Read original(opens in new tab)

Database DesignNatural Language ProcessingUser Experience DesignNo Code Tools+3

cloudflare·Mar 26, 2026·2 min readCurated summary

A one-line Kubernetes fix that saved 600 hours a year

Atlantis restarts were taking about 30 minutes, blocking infrastructure changes and consuming more than 50 engineering hours monthly. The delay was caused by Kubernetes recursively changing ownership on a large Ceph-backed PersistentVolume containing millions of files. Setting `fsGroupChangePolicy: OnRootMismatch` avoided unnecessary recursive ownership changes and reduced restart time dramatically. ### The Restart Bottleneck - Atlantis runs as a singleton Kubernetes `StatefulSet`. - Its PersistentVolume stores repository and Terraform state. - Credential rotations, onboarding, and offboarding required restarting Atlantis. - With roughly 100 restarts per month, each 30-minute delay created more than 600 hours of annual lost engineering time. - The volume had grown large enough to exhaust inodes, making storage expansion and pod restarts necessary. ### Kubernetes Made the Delay Look Like a Scheduling Problem - `kubectl rollout restart statefulset atlantis` terminated the old pod and created a replacement. - The new pod was scheduled quickly but remained stuck in `Init:0/1`. - Kubernetes events showed the image pulling successfully, but revealed no obvious cause for the long gap. - Kubelet logs showed the PersistentVolume mounting successfully, followed by repeated `context deadline exceeded` errors while syncing the pod. ### The Hidden Cost of `fsGroup` - Searching logs using the PersistentVolume name exposed the relevant message: - Kubernetes was “setting volume ownership” because an `fsGroup` was configured. - Kubernetes warned that ownership changes could be slow when a volume contained many files. - The default behavior recursively changed ownership across the entire mounted volume. - As Atlantis’s volume accumulated millions of files, this initialization step became the 30-minute bottleneck. ### The One-Line Fix - The volume configuration was changed to: ```yaml fsGroupChangePolicy: OnRootMismatch ``` - With this policy, Kubernetes checks the root directory’s ownership and only performs recursive changes when necessary. - Existing volumes with the correct ownership no longer require a full filesystem traversal during every restart. The practical lesson is to inspect kubelet and volume logs when a pod appears scheduled but remains stuck before initialization. For large persistent volumes, explicitly setting `fsGroupChangePolicy: OnRootMismatch` can eliminate costly recursive ownership changes and prevent substantial operational downtime.

Read original(opens in new tab)

Database DesignKubernetesTerraformPersistent Storage+4

aws·Mar 25, 2026·3 min readCurated summary

Announcing Amazon Aurora PostgreSQL serverless database creation in seconds | Amazon Web Services

Amazon’s new Aurora PostgreSQL express configuration lets developers create a serverless database in seconds with two console clicks or a single CLI/API call. It uses preconfigured defaults, IAM authentication, and an internet access gateway to simplify secure connections without requiring a VPC, VPN, or Direct Connect. The feature is designed to accelerate prototyping and application development while preserving Aurora capabilities such as read replicas and automated failover. ## Express Configuration for Aurora PostgreSQL - Creates an Aurora PostgreSQL serverless cluster and instance within seconds. - Uses preconfigured defaults to reduce setup complexity. - Allows customization of: - Cluster identifier - Serverless capacity range during creation - Read replicas and parameter groups after creation - Express-configured clusters do not require an Amazon VPC. - An internet access gateway is enabled by default for secure connections from development tools worldwide. - The gateway is distributed across multiple Availability Zones for high availability. - IAM authentication is configured for the administrator, enabling passwordless database authentication. ## Creating a Database - In the Aurora and RDS console: - Open the Dashboard. - Choose **Create** with the rocket icon. - Review or adjust the express configuration. - Choose **Create database**. - The AWS CLI and SDKs support the `--with-express-configuration` parameter. - A single `create-db-cluster` call creates both the cluster and its instance: ```bash aws rds create-db-cluster \ --db-cluster-identifier channy-express-db \ --engine aurora-postgresql \ --with-express-configuration ``` - The database becomes ready when its status changes to **Available**. ## Connecting to the Database The **Connectivity & security** tab provides several connection methods: - **Code snippets** - Generates connection examples for .NET, Go, JDBC, Node.js, PHP, PostgreSQL, Python, and TypeScript. - Python examples use `boto3` to generate an IAM authentication token and `psycopg2` to connect over SSL. - **AWS CloudShell** - Launches a shell with a preconfigured `psql` connection command. - Developers can immediately run SQL commands at the PostgreSQL prompt. - **Endpoints** - Supports tools such as pgAdmin that use username-and-password fields. - The password is an IAM authentication token valid for 15 minutes. - A new token must be generated if the connection ends or the token expires. ## Application Development Integrations - Aurora is now included among eligible AWS Free Tier database services. - AWS’s enhanced Free Tier offers up to $200 in credits: - $100 upon signup - Up to another $100 through usage of services such as RDS, Lambda, and Bedrock - Integrations with Vercel and v0 allow developers to create or connect to AWS databases quickly. - v0 can use natural-language prompts to generate full-stack applications backed by Aurora PostgreSQL, Aurora DSQL, or DynamoDB. - Existing Aurora databases created with express configuration can also be connected to Vercel. The express configuration is best suited for quickly starting development, experimentation, and prototypes. Developers can begin with minimal networking and authentication setup, then add capacity, replicas, and other Aurora features as their application grows.

Read original(opens in new tab)

Database DesignPostgreSQLServerlessAws Cli+4

dropbox·Mar 25, 2026·3 min readCurated summary

Reducing our monorepo size to improve developer velocity

Dropbox’s server monorepo grew to 87GB, making full clones take over an hour and threatening GitHub’s 100GB limit. The root cause was inefficient Git delta compression of internationalization files, not unusually large source files. By changing how the repository was repacked, Dropbox reduced it to about 20GB and cut clone times to under 15 minutes. ## Repository Size and Developer Velocity - The monorepo contains backend services and libraries used across Dropbox. - AI feature development often requires coordinated changes across ranking, retrieval, evaluation, and UI systems. - A full clone exceeded one hour at 87GB, slowing onboarding and affecting CI jobs that start from fresh clones. - Internal synchronization systems also processed more data, increasing timeout and reliability risks. - The repository grew by roughly 20–60MB per day, with occasional increases above 150MB. - At that rate, Dropbox expected to hit GitHub Enterprise Cloud’s 100GB hard limit within months. ## How Git Compression Caused the Growth - Git normally reduces storage by representing similar file versions as deltas rather than complete copies. - Its default file-matching heuristic considers only the final 16 characters of a path. - Dropbox’s i18n files used paths such as: - `i18n/metaserver/[language]/LC_MESSAGES/[filename].po` - Because the language component appears early in the path, Git often compared files from different languages instead of related versions of the same language. - Translation updates consequently produced oversized deltas and disproportionately large pack files. ## Testing `--path-walk` - Dropbox tested Git’s experimental `--path-walk` option during a local repack. - The option considers the full directory structure when selecting delta candidates. - A local repack reduced the repository from the low-80GB range to the low-20GB range, confirming that packing—not data volume—was the main issue. - GitHub could not use this approach because it conflicted with server-side optimizations such as bitmaps and delta islands. ## Why Server-Side Repacking Was Necessary - Local optimization cannot permanently change the packs GitHub generates for clones and fetches. - GitHub dynamically constructs transfer packs based on what each client needs. - Dropbox’s mirror experiment showed that an aggressive repack could reduce the repository from 84GB to 20GB: - `git repack -adf --depth=250 --window=250` - The repack took approximately nine hours. - Dropbox worked with GitHub Support to apply a compatible server-side solution. - Larger `window` and `depth` values make Git search more thoroughly for compression opportunities, trading increased repack time for smaller storage and transfer sizes. ## Results - Repository size fell from 87GB to approximately 20GB—a 77% reduction. - Clone time dropped from more than an hour to under 15 minutes. - The work reduced pressure on GitHub’s repository size limit and improved the performance of developer and CI workflows. Dropbox’s experience shows that monorepo growth can result from repository layout interacting poorly with Git’s compression heuristics. When large repositories exhibit abnormal growth, teams should inspect pack-file behavior and consider server-side repacking rather than focusing only on removing large files.

Read original(opens in new tab)

Database DesignCI/CDGithubGit+4

toss·Mar 25, 2026·3 min readCurated summary

Metric Review, Driving Execution

Metric Review is Toss Place’s weekly operating system for turning data insights into product and business action. By connecting OKRs to a hierarchy of driver metrics, analysts continuously detect risks, test hypotheses, and encourage execution rather than merely reporting results. The approach has improved data literacy and helped teams contribute directly to company-level Key Results. ### Building a Data-Literate Organization - Toss Place aims for everyone—not only analysts—to perform effective analysis. - The Data Platform Team strengthens data quality and infrastructure, while the Data Analysis Team provides domain knowledge and delivery capabilities. - Analysts are expected to develop three complementary skills: - Technical expertise with data and analysis tools - Logical communication - Deep product and business knowledge ### Why Metric Review Matters - Metrics serve as a shared language for aligning teams around organizational goals. - Metric Review helps teams identify: - Whether goals are on track - Emerging risks - New opportunities - Analysts act as **Metric Owners**, providing insights that support better decisions and following through until actions and outcomes are verified. ### Operating Model #### OKR-Linked Metric Hierarchy - Company-level Key Results flow down to team and silo-level Key Results. - The levers that influence each team’s KR become its driver metrics. - This hierarchy provides the structure for identifying opportunities and threats. #### A Continuous Analysis Cycle - The operating cycle is: - Goal setting → hypothesis formation → validation and execution → insight discovery - Metric Review translates this into: - Metric analysis → hypothesis testing → insight sharing → driving action - Exploratory data analysis (EDA) is also conducted when metric movements suggest deeper questions. #### Weekly Consistency - Reviewing metrics weekly helps teams detect small changes before they become significant. - Regular analysis also builds domain knowledge by requiring analysts to understand why metrics rise or fall. - Monthly or occasional reporting may explain past performance but often misses the window for timely action. ### Examples of Business Impact #### Growth Tribe: Establishing Shared Metrics - Weekly metric reviews initially focused on reporting performance and interpretation. - Over time, the practice changed how teams worked: - Designers defined product hypotheses around target metrics and incorporated logging requirements into designs. - Backend developers collaborated with analysts on analysis-friendly data structures. - Client developers prioritized measurable events when implementing logs. - Product Owners combined qualitative feedback with quantitative results to determine whether goals were on track. - This created a feedback loop that contributed to successful product launches and improved company metrics. #### POS Tribe: Segment-Specific Solutions - POS adoption varied significantly across partner dealerships. - Analysts used clustering to identify groups with different adoption patterns. - Product teams combined cluster analysis with interviews to design tailored interventions: - Low-adoption groups received stronger education and onboarding. - High-adoption groups received simplified store creation and installation flows. - Segment-specific actions accelerated POS expansion more effectively than a single broad solution. #### Supply Chain: Forecast-Based Optimization - Because Toss Place manufactures and distributes hardware, supply-chain metrics are strategically important. - Analysts and the SCM team monitored: - Device shipments - Market installation rates - Inventory and ordering forecasts - Potential improvement areas - Hypothesis-driven actions helped optimize distribution and reduce costs. ### How the Organization Changed - Analysts became Metric Owners rather than report writers. - Product teams began asking, “Which metric should we move?” before asking what to build. - Business teams increasingly aligned strategies using quantitative evidence. - Repeated Metric Reviews strengthened organization-wide data literacy and contributed to meaningful company Key Result achievement. The practical recommendation is to evaluate analysis by whether it leads to measurable action. Teams should structure problems, create testable hypotheses, define follow-up metrics, and maintain a consistent review rhythm until the execution loop is closed.

Read original(opens in new tab)

Database DesignData AnalysisOkrsData Platform+4

line·Mar 24, 2026·4 min readCurated summary

How We Built an SRE Bot That Reduced Our Team’s Repetitive Work by 90%

LINE Home DevOps created an SRE bot to reduce the repetitive work caused by growing services, Flava cloud migration, and increasing developer requests. By making Slack the central interface and automating Jira, Confluence, and workflow updates, the team reduced deployment-request handling from roughly 30 minutes to under one minute. The bot also improved tracking, consistency, and response speed, helping SREs move away from constant firefighting. ## Repetitive SRE Work and Its Costs - Developers frequently asked how to inspect Flava pod logs, request permissions, interpret errors, and access staging environments. - Deployment requests required manual movement between Slack, Confluence, and Jira: - Finding release checklists - Copying information into Jira - Creating missing Fix Versions - Linking Epics and active sprints - Sharing ticket links and deployment documentation - Each deployment request previously took about 30 minutes to an hour. - Manual processing caused omissions and mistakes, especially during urgent releases. - General requests were buried in Slack mentions, making ownership and completion status difficult to track. - Measurement showed that each SRE spent nearly half a day per week on repetitive work. ## Slack-Centered Automation The team adopted the principle that developers should only need Slack, while SREs should be able to manage work with a few clicks. - **Slack as the single source of truth:** Requests begin and remain trackable in Slack. - **Zero manual work:** Rule-based Jira and documentation tasks are automated. - **Immediate visibility:** Status changes and results are posted to Slack in real time. - **Permission control:** Only authorized SRE members can claim or complete requests. ## Key Technical Decisions ### Slack Workflows Instead of Slash Commands - Slash commands are easy to implement but depend on users entering correctly formatted text. - Slack Workflows provide structured forms with required-field validation. - Because Workflows are native Slack functionality, the team avoided building a separate user interface. - The lower usage barrier made adoption more likely. ### Asynchronous Processing - Slack requires event responses within three seconds. - Sequential calls to Jira, Confluence, and other APIs could exceed that limit. - The bot immediately acknowledges the request, then performs external work in the background. - Successes and failures are reported in the Slack thread, keeping processing transparent. ### Redis-Based State Management - In-memory state would be lost whenever the bot restarted. - Slack metadata APIs were considered too slow for real-time interactions such as emoji clicks. - Redis was selected for sub-100-millisecond lookups and persistent state. - A 30-day TTL limits stale data. - Redis transactions using `WATCH/MULTI/EXEC` ensure consistent updates when multiple SREs interact simultaneously. ### Hexagonal Architecture - The bot uses ports and adapters to isolate business logic from external systems. - The architecture separates: - Inbound Slack event adapters - Application use cases and business logic - Outbound Jira, Confluence, and Redis adapters - External API or SDK changes can be handled without modifying core business logic. - This structure also makes testing and future feature development easier. ## Automated Request Scenarios ### Deployment Requests - Developers submit required project, release-version, checklist, and other details through a Slack Workflow. - The bot automatically: - Creates a missing Jira Fix Version - Creates and configures the Jira ticket - Links the Epic - Adds the ticket to the active sprint - Finds the relevant deployment manual - Posts the result to the Slack thread - An SRE can click 👀 to claim the work. - Clicking ✅ completes the Jira ticket and posts a completion notification. - SRE effort falls from about 30 minutes to under one minute, with minimal risk of missing required fields. ### Emergency Deployments - Selecting an urgent request automatically sets Jira Priority to `Highest`. - The bot immediately announces the request in Slack. - An SRE can claim it with 👀, perform the deployment, and complete it with ✅. - The process reduces delays from roughly 30–40 minutes to about one minute. ### General SRE Requests - Requests such as production-access permissions are submitted through a structured Slack Workflow. - The bot creates a Jira ticket, links the Epic, assigns the active sprint, and sets an appropriate priority. - Slack retains the ticket link and status, eliminating the need to search through message history later. - SREs claim and complete the request using the same emoji-based workflow. The main recommendation is to automate repetitive, rule-based operations at the point where requests already occur. A Slack-centered, asynchronous bot with durable state and clean system boundaries can reduce manual effort while making ownership, progress, and completion visible to everyone.

Read original(opens in new tab)

Database DesignSlackJiraRedis+4

aws·Mar 23, 2026·3 min readCurated summary

AWS Weekly Roundup: NVIDIA Nemotron 3 Super on Amazon Bedrock, Nova Forge SDK, Amazon Corretto 26, and more (March 23, 2026) | Amazon Web Services

This week’s AWS roundup highlights major updates across generative AI, data analytics, Java, serverless, logging, and Kubernetes. Notable announcements include NVIDIA Nemotron 3 Super on Amazon Bedrock, the Nova Forge SDK for customizing models, faster Redshift queries, and expanded EKS scaling and availability guarantees. The roundup also points readers to community initiatives, developer resources, and upcoming AWS events. ## Generative AI and Developer Tools - **NVIDIA Nemotron 3 Super** is now available through Amazon Bedrock. - Supports text generation, reasoning, summarization, and code generation. - Can be invoked through Bedrock’s unified API without managing infrastructure. - **Nova Forge SDK** simplifies fine-tuning and customizing Amazon Nova models. - Enables domain-specific adaptations for enterprise use cases. - Handles much of the underlying customization and deployment complexity. - **Kiro for students** provides free access to AI-powered development tools. - **Strands Steering Hooks** reportedly achieved 100% agent accuracy, outperforming prompt engineering and rigid workflows for controlling agent behavior. ## Data, Java, and Serverless Updates - **Amazon Redshift** now delivers up to 7x faster execution for new, uncached queries in dashboards and ETL workloads. - The improvement is especially useful for workloads with high query variability. - **Amazon Corretto 26** is generally available. - Includes current Java features, performance improvements, and security updates. - Supports Amazon Linux, Windows, macOS, and Docker environments. - **AWS Lambda** now exposes Availability Zone metadata for function invocations. - Helps with observability, troubleshooting, latency analysis, and multi-AZ architecture decisions. - **CloudWatch Logs** supports log ingestion through an HTTP-based protocol, reducing the need for custom agents or SDK integrations. ## Amazon EKS Enhancements - Provisioned Control Plane clusters now receive a **99.99% SLA**, compared with 99.95% for the standard control plane. - A new **8XL scaling tier** doubles Kubernetes API server request-processing capacity compared with the 4XL tier. - The larger tier targets demanding workloads such as AI/ML training, HPC, and large-scale data processing. ## AWS Community and Events - **AWS Builder Center badges** recognize contributions, challenges, and community participation. - AWS promotes community-driven learning through the “Keep Building Together” initiative. - Upcoming events include AWS Summits in cities such as Paris, London, Bengaluru, Singapore, Tel Aviv, and Stockholm; AWS Community Days in San Francisco and Romania; and the AWSome Women Summit LATAM in Mexico City. Overall, the announcements emphasize AWS’s continued investment in enterprise AI customization, higher-performance infrastructure, improved observability, and developer communities. Teams should evaluate the new Bedrock, Redshift, Lambda, and EKS capabilities according to their workload scale, reliability, and customization needs.

Read original(opens in new tab)

Database DesignKubernetesGenerative AIAmazon Bedrock+4

cloudflare·Mar 23, 2026·3 min readCurated summary

Launching Cloudflare’s Gen 13 servers- trading cache for cores for 2x edge compute performance

Cloudflare’s Gen 13 servers use AMD EPYC 5th Gen Turin processors to provide up to twice as many cores as Gen 12. However, Turin’s much smaller per-core cache caused the legacy FL1 request-handling layer to suffer severe latency increases, despite higher throughput. Cloudflare found that tuning alone could not fully solve the problem, reinforcing the need for FL2, a Rust-based rewrite designed to scale with cores rather than depend heavily on cache. ## Turin’s Core-Heavy Architecture - Gen 13 Turin processors offer: - Up to 192 cores and 384 SMT threads, compared with Gen 12’s 96 cores. - Improved instructions per cycle through the Zen 5 architecture. - Up to 32% lower power consumption per core. - DDR5-6400 support for greater memory bandwidth. - The tradeoff is substantially less cache: - Gen 12 Genoa-X provides 12 MB of L3 cache per core through 3D V-Cache. - The 192-core Turin 9965 provides only 2 MB per core. - This architecture favors aggregate throughput but challenges workloads dependent on cache locality. ## FL1’s Cache and Latency Problems - FL1, based on NGINX and LuaJIT, was optimized for Gen 12’s large cache. - AMD uProf measurements showed: - Dramatically higher L3 cache miss rates on Turin. - More requests requiring slow DRAM access. - Increasing latency as CPU utilization and cache contention rose. - An L3 hit takes roughly 50 CPU cycles, while a DRAM fetch can take more than 350 cycles. - As a result, Gen 13’s additional cores delivered throughput gains but introduced unacceptable latency penalties. ## Throughput Gains at an Unacceptable Cost - With FL1, Gen 13 produced: - 10% more throughput on the 128-core Turin 9755. - 31% more on the 160-core Turin 9845. - 62% more on the 192-core Turin 9965. - The Turin 9965 offered the strongest total-cost-of-ownership benefits. - However, latency increased by more than 50% at high CPU utilization, which would negatively affect customer experience and violate performance requirements. ## Hardware and Resource Tuning - Cloudflare tested several mitigations with AMD: - Hardware prefetcher and Data Fabric Probe Filter adjustments produced only marginal improvements. - Adding FL1 workers increased throughput but took resources away from other services. - CPU pinning and isolation provided limited benefits. - AMD’s Platform Quality of Service (PQOS) was used to control cache and memory-bandwidth sharing across Turin’s Core Complex Dies. ## Cache Isolation with PQOS - Reserving part of a single CCD’s cache for FL1 produced less than 5% additional throughput. - Configurations assigning FL1 50–75% of each CCD’s cache also delivered less than 5% improvement and caused minor degradation elsewhere. - A socket-level approach was more successful: - Six of twelve CCDs, aligned with a NUMA domain, were dedicated to FL1. - This provided more than 15% incremental throughput while keeping latency acceptable. - These results showed that workload placement and cache locality could help, but they were not a complete substitute for software designed around Turin’s cache profile. Cloudflare’s broader solution was FL2, a Rust-based rewrite of its core request-handling layer. By reducing dependence on large per-core caches, FL2 enabled Gen 13’s higher core count to translate into scalable edge-compute performance without the latency penalties seen with FL1.

Read original(opens in new tab)

Database DesignRustNginxParallel Processing+4

datadog·Mar 23, 2026·3 min readCurated summary

When upserts don't update but still write: Debugging Postgres performance at scale

Datadog needed to track when ephemeral hosts were last seen so inactive hosts could be deleted after seven days. A seemingly inexpensive PostgreSQL upsert caused disk writes to double and WAL syncs to quadruple, despite most operations not changing any data. Investigating the WAL revealed that conflict-handling upserts still lock conflicting rows and generate WAL activity, consuming the database’s limited write capacity. ## Tracking Host Activity Efficiently - Hosts stop reporting telemetry when they terminate, but Datadog has no direct termination signal. - Hosts inactive for seven days can be safely removed from the metadata store. - Updating the main host table on every observation would be too expensive because: - Large data centers generate more than 25,000 observations per second. - PostgreSQL MVCC creates a new row version for every update. - Updating the main table would rewrite all host metadata. - Datadog created a separate `host_last_ingested` table containing: - `host_id` as the primary key - `last_ingested` with a default timestamp - The table used `fillfactor=80` to leave page space for future updates. - No index was created on `last_ingested`, allowing updates to use Heap-Only Tuples (HOT) and avoid additional index writes. - Because only daily freshness was required, the timestamp needed to change at most once per day. ## The Conditional Upsert The initial query inserted a host if it did not exist and otherwise updated `last_ingested` only when the previous value was more than a day old: ```sql INSERT INTO host_last_ingested AS t VALUES ($1, now()) ON CONFLICT (host_id) DO UPDATE SET last_ingested = EXCLUDED.last_ingested WHERE t.last_ingested < EXCLUDED.last_ingested - '1 day'::interval; ``` - New hosts produced an insert. - Recently seen hosts matched the conflict but were expected to be no-ops because of the `WHERE` clause. - The team therefore expected most queries to avoid meaningful writes. ## Unexpected Disk and WAL Activity - During a gradual rollout at roughly 500 upserts per second: - Insertions initially increased as expected. - Actual updates remained mostly flat. - Write IOPS more than doubled. - WAL syncs increased by approximately the same amount. - This showed that the absence of an applied update did not mean the query was free. - Since PostgreSQL must flush WAL records at transaction commit, additional WAL activity directly increased disk pressure. - A PostgreSQL cluster’s single-writer design makes this write budget particularly important. ## Inspecting PostgreSQL WAL - PostgreSQL records database changes in its Write-Ahead Log, including table changes, index modifications, and related transaction activity. - The team used the `pg_walinspect` extension, available starting in PostgreSQL 15: ```sql CREATE EXTENSION pg_walinspect; ``` - Its `pg_get_wal_records_info` function allows inspection of WAL records between two Log Sequence Numbers (LSNs). - Examining the WAL helped explain why the conditional upsert generated writes even when the `WHERE` condition prevented the row update. - The underlying issue was that `ON CONFLICT DO UPDATE` still locks the conflicting row, and that locking activity is recorded in the WAL. The key lesson is that a PostgreSQL upsert that reports zero processed rows is not necessarily a true no-op. Conditional conflict updates can still create substantial WAL and locking overhead, so WAL inspection is essential when database write metrics do not match apparent update volume.

Read original(opens in new tab)

Database DesignPostgreSQLUpsertWal+4

line·Mar 20, 2026·4 min readCurated summary

Internalizing without specifications: Proving equivalence through validation logic

The article describes how LINE Plus safely internalized black-box e-commerce systems without specifications or source code. The team built an automated equivalence-testing loop using Kafka, CDC, OpenSearch, and ksqlDB to compare legacy and new behavior at massive scale. By repeatedly identifying differences, fixing logic, and rechecking results, they could reduce discrepancies toward zero while also measuring performance and protecting production stability. ## Domain: Products, Catalogs, and Data Ingestion - **Products** are individual seller-listed items, potentially with different prices and shipping conditions. - **Catalogs** group products representing the same model or product type and provide derived value such as: - Real-time lowest prices - Unit-price metrics such as price per 100 ml - **Ingestion** receives large product files from sellers, validates and transforms them into internal formats, and updates product and catalog data. - Because the platform contains tens of millions of catalogs and hundreds of millions of products, small logic differences can affect the entire service. ## The Verification Loop - The goal was not merely to find errors, but to help developers understand and correct them quickly. - Inputs had to be identical for both systems, such as: - The same IDs - The same time-based snapshot - The same product files - Outputs were compared according to system type: - API response objects - Database update values - Final registered product data - The general loop consisted of: - **Trigger:** Database changes, developer requests, or file arrivals - **Execution:** Send identical inputs to legacy and new systems - **Comparison:** Apply logic suited to reads, updates, or end-to-end flows - **Processing:** Store detailed differences and produce real-time statistics - **Action:** Developers inspect dashboards or Slack alerts, fix the implementation, and repeat ## Query Logic Verification - The catalog API was difficult to reproduce because it had over 100 response fields, complex filters, undocumented defaults, and unknown sorting behavior. - CDC streamed database binary-log changes into Kafka, allowing verification to begin from many real catalog states. - The verifier made dual API calls and compared legacy and new responses field by field. - Responses were converted into `Map<String, Object>` structures and compared recursively, avoiding the need to model every response class. - If values differed only because list ordering varied, the verifier sorted serialized values and performed a second comparison. - This helped distinguish real implementation defects from harmless ordering differences. - Kafka isolated verification traffic from production services while handling large event volumes. - Difference events were written to Kafka topics and indexed in OpenSearch for detailed investigation. - ksqlDB aggregated streaming discrepancies and sent Slack notifications when abnormal patterns appeared. - Rate limiting restricted repeated errors, such as those from the same field, to a manageable sample per minute. - Because both APIs were called in parallel, the same pipeline also measured and compared their response times. ## Update Logic Verification - The second case involved recalculating catalog statistics whenever product or catalog data changed. - Unlike read verification, this process tested state transitions and asynchronous updates. - When CDC detected a relevant change: - The new statistics logic calculated an expected result. - The verifier compared it with the result actually written by the legacy logic. - Recursive Map-based comparison checked deeply nested statistics fields. - To avoid wasting resources, verification was triggered only for updates related to the catalog-statistics module. ## Handling Asynchronous Lag - Kafka-based processing caused timing gaps: the verifier could read the database before the legacy update had completed. - The team introduced an **N-attempt retry queue**: - Temporarily inconsistent events were requeued. - Only differences that remained after several retries were treated as genuine defects. - The verifier remained a separate process rather than being embedded in the production statistics stream. - This avoided adding load or latency to the existing processing pipeline while preserving independent verification. ## ETL Batch Verification for Missing Triggers - Real-time comparison could detect incorrect results, but not cases where an update should have happened and never occurred. - During refactoring, a complex combination of product and catalog field changes contained a missing trigger condition. - As a result, some statistics remained stale without generating any comparison event. - To detect these silent omissions, the team designed a separate batch-verification process using ETL data alongside the real-time stream checks. The practical recommendation is to treat system internalization as an evidence-building process: define identical inputs and observable outputs, compare legacy and replacement systems continuously, isolate verification through event streams, and supplement real-time checks with batch validation for silent or missing updates.

Read original(opens in new tab)

Database DesignPostgreSQLApache KafkaAPI Design+4

toss·Mar 20, 2026·3 min readCurated summary

Automating Service Vulnerability Analysis using LLM #2

The post explains how Toss Security Research improved AI-driven vulnerability analysis in a research network. Its main challenges were efficiently providing large codebases to an AI and making analysis results consistent and complete. The solution combined a custom code-browsing MCP server with SAST tools used not to identify vulnerabilities directly, but to enumerate all input-to-function paths that the AI must review. ## Efficiently Providing Large Codebases - Tools such as Cursor and Claude Code can search large projects, but primarily rely on pattern matching with tools like ripgrep. - Without prebuilt indexes, they may miss relevant code or waste tokens exploring unnecessary files. - The team built an MCP server that: - Uses **ctags** to index symbol definitions. - Uses **tree-sitter** to parse function boundaries. - Allows AI to access code remotely, similar to IDE features such as “Go to Definition” and “Find References.” ### SourceCode Browse MCP The MCP server provides four main tools: - **`find_references()`** - Searches for symbols or patterns using ripgrep. - Returns file paths, line numbers, snippets, total matches, and whether results were truncated. - **`read_definition()`** - Looks up definitions through the ctags index. - Returns metadata such as file, line, symbol type, language, signature, and scope. - Uses tree-sitter to include the complete function body when requested. - **`read_source()`** - Reads a configurable number of lines before and after a target line. - Lets the AI retrieve only the relevant local context instead of entire files. - **`get_project_structure()`** - Returns the indexed project’s directory structure. - Provides the AI with a project “blueprint,” which is especially important in remote environments where it cannot inspect the repository locally. The MCP workflow is to locate relevant symbols with `find_references()` and `read_definition()`, inspect nearby code with `read_source()`, and use `get_project_structure()` to understand the overall project. ## Improving Consistency and Accuracy - AI analysis produced inconsistent results: for example, it might find all 10 XSS vulnerabilities in one run but only 8 in another. - This variability made the results difficult to trust. - The team combined AI analysis with SAST tooling to ensure complete coverage. ## Using SAST to Enumerate Review Candidates - Rather than passing SAST-detected vulnerabilities directly to the AI, the team used SAST as a candidate-generation tool. - This avoids limiting the AI to vulnerabilities that the SAST engine itself knows how to detect. - SAST extracts every location where untrusted input enters the application and tracks its possible flow to function calls. - Custom Semgrep taint rules identify sources such as: - Spring `@RequestParam` - `@PathVariable` - `@RequestHeader` - Fields read from `@RequestBody` DTOs - `@RequestPart` - `@ModelAttribute` - `@RequestAttribute` - Potential sinks include generic function calls and object method calls. - The AI then reviews every extracted source-to-sink path, combining the completeness of static analysis with the broader reasoning ability of an LLM. The overall approach is to use deterministic indexing and SAST for coverage, while relying on AI for deeper vulnerability interpretation.

Read original(opens in new tab)

Database DesignLarge Language ModelsKotlinAi Security+4

cloudflare·Mar 19, 2026·3 min readCurated summary

Powering the agents: Workers AI now runs large models, starting with Kimi K2.5

Cloudflare is expanding Workers AI beyond smaller models by adding Moonshot AI’s Kimi K2.5, a frontier open-source model designed for agentic workloads. With a 256k context window, tool calling, vision, and structured outputs, Kimi can power an agent’s full lifecycle directly on Cloudflare’s platform. Cloudflare argues that its price-performance makes open-source models essential as personal and enterprise agents dramatically increase inference demand. ## Kimi K2.5’s Price-Performance Advantage - Cloudflare uses Kimi internally for: - Agentic coding through OpenCode - Automated code review via the Bonk public code review agent - Security analysis of Cloudflare codebases - A security-review agent processes more than 7 billion tokens daily and has found over 15 confirmed issues in one codebase. - Compared with a mid-tier proprietary model, switching to Kimi reduced the estimated cost of this workload by 77%, from roughly $2.4 million annually. - As employees increasingly run multiple agents continuously, inference costs become a major barrier to scaling. - Cloudflare positions open-source, frontier-quality models as a more economical alternative to proprietary systems. ## Serving Large Models on Workers AI - Supporting Kimi required upgrades to Workers AI’s inference stack, which historically focused on smaller models. - Cloudflare uses its proprietary Infire inference engine and custom kernels to improve: - Model performance - GPU utilization - Throughput - The platform applies advanced serving strategies such as: - Data, tensor, and expert parallelization - Disaggregated prefill, separating input processing from generation across machines - Workers AI handles these infrastructure optimizations so developers do not need specialized machine learning, DevOps, or reliability engineering expertise. ## Prefix Caching for Agent Workloads - Agents frequently resend large prompts containing: - System instructions - Tool definitions - MCP server tools - Conversation history - Entire codebases - Prefix caching avoids reprocessing unchanged input tokens during multi-turn interactions. - This reduces prefill work, improving: - Time to First Token (TTFT) - Tokens Per Second (TPS) - Overall inference cost - Workers AI now exposes cached tokens as a usage metric and charges less for them than regular input tokens. - Cloudflare has also introduced techniques to improve cache hit rates. ## Session Affinity - Workers AI provides an `x-session-affinity` header to route requests from the same session or agent to the same model instance. - Keeping requests on the same instance increases prefix-cache reuse. - Higher cache hit rates lead to faster responses, greater throughput, and lower costs. - Clients should provide a unique session or agent identifier with the header. Cloudflare’s recommendation is to use Workers AI when building agents that need frontier-level reasoning without the cost and operational burden of proprietary models or self-hosted infrastructure.

Read original(opens in new tab)

Database DesignLarge Language ModelsKubernetesCloudflare+4