Database Design | Techlist.io

netflixFeb 12, 20263 min readCurated summary

Automating RDS Postgres to Aurora Postgres Migration

Netflix standardized on Amazon Aurora PostgreSQL after finding that PostgreSQL already supported most relational workloads and that Aurora offered stronger scalability, availability, elasticity, and ecosystem alignment. To migrate nearly 400 RDS PostgreSQL clusters efficiently, Netflix built a self-service workflow that automates replication, traffic quiescence, validation, and cutover while minimizing downtime and eliminating data loss. The Aurora read-replica method is preferred over snapshot migration because it keeps the target nearly synchronized while production continues running. ## Why Netflix Chose Aurora PostgreSQL - PostgreSQL already supported the majority of Netflix’s relational workloads. - Internal evaluations found Aurora PostgreSQL could support more than 95% of workloads running on other relational database systems. - PostgreSQL benefits from: - A broad open-source ecosystem - Strong community adoption - Compatibility with modern data platforms - Aurora’s distributed, cloud-native architecture provides: - Better scalability and elasticity - High availability - Support for globally distributed applications - The migration effort began with RDS PostgreSQL and is intended to expand to other relational systems. ## Database Migration Requires More Than Data Copying A safe migration must move both data and database functionality while preserving correctness, availability, and performance. - **Data replication:** Copy existing data and continuously apply source changes to the destination. - **Quiescence:** Stop writes to the source so the destination can catch up completely. - **Validation:** Confirm that source and destination data are synchronized. - **Cutover:** Redirect applications to the new Aurora database as the system of record. ## Operational and Technical Challenges - Manually migrating almost 400 PostgreSQL clusters would be slow, error-prone, and operationally expensive. - Coordinating downtime across dependent services is difficult. - Netflix therefore created a self-service workflow that handles orchestration, safety checks, and correctness guarantees automatically. - The system must guarantee: - Zero data loss - Extremely short downtime, especially for critical services - No performance degradation during or after migration - Migration of related resources such as parameter groups, read replicas, and replication slots - Application teams control database clients, so the platform cannot depend on them manually pausing writes. - The migration system must provide control-plane mechanisms to halt traffic safely during validation and cutover. - The workflow must operate without obtaining RDS credentials from users, since databases may be tightly secured and the migration platform may lack direct database access. - Because non-experts operate the process, the experience must be self-guided and require minimal user effort. ## Snapshot-Based Migration The snapshot approach is straightforward but requires stopping writes before migration. - Halt write traffic to the RDS PostgreSQL source. - Create a manual snapshot. - Convert the snapshot into an Aurora-compatible format. - Create an Aurora PostgreSQL cluster from the converted snapshot. - Validate the new cluster. - Redirect applications to the Aurora endpoint. This method is simple but can involve a longer interruption because the target is not continuously updated while the snapshot is created and converted. ## Aurora Read-Replica Migration The read-replica approach reduces downtime by continuously replicating the RDS database into Aurora. - Create an Aurora PostgreSQL read replica from the RDS source. - Stream changes asynchronously from RDS to Aurora while applications continue using the source. - Provision and validate Aurora configuration, connectivity, and performance in advance. - When replication lag is sufficiently low, briefly pause writes. - Allow the replica to catch up fully. - Promote it to a standalone Aurora PostgreSQL cluster. - Redirect application traffic to the Aurora endpoint. This approach keeps the destination nearly synchronized before cutover, making it substantially less disruptive than snapshot-based migration. Netflix’s automation focuses on making the read-replica migration process safe, repeatable, and self-service, with the platform handling replication, traffic control, validation, and cutover rather than relying on manual application-team coordination.

Read original

Database Design PostgreSQL AWS Database Migration+4

lineFeb 4, 20264 min readCurated summary

Scaling to Infinity: LY Corporation’s

LY Corporation’s observability team evolved its time-series database to handle rapidly growing infrastructure and Kubernetes workloads. After outgrowing MySQL and OpenTSDB, the team built an engine optimized for high-cardinality metrics, low-latency queries, and seamless API compatibility. Its architecture now combines in-memory, Cassandra, and S3-compatible storage, enabling cost-efficient scaling while supporting trillions of daily metrics. ## Why Time-Series Storage Matters - Metrics record system state as timestamped numerical values. - They support dashboards, threshold-based alerts, and predictive analysis using tools such as ARIMA and Prophet. - Even a small metric record can consume about 280 bytes when timestamps, values, and tags are included. - One CPU metric collected every 15 seconds requires roughly 562 MiB per server annually; across 1,000 servers, this grows to about 548 GiB before adding memory, disk, and network metrics. - High-cardinality cloud environments make both storage cost and query latency critical operational concerns. ## Moving Beyond MySQL and OpenTSDB - MySQL initially became inadequate as the organization moved from SOA to MSA: - Write load increased sharply. - Storage costs and capacity requirements grew. - Query latency worsened for large datasets. - Rigid schemas could not easily represent changing cloud resources. - MySQL sharding provided temporary relief but could not support high-resolution metrics collected at intervals under one minute. - OpenTSDB, introduced in 2016 on Apache HBase, improved write performance but had important limitations: - Tag growth harmed UID-table lookup performance. - Metadata was restricted to a narrow character set. - Large queries required cache warm-up procedures. - These constraints led to the development of an internal database beginning in 2018. ## Building the Internal Time-Series Database - The 2019 engine was designed around: - Flexible protocol support independent of a particular agent. - Linear scalability without downtime. - Low-latency processing of high-resolution metrics. - Strong availability during failures. - Inspired by Meta’s Gorilla research, the team used access patterns in which most queries target recent data. - Frequently accessed metrics were kept in an in-memory database, while colder data was stored in Apache Cassandra. - The new engine enabled metric volumes to grow by more than 200 billion records annually while preserving existing APIs. - Users benefited from the new backend without migration work or code changes. ## Scaling for Kubernetes Workloads - Kubernetes introduced rapidly changing pods, dynamically allocated volumes, and much higher metric churn. - Both major storage layers encountered scaling problems: - IMDB initially required adding identical hardware, limiting expansion options. - Cassandra rebalancing could take tens of hours because of its data volume. - The team improved IMDB with weighted load balancing so nodes with different capacities could be used effectively. - Storage was divided into tiers: - Recent 14-day data remained in Cassandra for high-performance access. - Older data was moved to S3-compatible storage. - This reduced Cassandra dependency, lowered costs, simplified operations, and enabled more flexible hardware and Kubernetes-based deployment. ## Writing and Reading Through S3 - The write path separates data processing from long-term storage: - A Dumper reads metric slots from IMDB. - It converts them into internally defined sub-blocks. - A Block Dumper combines sub-blocks into blocks and writes them to S3. - A Storage Gateway reads the blocks for queries and caches them on local disks. - Disk caching initially caused excessive page-cache use and rapid memory exhaustion. - Direct I/O was considered but withdrawn after the cloud storage team warned that it consumed too much shared bandwidth. - Through cross-team collaboration, the team adopted a B+ tree-based cache that made better use of the kernel page cache without overloading infrastructure. ## Future Direction: From Storage to Intelligence - The team aims to move beyond recording metrics toward prediction and AI-assisted operations. - Achieving this requires consolidating time-series data currently scattered across internal systems. - A key requirement is to perform this integration without imposing migration work or breaking changes on users. - The broader goal is an observability platform that turns unified metrics into predictive and intelligent operational capabilities. The main recommendation is to design time-series platforms around real access patterns, tier storage according to data age, and preserve compatibility while evolving the backend. At extreme scale, careful storage architecture and collaboration across infrastructure teams are as important as raw database performance.

Read original

Database Design AI Kubernetes Observability+4

grammarlyFeb 3, 2026Original article

From Idea to Demo in Two Days: Inside Superhuman’s 2025 Global Hackathon (opens in new tab)

Superhuman’s 2025 hackathon brought together nearly 500 employees to prototype innovative product features by leveraging cutting-edge AI coding tools like Claude Code and Cursor. By integrating AI-driven agents and keyboard-centric workflows, teams demonstrated how rapid experimentation can bridge functional gaps across mail, documentation, and collaboration platforms. The event highlighted a significant shift toward "vibe-coding" and accessible development, where cross-functional teams and non-engineers could ship functional MVPs in just 48 hours. ## Superhuman Command Everywhere (SCE) * This project extends the Superhuman Mail Command Center to the browser, allowing users to trigger Grammarly features, set reminders, and snooze items from any web page. * The tool enables keyboard-only navigation for AI agents; for example, users navigate Grammarly’s Proofreader cards using "J" and "K" and accept or dismiss suggestions with "E" and "D." * Developers used AI tools to quickly interpret an unfamiliar codebase, allowing engineers without frontend expertise to "vibe-code" a working MVP within a few hours. ## Whiteboarding in Coda * This feature introduces a native canvas within Coda documents where users can draw freely, add shapes, and import images for brainstorming and diagramming. * The prototype includes an AI diagramming tool that generates editable visual versions of diagrams based on plain-text descriptions. * Built by a solo team member with no formal coding background, the project utilized Claude Code and Cursor to focus on UX refinement and smooth interactions rather than just technical functionality. ## Superhuman Listening * This system centralizes fragmented customer feedback from tools like Gong, Salesforce, and Zendesk into a single, queryable source of truth. * By linking unstructured data to product roadmaps in Coda, the tool helps sales engineers and product managers determine if specific customer feedback is already being addressed. * Technical challenges included using LLM APIs to extract urgency and sentiment, though the team noted the difficulty of filtering "noise" from high-volume sources like Zendesk tickets. ## Inclusive Language Agent * Developed by a team of linguists, this agent identifies non-inclusive phrasing or unconscious bias in professional writing. * The goal is to provide real-time suggestions that improve workplace culture and customer trust by making word choices more inclusive and intentional. The results of this hackathon suggest that AI-assisted development tools are significantly lowering the barrier to entry for complex product builds. For organizations aiming to accelerate innovation, encouraging "maker" identities across all departments and utilizing AI to bridge technical skill gaps can surface high-value solutions that traditional product cycles might miss.

Database Design Keyboard Shortcuts Ai Tools Grammarly+4

woowahanFeb 2, 20265 min readCurated summary

The First Action That Determines the Success or

Woowa Brothers argues that incident impact depends not only on how quickly an outage is detected, but also on how quickly an effective First Action is executed. Analysis of more than 70 incidents showed that incidents beginning with a hotfix tended to last nearly twice as long as those beginning with a rollback. The company therefore established a standardized incident lifecycle and metrics system to make early response measurable and improve it through automation and process design. ## Why First Action Matters - Detection was generally fast in Woowa Brothers’ 2025 incidents, but customer impact often continued for a long time. - Rollbacks can immediately undo a problematic change, while hotfixes require diagnosis, coding, and redeployment. - During hotfix preparation, the service may remain impaired and customer impact continues to accumulate. - Predefined mechanical mitigations—such as rollback or scaling—are particularly effective because they can be executed without lengthy additional decisions. - First Action is defined by both: - **What** action was taken - **When** it was executed after the incident was recognized ## Standardizing Incident Response - Comparing First Actions across incidents was difficult because teams used different starting points: - The moment customer impact was recognized - The moment an incident was announced company-wide - Other team-specific interpretations - Woowa Brothers concluded that First Action could only be measured consistently if the entire incident timeline used shared definitions. - The lifecycle provides a common framework for understanding where an incident is, what action should happen next, and how long each stage takes. ## The Seven-Stage Incident Lifecycle ### Potential-Incident Lifecycle #### 1. Anomaly - A service or system shows abnormal behavior and the responsible team detects and acknowledges it. - Acknowledgment must leave objective evidence, such as: - An on-call alert acknowledgment - A customer-service response comment - An alert comment - If there is no customer impact, the process may end after internal handling. - If any part of the ordering process becomes unavailable, the event transitions into the Incident Lifecycle. ### Incident Lifecycle #### 2. Open - The service owner recognizes the event as an incident. - An incident-response channel is created and relevant technical and business organizations are invited to coordinate. #### 3. Investigating - Teams assess customer impact and investigate likely causes. - They first examine recent deployments, configuration changes, and failures in external dependencies. #### 4. Identified - Teams execute actions to reduce customer impact. - Rollbacks and scaling adjustments are prioritized as First Actions. - Multiple mitigation options may be evaluated and applied in parallel rather than waiting for a single definitive root cause. #### 5. Monitoring - Teams verify whether the applied actions are actually reducing customer impact. - If not, the process returns to investigation and mitigation for another response attempt. #### 6. Resolved - Customer impact has been eliminated and the incident is considered resolved. - Findings and actions are communicated across the organization according to an established process. #### 7. Closure Time - The team documents the root-cause analysis and tracks preventive follow-up work. - Woowa Brothers separates incident reporting from execution of corrective actions to ensure resolution leads to real operational improvement. ## Metrics for Measuring Response Performance The lifecycle is useful because it connects incident stages to time-based metrics. These metrics are intended to reveal bottlenecks and guide improvement, not serve as goals in themselves. ### MTTD: Mean Time to Detect - Measures the average time from incident occurrence until detection and acknowledgment. - Woowa Brothers includes acknowledgment evidence—similar to MTTA—because an alert alone does not prove that response has begun. - A high MTTD may indicate: - Inadequate monitoring coverage or thresholds - Excessive alert noise - Missing acknowledgment records ### MTTR: Mean Time to Repair - Measures the average time from incident acknowledgment until service recovery. - A high MTTR can indicate: - Poor First Action readiness - Insufficient service visibility - Complex recovery procedures - Slow decisions or communication bottlenecks - It points to structural improvements such as automation, standardized procedures, and clearer decision-making authority. ### MTTA: Mean Time to Action - Measures how quickly mitigation is initiated. - Rather than judging whether a response was “good,” it evaluates whether standardized response mechanisms activate promptly. - Woowa Brothers divides it into two more specific metrics. #### MTTFA: Mean Time to First Action - Measures the time until the first predefined mechanical mitigation, such as a rollback or scaling adjustment. - A high MTTFA may result from: - Complicated rollback paths - Manual scaling operations - Excessive decisions or preparation required before execution - The recommended solution is to automate and simplify First Action procedures. #### MTTEA: Mean Time to Effective Action - Measures the time from incident occurrence until abnormal metrics begin improving after an effective mitigation is applied. - It captures whether an action actually worked, not merely whether it was executed. - The relationship between MTTFA and MTTEA provides useful signals: - **MTTEA ≈ MTTFA:** the initial action was fast and immediately effective. - **MTTEA > MTTFA:** the First Action was insufficient and additional response was needed. - **MTTEA without MTTFA:** no mechanical initial action was possible, or responders relied directly on a hotfix or similar intervention. - **Increasing MTTEA:** standard scenarios or automation require improvement. The practical recommendation is to treat incident response as an organizational system rather than an individual skill. Define a shared lifecycle, prioritize fast and reversible mitigations such as rollback, automate their execution, and use metrics like MTTFA and MTTEA to continuously remove response bottlenecks.

Read original

Database Design Incident Management Rollback Root Cause Analysis+4

gitlabFeb 2, 2026Original article

What’s new in Git 2.53.0? (opens in new tab)

Git 2.53.0 introduces significant performance and maintenance improvements, specifically targeting large repositories and complex history rewriting workflows. Key updates include compatibility between geometric repacking and partial clones, as well as more granular control over commit signatures during imports. These enhancements collectively move Git toward more efficient repository management and better data integrity for modern development environments. ## Geometric Repacking Support with Promisor Remotes * Git utilizes repacking to consolidate loose objects into packfiles, with the "geometric" strategy maintaining a size-based progression to minimize the computational overhead found in "all-into-one" repacks. * Previously, geometric repacking was incompatible with partial clones because it could not correctly identify or manage "promisor" packfiles, which contain the metadata for objects expected to be backfilled from a remote. * The 2.53.0 release enables geometric repacking to process promisor packfiles separately, preserving the promisor marker and preventing the tool from crashing when used within a partial clone repository. * This fix removes a major blocker for making the geometric strategy the default repacking method for all Git repositories. ## Preserving Valid Signatures in git-fast-import(1) * The `git-fast-import` tool, a backend for high-volume data ingestion and history rewriting, previously lacked the nuance to handle commit signatures during partial repository edits. * A new `strip-if-invalid` mode has been added to the `--signed-commits` option to solve the "all-or-nothing" problem where users had to choose between keeping broken signatures or stripping valid ones. * This feature allows Git to automatically detect which signatures remain valid after a rewrite and only strip those that no longer match their modified commits. * This provides a foundation for tools like `git-filter-repo` to preserve the chain of trust for unchanged commits during migration or cleaning operations. ## Expanded Data in git-repo-structure * The `structure` subcommand of `git-repo`, intended as a native alternative to the `git-sizer` utility, now provides deeper insights into repository scaling. * The command now reports the total inflated size and actual disk size of all reachable objects, categorized by type: commits, trees, blobs, and tags. * These metrics are essential for administrators managing massive repositories, as they help identify which object types are driving disk consumption and impacting performance. These updates reflect Git’s continued focus on scalability and developer experience, particularly for organizations managing massive codebases. Users of partial clones and repository migration tools should consider upgrading to 2.53.0 to leverage the improved repacking logic and more sophisticated signature handling.

Database Design Open Source Git Git Fast Import+3

gitlabJan 15, 2026Original article

Announcing general availability for GitLab Duo Agent Platform (opens in new tab)

The GitLab Duo Agent Platform has reached general availability, marking a shift from basic AI code assistance to comprehensive agentic automation across the entire software development lifecycle. By orchestrating intelligent agents to handle complex tasks like security analysis and planning, the platform aims to resolve the "AI paradox" where faster code generation often creates downstream bottlenecks in review and deployment. ### Usage-Based Economy via GitLab Credits * GitLab is introducing "GitLab Credits," a virtual currency used to power the platform’s usage-based AI features. * Premium and Ultimate subscribers receive monthly credits ($12 and $24 respectively) at no additional cost to facilitate immediate adoption. * Organizations can manage a shared pool of credits or opt for on-demand monthly billing, with existing Duo Enterprise contracts eligible for conversion into credits. ### Agentic Chat and Contextual Orchestration * The Duo Agentic Chat provides a unified experience across the GitLab Web UI and various IDEs, including VS Code, JetBrains, Cursor, and Windsurf. * The chat utilizes multi-step reasoning to perform actions autonomously, drawing from the context of issues, merge requests, pipelines, and security findings. * Capabilities extend beyond code generation to include infrastructure-as-code (IaC) creation, pipeline troubleshooting, and explaining vulnerability reachability. ### Specialized Foundational and Custom Agents * **Foundational Agents:** Pre-built specialists designed for specific roles, such as the Planner Agent for breaking down work and the Security Analyst Agent for triaging vulnerabilities. * **Custom Agents:** Developed through a central AI Catalog, these allow teams to build and share agents that adhere to organization-specific engineering standards and guardrails. * **External Agents:** Native integration of third-party AI tools, such as Anthropic’s Claude Code and OpenAI’s Codex CLI, provides access to external LLM capabilities within the governed GitLab environment. ### Automated End-to-End Flows * The platform introduces "Flows," which are multi-step agentic sequences designed to automate repeatable transitions in the development cycle. * The "Issue to Merge Request" flow builds structured code changes directly from defined requirements to jumpstart development. * Specialized CI/CD flows help teams modernize pipeline configurations and automatically analyze and suggest fixes for failed pipeline runs. * The Code Review flow streamlines the feedback loop by providing AI-native analysis of merge request comments and code changes. To maximize the impact of agentic AI, organizations should move beyond basic chat interactions and begin integrating these specialized agents into their broader orchestration workflows to eliminate manual handoffs between planning, coding, and security.

Database Design AI Agents CI/CD Gitlab+4

gitlabJan 14, 2026Original article

Understanding agents: Foundational, custom, and external (opens in new tab)

The GitLab Duo Agent Platform provides a tiered framework for integrating AI into the software development lifecycle through foundational, custom, and external agents. By combining built-in expertise with the ability to define bespoke behaviors or connect to specialized external models, the platform enables teams to automate complex tasks ranging from product planning to runtime debugging. This structured approach ensures that AI assistance is deeply integrated into GitLab’s ecosystem while remaining flexible enough to meet specific organizational standards. ## Foundational Agents These are pre-configured, GitLab-maintained agents available immediately in the IDE or Web UI for general and specialized SDLC tasks. * **GitLab Duo:** The primary general-purpose partner for code modification, merge request management, and issue triaging within the full platform context. * **Planner Agent:** Specifically designed to assist with product management by breaking down epics into structured issues and generating acceptance criteria. * **Security Analyst Agent:** Focuses on triaging vulnerabilities, identifying false positives from scans, and prioritizing risks based on actual impact. * **Data Analyst Agent:** Leverages GitLab Query Language (GLQL) to visualize platform data, such as merge request trends, team workloads, and issue resolution times. ## Custom Agents Organizations can create specialized agents tailored to internal workflows by defining unique system prompts and visibility settings. * **Configuration and Control:** Custom agents are defined by a system prompt that dictates their persona and expertise—such as a DevOps agent that correlates static code data with CI/CD logs. * **Visibility Tiers:** Agents can be set to "Private" for use within a specific project or "Public" to be listed in the AI Catalog for broader organizational discovery. * **Operational Use Cases:** Common implementations include onboarding assistants for company-specific practices, compliance monitors for regulatory requirements, and localized support agents for non-English languages. * **Deployment Best Practices:** It is recommended to start with read-only permissions and highly specific constraints before granting agents write access to the repository or platform. ## External Agents External agents operate asynchronously and are triggered by mentions or assignments within issues and merge requests, rather than through interactive chat. * **Asynchronous Automation:** These agents, such as Anthropic Claude or OpenAI Codex, execute tasks in the background when triggered by commands like `@ai-codex`. * **Managed Credentials:** GitLab handles API key management and rotation for these integrations, simplifying the security overhead for teams using third-party models. * **Specialized Performance:** External agents allow teams to leverage provider-specific strengths, such as Claude’s code analysis or Codex’s task delegation, while maintaining compliance with specific data residency requirements. * **Integrated Review:** A typical workflow involves assigning an external agent as a reviewer on a merge request, where it automatically analyzes code quality and posts improvement suggestions directly as comments. To maximize the value of the platform, teams should begin by leveraging foundational agents for immediate productivity gains before developing custom agents that encode specific organizational knowledge. External agents should be reserved for specialized automation tasks or when specific third-party large language models (LLMs) are required for compliance or advanced code generation.

Database Design Machine Learning CI/CD Gitlab Duo+4

gitlabJan 14, 2026Original article

Getting started with GitLab Duo Agentic Chat (opens in new tab)

GitLab Duo Agentic Chat marks a shift from traditional Q&A chatbots to autonomous AI collaboration partners integrated directly into the software development lifecycle. By leveraging specialized agents and context-aware large language models, the platform enables developers to automate complex tasks like code refactoring, security remediation, and issue triaging. This system serves as a centralized interface across both the GitLab Web UI and IDEs to streamline workflows from initial planning to production deployment. ## Capabilities of Agentic AI * **Autonomous Actions:** The system can move beyond simple chat by creating files, modifying existing code, and opening merge requests on behalf of the user. * **Deep Context Integration:** Agents have access to the full GitLab ecosystem, including issues, epics, Git commits, CI/CD pipelines, and security scans. * **Extensibility:** Through the Model Context Protocol (MCP), the chat can integrate with external services to expand its functional scope. * **Information Retrieval:** Users can query project architecture or use GitLab Query Language (GLQL) to pull specific project analytics and insights. ## Model and Agent Customization * **Flexible Model Selection:** Users and administrators can choose from different LLMs based on task requirements, with configuration available at both the group and individual user levels. * **Specialized Agents:** The platform features dedicated agents for specific roles, such as the **Planner Agent** for product management and the **Security Analyst Agent** for vulnerability management. * **Contextual Switching:** In IDEs, users can switch between agents via a dropdown menu, while the Web UI allows for agent selection when starting new chat sessions. ## Specialized Workflow Use Cases * **Project Planning:** The Planner Agent can break down epics into smaller tasks, list high-priority bugs, and generate technical requirements for new features. * **Security Remediation:** Security-focused agents can explain vulnerabilities in simple terms, identify false positives in scans, and suggest specific code fixes for SQL injection or XSS risks. * **Troubleshooting and Debugging:** The system can analyze CI/CD pipeline logs to identify why a build failed and suggest optimizations for job performance. * **Legacy Modernization:** Specific prompts can guide the AI to refactor code to follow SOLID principles or create migration plans for modernizing legacy languages like COBOL to Java or Python. ## Access and Integration * **Interface Options:** The chat is accessible via a collapsible sidebar in the Web UI and through dedicated plugins in popular IDEs. * **Future Development:** While currently limited to UI and IDE interfaces, a GitLab Duo CLI is in development to bring agentic capabilities to the terminal. To get the most out of GitLab Duo Agentic Chat, it is recommended to transition between specialized agents as you move through different project phases. Using the Security Analyst for code reviews and the Planner for backlog grooming ensures that the underlying models are optimized for the specific metadata and constraints of those tasks.

Database Design AI Agents CI/CD Gitlab+4

gitlabJan 14, 2026Original article

How to customize GitLab Duo Agent Platform (opens in new tab)

The GitLab Duo Agent Platform provides a multi-layered framework for customizing AI behavior to align with specific team workflows and coding standards. By leveraging configuration files at the user, workspace, and project levels, teams can ensure that AI-driven assistance remains context-aware and adheres to internal development policies. This extensibility allows organizations to move from generic AI interactions to highly specialized automation that respects unique architectural patterns and security requirements. ### Levels of Customization GitLab offers a hierarchical approach to tailoring agent behavior, ensuring the right balance between global consistency and project-specific flexibility: * **User-level:** Personal preferences and rules applied across all projects, typically stored in the user’s home directory (e.g., `~/.gitlab/duo/`). * **Workspace-level:** Project-specific configurations located in the repository root that override user-level settings for that specific codebase. * **Project-level:** The creation of entirely custom agents and workflows managed within a specific project to handle complex, specialized tasks. ### Custom Rule Configuration Custom rules provide a mechanism to enforce specific coding styles and instructional sets without repeating prompts in every interaction. * **File implementation:** Rules are defined in `chat-rules.md` files located either in the user's home directory for global application or within the `.gitlab/duo/` directory for project-specific application. * **Functional scope:** They are best used for granular instructions such as forcing the use of the Vue 3 Composition API, requiring JSDoc comments for public functions, or mandating single quotes for strings. * **Governance:** Teams are encouraged to use GitLab Code Owners to manage who can approve changes to these rules, ensuring that AI behavior remains aligned with official team standards. ### Architectural Control with AGENTS.md The platform supports `AGENTS.md`, an industry-standard configuration file used to define broader agent personality, tone, and deep repository context. * **Versatility:** Unlike basic rules, `AGENTS.md` is consumed by both foundational and custom flows and can be understood by external agents like Claude Code. * **Contextual Depth:** These files can be placed in subdirectories to provide specific instructions for different parts of a monorepo, helping the agent understand complex folder structures and internal dependencies. * **Key Parameters:** It typically controls high-level preferences such as security protocols (e.g., "never suggest hardcoding secrets"), documentation requirements, and preferred tool usage. ### Technical Requirements and Deployment Implementing these customizations requires specific environment versions to ensure compatibility across the GitLab ecosystem. * **GitLab Version:** Requires GitLab 18.8 or later. * **IDE Support:** For VS Code users, the GitLab Workflow extension must be version 6.60 or later. * **Update Cycle:** Changes to `AGENTS.md` or custom rules generally require starting a new chat session or triggering a new flow to take effect. To achieve the best results, teams should adopt a "standardize-then-specialize" approach: establish global security and documentation rules at the user level, while using workspace-level `AGENTS.md` files to define the unique architectural patterns and tech stacks of individual projects.

Database Design Machine Learning TypeScript Gitlab Duo+4

gitlabJan 14, 2026Original article

AI Catalog: Discover, create, and share agents and flows (opens in new tab)

The GitLab AI Catalog serves as a centralized repository designed to streamline the discovery, creation, and distribution of AI agents and automated flows across an organization. By providing a structured environment for managing foundational and custom AI assets, it fosters team collaboration and ensures consistency throughout the development lifecycle. Ultimately, the catalog enables developers to scale AI-driven automation from experimental private prototypes to production-ready, instance-wide solutions. ## Discovering and Enabling AI Assets * The catalog acts as a central hub for two distinct asset types: Agents, which handle on-demand or context-specific tasks, and Flows, which are multi-step automations that orchestrate multiple agents. * Users can browse assets via the Explore menu, inspecting titles, descriptions, and visibility statuses before implementation. * To utilize an asset, it must first be added to a top-level group via the "Enable in group" button and then activated within specific projects. * The duplication feature allows teams to copy existing agents or flows to serve as templates for further customization. ## Development and Configuration * Custom agents are built by defining specialized system prompts and configuring specific tool access, such as granting read-only permissions for code and merge requests. * Custom flows utilize a YAML-based structure to define complex behaviors, incorporating components like prompts, routers, and agent hierarchies. * New assets are typically assigned a unique display name (e.g., `ci-cd-optimizer`) and initially set to private visibility to allow for safe experimentation. * Effective creation requires thorough documentation of prerequisites, dependencies, and specific use cases to ensure the asset is maintainable by other team members. ## Managing Visibility and Sharing * Private visibility restricts access to project members with at least a Developer role or top-level group Owners, making it ideal for sensitive or team-specific workflows. * Public visibility allows anyone on the GitLab instance to view and enable the asset in their own projects. * Best practices for sharing include using descriptive, purpose-driven names like `security-code-review` rather than generic identifiers. * Organizations are encouraged to validate and test assets privately before moving them to public status to ensure they solve real problems and handle edge cases. ## Versioning and Lifecycle Management * GitLab employs automated semantic versioning (e.g., 1.1.0) where any change to a prompt or configuration triggers an immutable version update. * The platform uses "version pinning" to ensure stability; when an asset is enabled, projects remain on a fixed version rather than updating automatically. * Updates are strictly opt-in, requiring users to manually review changes and click an "Update" button to adopt the latest version. * Version history and current status can be monitored through the "About" section in the Automate menu for both agents and flows. To maximize the benefits of the AI Catalog, organizations should establish a clear transition path from private experimentation to public sharing. By leveraging version pinning and granular tool access, teams can safely integrate powerful AI automations into their development workflows while maintaining full control over environment stability and security.

Database Design Gitlab Agents Flows+4

datadogJan 7, 20263 min readCurated summary

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection

eBPF gives security tools broad, efficient, and relatively safe access to Linux kernel activity, making it well suited for runtime threat detection. Datadog chose it for Workload Protection after comparing kernel modules, tracing interfaces, ptrace, seccomp, Linux Audit, and other approaches. However, five years of production use across diverse kernels showed that eBPF’s safety and performance benefits are not automatic; reliability, compatibility, observability, and operational discipline are essential at scale. ## Why Runtime Workload Protection Is Needed - Static analysis and vulnerability scanning cannot catch every threat. - Zero-days and vulnerable third-party dependencies can remain active while patches are being prepared or deployed. - Workload Protection is intended to: - Monitor known-vulnerable workloads until they can be patched. - Continuously observe all workloads. - Detect and help mitigate previously unknown vulnerabilities during incident response. ## Alternatives Evaluated Datadog evaluated a broad range of Linux monitoring and instrumentation mechanisms: - **Linux kernel modules** - Offer deep access and can hook or replace almost any kernel function. - Are invasive and often considered too risky for production infrastructure. - **Traditional tracing interfaces** - Include inotify, fanotify, kprobes, tracepoints, and perf events. - Provide useful visibility but generally need to be combined for comprehensive coverage. - **ptrace and seccomp-bpf** - Can provide detailed user-space process visibility. - Are less suitable as a unified solution for monitoring the whole system. - **Linux Audit** - Produces configurable streams for process execution, file access, and network activity. - Is widely used by security tooling but has its own performance and operational tradeoffs. - **Other mechanisms** - Netlink, LD_PRELOAD, and binfmt_misc were also considered. - Each involves compromises in reliability, visibility, or system impact. ## Why eBPF Stood Out - **Safety checks** - The kernel statically verifies eBPF bytecode before loading it. - Verification detects issues such as infinite loops and unsafe memory access. - This is safer than deploying custom kernel modules, though eBPF can still cause harm or performance problems. - **Performance** - eBPF generally has lower overhead than approaches such as Linux Audit or ptrace. - Actual impact depends heavily on implementation and workload. - **Unified visibility** - A single mechanism can observe process, filesystem, and network activity. - This avoids assembling multiple specialized tracing systems. - **Container and namespace coverage** - eBPF provides consistent visibility across namespaces, cgroups, and containers. - CO-RE (Compile Once–Run Everywhere) improves portability across Linux distributions and kernel versions. - **Enforcement capabilities** - BPF LSM programs support mandatory access controls. - This gives eBPF enforcement power beyond ordinary tracing mechanisms, which is important for runtime security. ## Lessons from Operating eBPF at Scale After five years of operating an agent that hooks process scheduling, filesystem, and networking internals, Datadog emphasizes that production eBPF is more complicated than its reputation suggests. The six areas of operational experience are: - Ensuring programs load, attach, and continue firing across kernel versions. - Capturing and enriching event data accurately. - Monitoring and auditing eBPF usage to reduce the attack surface. - Coexisting with other eBPF-based tools on the same host. - Measuring and controlling performance overhead. - Shipping changes safely through disciplined rollout practices. The practical recommendation is to treat eBPF as powerful infrastructure rather than a maintenance-free kernel feature: validate behavior across kernels and workloads, monitor its own operation, measure overhead continuously, and use cautious deployment practices.

Read original

Database Design Observability Ebpf Linux Kernel+4

tossDec 10, 2025Original article

Legacy Settlement Modernization: From the (opens in new tab)

Toss Payments recently overhauled its 20-year-old legacy settlement system to overcome deep-seated technical debt and prepare for massive transaction growth. By shifting from monolithic SQL queries and aggregated data to a granular, object-oriented architecture, the team significantly improved system maintainability, traceability, and batch processing performance. The transition focused on breaking down complex dependencies and ensuring that every transaction is verifiable and reproducible. ### Replacing Monolithic SQL with Object-Oriented Logic * The legacy system relied on a "giant common query" filled with nested `DECODE`, `CASE WHEN`, and complex joins, making it nearly impossible to identify the impact of small changes. * The team applied a "Divide and Conquer" strategy, splitting the massive query into distinct domains and refined sub-functions. * Business logic was moved from the database layer into Kotlin-based objects (e.g., `SettlementFeeCalculator`), making business rules explicit and easier to test. * This modular approach allowed for "Incremental Migration," where specific features (like exchange rate conversions) could be upgraded to the new system independently. ### Improving Traceability through Granular Data Modeling * The old system stored data in an aggregated state (Sum), which prevented developers from tracing errors back to specific transactions or reusing data for different reporting needs. * The new architecture manages data at the minimum transaction unit (1:1), ensuring that every settlement result corresponds to a specific transaction. * "Setting Snapshots" were introduced to store the exact contract conditions (fee rates, VAT status) at the time of calculation, allowing the system to reconstruct the context of past settlements. * A state-based processing model was implemented to enable selective retries for failed transactions, significantly reducing recovery time compared to the previous "all-or-nothing" transaction approach. ### Optimizing High-Resolution Data and Query Performance * Managing data at the transaction level led to an explosion in data volume, necessitating specialized database strategies. * The team implemented date-based Range Partitioning and composite indexing on settlement dates to maintain high query speeds despite the increased scale. * To balance write performance and read needs, they created "Query-specific tables" that offload the processing burden from the main batch system. * Complex administrative queries were delegated to a separate high-performance data serving platform, maintaining a clean separation between core settlement logic and flexible data analysis. ### Resolving Batch Performance and I/O Bottlenecks * The legacy batch system struggled with long processing times that scaled poorly with transaction growth due to heavy I/O and single-threaded processing. * I/O was minimized by caching merchant contract information in memory at the start of a batch step, eliminating millions of redundant database lookups. * The team optimized the `ItemProcessor` in Spring Batch by implementing bulk lookups (using a Wrapper structure) to handle multiple records at once rather than querying the database for every individual item. This modernization demonstrates that scaling a financial system requires moving beyond "convenient" aggregations toward a granular, state-driven architecture. By decoupling business logic from the database and prioritizing data traceability, Toss Payments has built a foundation capable of handling the next generation of transaction volumes.

Database Design Kotlin Refactoring Query Optimization+3

tossNov 30, 2025Original article

From Legacy Payment Ledger to Scalable System (opens in new tab)

Toss Payments successfully modernized a 20-year-old legacy payment ledger by transitioning to a decoupled, MySQL-based architecture designed for high scalability and consistency. By implementing strategies like INSERT-only immutability and event-driven domain isolation, they overcame structural limitations such as the inability to handle split payments. Ultimately, the project demonstrates that robust system design must be paired with resilient operational recovery mechanisms to manage the complexities of large-scale financial migrations. ### Legacy Ledger Challenges * **Inconsistent Schemas:** Different payment methods used entirely different table structures; for instance, a table named `REFUND` unexpectedly contained only account transfer data rather than all refund types. * **Domain Coupling:** Multiple domains (settlement, accounting, and payments) shared the same tables and columns, meaning a single schema change required impact analysis across several teams. * **Structural Limits:** A rigid 1:1 relationship between a payment and its method prevented the implementation of modern features like split payments or "Dutch pay" models. ### New Ledger Architecture * **Data Immutability:** The system shifted from updating existing rows to an **INSERT-only** principle, ensuring a reliable audit trail and preventing database deadlocks. * **Event-Driven Decoupling:** Instead of direct database access, the system uses Kafka to publish payment events, allowing independent domains to consume data without tight coupling. * **Payment-Approval Separation:** By separating the "Payment" (the transaction intent) from the "Approval" (the specific financial method), the system now supports multiple payment methods per transaction. ### Safe Migration and Data Integrity * **Asynchronous Mirroring:** To maintain zero downtime, data was initially written to the legacy system and then asynchronously loaded into the new MySQL ledger. * **Resource Tuning:** Developers used dedicated migration servers within the same AWS Availability Zone to minimize latency and implemented **Bulk Inserts** to handle hundreds of millions of rows efficiently. * **Verification Batches:** A separate batch process ran every five minutes against a Read-Only (RO) database to identify and correct any data gaps caused by asynchronous processing failures. ### Operational Resilience and Incident Response * **Query Optimization:** During a load spike, the MySQL optimizer chose "Full Scans" over indexes; the team resolved this by implementing SQL hints and utilizing a 5-version Docker image history for rapid rollbacks. * **Network Cancellation:** To handle timeouts between Toss and external card issuers, the system uses specific logic to automatically send cancellation requests and synchronize states. * **Timeout Standardization:** Discrepancies between microservices were resolved by calculating the maximum processing time of approval servers and aligning all upstream timeout settings to prevent merchant response mismatches. * **Reliable Event Delivery:** While using the **Outbox pattern** for events, the team added log-based recovery (Elasticsearch and local disk) and idempotency keys in event headers to handle both missing and duplicate messages. For organizations tackling significant technical debt, this transition highlights that initial design is only half the battle. True system reliability comes from building "self-healing" structures—such as automated correction batches and standardized timeout chains—that can survive the unpredictable nature of live production environments.

Database Design Apache Kafka Microservices Mysql+4

datadogNov 18, 20253 min readCurated summary

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute

File integrity monitoring must provide more than proof that a file changed: security teams need to know how, why, and by whom it changed. Datadog found that filesystem scans, inotify, and auditd could not provide sufficient context, reliability, or scalability. An eBPF-based approach delivered kernel-level visibility into processes and containers, but required extensive filtering and edge processing to handle more than 10 billion events per minute. ## Why Traditional Monitoring Falls Short - Periodic scans can miss changes that are made and reverted between scans. - Scans show that a file changed, but not the process, container, or mechanism responsible. - `inotify` lacks the system-level context needed to correlate file events with processes and containers. - `auditd` offers richer information but can impose significant performance overhead and struggle under heavy load. ## eBPF for Context-Rich File Monitoring - eBPF observes file activity directly in the Linux kernel in real time. - Events can include: - The modified file - The process that triggered the change - The container in which the process ran - Additional security-relevant metadata - This context makes events more useful for investigations than simple “file changed” notifications. ## Scaling at the Agent and Backend - Datadog observed more than 10 billion file-related events per minute across its infrastructure. - Each serialized event was approximately 5 KB, making unrestricted transmission infeasible—potentially several terabytes per second. - Sending every event would also overload Agents through excessive CPU, memory, serialization, and network usage. - Agent-side rules filter events locally, discarding noise before transmission. - This reduced the stream to roughly one million events per minute while preserving detection coverage. ## Filtering Events in the Kernel - A basic architecture loads eBPF programs into the Agent, observes system activity, writes events to a ring buffer, and evaluates them in user space. - Sensitive workloads can generate up to 5,000 relevant syscalls per second. - Initial implementations risked ring-buffer backlogs and dropped events, creating security blind spots. - Datadog moved as much evaluation as possible into eBPF programs to reduce the number of events reaching user space. - The Agent could then perform a deeper second-stage evaluation before forwarding events to the backend. ## Two-Stage Evaluation: Approvers and Discarders - eBPF’s safety constraints limit computation, especially on older Linux kernels. - The system therefore separates evaluation into: - **In-kernel filtering:** Lightweight decisions that quickly approve or discard events. - **User-space evaluation:** More complex analysis using richer context, correlations, and logic unsuitable for the kernel. - This design balances kernel safety and performance with the need for detailed security detection. Datadog’s approach shows that scalable FIM requires combining eBPF’s deep visibility with aggressive filtering at the edge and in the kernel. The practical recommendation is to keep expensive analysis in user space while rejecting irrelevant events as early as possible.

Read original

Database Design Linux Ebpf Security Monitoring+4

datadogNov 4, 20253 min readCurated summary

Replication redefined: How we built a low-latency, multi-tenant data replication platform

Datadog built a managed, multi-tenant data replication platform to move data reliably across thousands of services without brittle, point-to-point integrations. The effort began by separating analytical search workloads from a shared PostgreSQL database, then evolved into automated pipeline provisioning with Temporal. The platform favors asynchronous replication to improve scalability and resilience, accepting limited replication lag in exchange for lower application latency and reduced operational coupling. ## Scaling Search Beyond PostgreSQL - A shared PostgreSQL database initially provided low-latency access, ACID guarantees, and low operational cost. - As data volumes grew, complex joins and aggregations became increasingly slow. - Datadog’s Metrics Summary page had to join: - 82,000 active metrics - 817,000 metric configurations - Page latency reached approximately 7 seconds at p90, while repeated facet changes generated additional expensive queries. - Index and disk bloat, memory pressure, VACUUM and ANALYZE overhead, and rising I/O wait further reduced throughput. - Rather than continuing to optimize PostgreSQL for analytical search, Datadog moved search and aggregation workloads to a dedicated search platform. - Data was denormalized during replication, producing document-oriented indexes better suited to faceted search. - The resulting system reduced page-load times by as much as 97%—from roughly 30 seconds to 1 second—while maintaining about 500 ms of replication lag. ## Automating Pipeline Provisioning with Temporal Provisioning a replication pipeline required coordinating multiple systems and configuration steps: - Enabling PostgreSQL logical replication with `wal_level`. - Creating users and assigning replication permissions. - Configuring publishers and replication slots. - Deploying Debezium instances to capture PostgreSQL changes. - Creating Kafka topics and mapping them to Debezium instances. - Adding heartbeat tables to monitor replication and prevent excessive WAL retention. - Configuring sink connectors to write Kafka data into the search platform. Manual management became increasingly difficult across many pipelines and data centers. Datadog used Temporal workflows to split provisioning into modular, repeatable tasks and combine them into higher-level orchestrations. This reduced errors, improved consistency, and allowed engineers to create and modify pipelines without repeating complex operational procedures. ## Choosing Asynchronous Replication - Synchronous replication provides strong consistency by waiting for replicas to acknowledge each write. - However, it increases latency and operational complexity, particularly across distributed environments. - Asynchronous replication allows the primary system to acknowledge writes immediately while replicas catch up afterward. - Datadog selected the asynchronous model because it decouples application performance from network latency and replica availability. - The trade-off is temporary replication lag during failures or periods of pressure, but the model offers better scalability and resilience for high-throughput systems. Datadog’s experience suggests that replication should be treated as a managed platform rather than a collection of custom integrations. Separating workloads, automating provisioning, and choosing asynchronous delivery can improve performance and reliability while reducing the operational burden on individual engineering teams.

Read original

Database Design PostgreSQL Apache Kafka Elasticsearch+4