PostgreSQL - Kakao | Techlist.io

kakaoAug 6, 20264 min readCurated summary

Experience Building and Operating a Personalized Airflow Testing Environment

Kakao’s data engineering team built AirZone to make Airflow DAG testing faster, easier, and safer across an ecosystem containing thousands of DAGs and multiple Hadoop clusters. Existing approaches required local setup, repeated Git synchronization, file copying, VPN access, or risky testing on production Airflow. AirZone instead creates an isolated, production-like Airflow environment for each pull request, managed through GitHub comments and Kubernetes automation. ## Limitations of Existing Testing Methods - **Local Airflow** - Requires configuring Airflow, Hadoop authentication, connections, and Docker locally. - Has a high initial setup cost and may differ from production. - **Development Airflow** - Requires committing and pushing every code change. - Git submodule updates and DAG parsing introduce long feedback delays. - **Test Airflow with SSH** - Allows files to be copied directly into a container. - Still requires copying files after every edit. - Access to production Hadoop requires connecting to a production VPN. - **Testing on production Airflow** - Heavy test DAGs consume shared scheduler, worker, and node resources. - A resource-intensive test can delay or interrupt unrelated projects. - Per-user isolation is therefore essential. ## AirZone Requirements - Provide an Airflow environment without requiring users to understand Kubernetes or Helm. - Allow code editing through a browser using Jupyter Notebook. - Execute DAGs against Hadoop and authentication mechanisms similar to production. - Create an independent environment for each pull request. - Prevent one user’s tests from affecting other workflows. ## PR-Based, Isolated Architecture - GitHub pull request comments serve as the user interface. - Users can create or delete an environment directly from a PR. - The resulting environment link is posted back to the PR. - Each PR receives a dedicated Kubernetes namespace based on the repository and PR number. - Airflow web server, scheduler, PostgreSQL, Jupyter, DAG volumes, and logs are isolated. - Multiple PRs can be tested simultaneously. - Cleanup is straightforward because the namespace defines the environment boundary. - A dedicated AirZone Helm chart packages the complete test environment. - Production-only components such as PGBouncer and external database connections are omitted where unnecessary. - Airflow, PostgreSQL, DAG storage, Jupyter, authentication, TLS, and logging are deployed together. ## Separating Requests from Deployment - `airzone-api` only validates requests: - Confirms that the PR exists and is open. - Checks branch information. - Prevents duplicate namespaces. - Kubernetes Jobs perform the long-running work: - Install the Helm release. - Run health checks. - Handle creation and deletion independently from the API process. - Job names include the operation and namespace, such as: - `create-airzone-{namespace}` - `delete-airzone-{namespace}` - Failed Jobs can be removed and recreated for retries. - Independent Job logs and status make deployment failures easier to diagnose. - A daily CronJob removes environments that remain after their PRs are closed. ## Building the Airflow Environment Each Helm deployment includes the components needed for a realistic test environment: - **Git integration:** Synchronizes the PR’s head repository and branch. - **DAG PVC:** Lets the scheduler and Jupyter use the same working directory. - **Airflow configuration:** Uses KubernetesExecutor and test-specific DAG scanning, logging, and Hadoop settings. - **Authentication:** Injects user and shared principals, keytabs, Jupyter tokens, and TLS certificates. - **Infrastructure placement:** Selects suitable node groups and a storage class in the same region. - **Centralized logging:** Connects Airflow logs to Elasticsearch and Kibana. - **Hadoop execution:** Existing infrastructure runs Hadoop tasks in dedicated pods using custom Hadoop images, Kerberos initialization, Spark, and Hive. ## Notifications and Security - KakaoWork sends: - An initial notification when a request is received. - A completion notification after deployment. - Operational error alerts. - Sensitive information, including Jupyter and Kubernetes namespace tokens, is not posted in public PR comments. - Tokens are delivered through KakaoWork instead, keeping authentication data separate from the broader PR audience. AirZone’s main recommendation is to make testing a disposable, reproducible environment tied to the pull request itself. By combining per-PR Kubernetes namespaces, Helm-based deployment, asynchronous Jobs, production-like Hadoop access, and automatic cleanup, teams can test DAGs quickly without burdening shared Airflow or production resources.

Read original

PostgreSQL Kubernetes Apache Spark Helm+3

kakaoMar 12, 20264 min readCurated summary

From Student to Developer: Learning Rational Choices Over Right Answers—From DB and Security to AI

The onboarding of 40 new Kakao developers shifted their perspective from making features work to designing systems that survive real-world operations. Across databases, security, and AI, they learned that there is rarely one perfect answer; the best choice depends on scale, risk, maintainability, and business needs. The central lesson was to replace theoretical correctness with responsible, adaptable engineering judgment. ## Database: From Finding the Right Answer to Preparing for Change - Database design must be evaluated by whether it can withstand traffic, schema changes, and operational demands—not only by theoretical correctness. - Foreign keys are not automatically the best choice: - They can introduce locking, performance, and flexibility concerns. - Referential integrity can instead be managed at the application layer, provided testing and correction processes are strong. - Soft deletion, using fields such as `deleted_at`, supports auditability and recovery and is often an essential operational strategy. - Indexes should be selected according to the questions the database must answer: - B-tree, GIN, GiST, SP-GiST, and vector indexes serve different data and query patterns. - Execution plans reveal whether SQL uses indexes or performs full table scans, directly affecting I/O and response times. - Duplication is not always harmful: - Intentional denormalization can avoid expensive joins. - Snapshot data can simplify reads and preserve the information needed by a business workflow. - In MongoDB, embedding selected related data can make screen queries much simpler than relying exclusively on references. - Different database systems embody different trade-offs among performance, consistency, scalability, and operational cost. - The training covered MySQL high availability, PostgreSQL primary-key structures, cloud-native systems such as Neon, and the broader storage-to-analysis pipeline of Hadoop and Spark. - The resulting mindset favors designs that are safe to change and affordable to operate over designs that are theoretically perfect. ## Security and IT: From Someone Else’s Responsibility to a Personal Default - Security became a direct consequence of developers’ code rather than merely a compliance or infrastructure concern. - Everyday safeguards such as development/production separation, VPNs, and antivirus software demonstrate that safety often requires accepting some inconvenience. - DDoS defense is not only about blocking traffic: - It can be difficult to distinguish an attack from legitimate traffic spikes caused by a popular event. - Developers should apply basic controls such as rate limiting and escalate suspicious activity through established response channels. - Hands-on API exploitation made vulnerabilities concrete and encouraged developers to view security through an attacker’s perspective. - Security must be continuous: - AI is increasingly being used both to discover vulnerabilities and to strengthen attacks. - Social-engineering methods involving QR codes, app permissions, and human behavior require more than purely technical defenses. - Security checks should be integrated from the beginning of development, not performed only at the end. - Software quality also depends on people: - Code should remain understandable enough for another developer to take over quickly. - Strong engineering means choosing and communicating the most appropriate solution for the business context, not merely finding a technically possible one. ## AI: From Chatting with Models to Designing Systems - An AI agent is not simply a model; it is an architecture composed of tools, routing logic, error handling, and model calls. - Agent development applies familiar software-engineering practices to probabilistic models. - Because LLM outputs can vary, reliable systems need deliberate controls: - Prompt chaining breaks large tasks into smaller steps and limits context contamination. - Few-shot examples clarify required output formats. - Routing selects different prompts or workflows based on conditions. - Multi-agent systems divide responsibilities among specialized agents, echoing the modularity and scalability principles of microservices. - RAG reduces hallucinations structurally by: - Chunking documents. - Searching for semantically similar vectors. - Supplying retrieved information to the model as additional context. - MCP exposes internal systems and data as callable tools, effectively enabling remote function calling and connecting AI to enterprise capabilities. - Effective AI use shifted from criticizing poor answers to specifying clear objectives, formats, examples, context, and supporting data. - The goal is not merely to receive an intelligent response, but to design a system that consistently produces intelligent behavior. The training ultimately marked a transition from student-style problem solving to professional engineering. Developers should consider operational resilience, security, maintainability, and business value, then make and clearly explain the most reasonable choice for the circumstances.

Read original

PostgreSQL Database Design Large Language Models AI Agents+4

kakaoMar 12, 20264 min readCurated summary

From Student to Developer: Learning Server Flow from Lotto Implementation to Legacy Improvement

The post describes Kakao’s 2026 server-engineering onboarding program, which turns uncertainty into practical understanding through structured implementation, testing, and refactoring. Rather than supplying fixed answers, the program repeatedly asks developers to explain their design decisions and assess what their tests protect. Its central lesson is that server development becomes manageable when engineers build clear reasoning, maintainable structures, and safe change processes. ## Onboarding Through Three Stages - The program follows a progression: 1. TDD- and OOP-based implementation 2. Acceptance testing for legacy code 3. Refactoring legacy code - The focus is not only on what to build, but on how to make engineering decisions. - Core goals include: - Designing maintainable structures - Analyzing and safely improving legacy systems - Collaborating effectively, including responsible AI usage - Although originally designed for server developers, the program expanded to frontend, Android, and iOS engineers because engineering principles apply across technology stacks. ## Learning Through Questions and Collaboration - Participants were repeatedly asked: - Why was this design chosen? - Does this object truly own this responsibility? - What behavior does this test protect? - Daily meetings, pair programming, troubleshooting discussions, and PR reviews made development a collaborative activity. - The program aimed to develop engineers who could explain and defend their designs, rather than merely produce working code. ## Mission 1: Building a Lottery Game with TDD and OOP - The first assignment implemented: - Automatic and manual lottery purchases - A fixed ticket price of 1,000 won - Winning-statistics calculations - Constraints encouraged better design: - One level of indentation - Methods limited to 10 lines - Primitive values wrapped in value objects - First-class collections - Avoiding `else` through early returns - TDD required tests to be written before implementation. ### Making Randomness Testable - Random lottery-number generation initially made tests unpredictable and tightly coupled to concrete implementations. - The solution was to: - Introduce a number-generation interface - Inject the generation strategy - Create a separate test generator - This made test results controllable and encouraged a more flexible design. ### Considering Value Objects and Caching - The team also questioned whether identical number values should always create new objects. - This led to discussions about caching and the difference between object identity and value equality. - The main lesson was to evaluate design decisions, not just make the feature work. ## Mission 2: Writing Acceptance Tests for Legacy Code - Participants first protected the existing system before modifying it. - Tests focused on externally observable behavior: - User actions - System responses - State changes - Strong assertions verified not merely that an operation succeeded, but that it produced the correct result. - Cucumber-based BDD expressed scenarios in a form understandable to non-developers, treating tests as shared specifications. ### Achieving Production Parity - To avoid “works on my machine” problems, the test environment was aligned with production: - PostgreSQL replaced H2 - Docker standardized execution environments - Gradle tasks automated test execution - Test-data isolation used: - Reverse-order foreign-key deletion - `TRUNCATE ... CASCADE` - Shared cleanup utilities - These measures ensured tests started from consistent, independent states. ## Mission 3: Refactoring Legacy Code Safely - The final mission treated refactoring as training in decision-making, not simply an exercise in clean code. - The central rule was to separate structural and behavioral changes: - Structural changes must preserve behavior. - Behavior changes must avoid unrelated structural modifications. - PR reviews helped identify unintended behavior changes and taught participants to predict and control the effects of modifications. - AI was used during refactoring to accelerate broad code changes, but large changes were difficult to verify, highlighting the need to control scope and validate changes carefully. The onboarding’s practical recommendation is to approach server development through small, explainable decisions: write controllable tests, protect legacy behavior before changing it, separate refactoring from feature changes, and use AI as an assistant rather than a substitute for engineering judgment.

Read original

PostgreSQL Docker & Containers Java Oop+4