Automating KakaoTalk Recommendation Metric Analysis with an AI Agent
The post describes Kakao’s use of an AI agent to automate repetitive analysis of KakaoTalk recommendation metrics on an existing Hadoop environment. Rather than building a new platform or granting the model broad permissions, the team documented existing procedures, data definitions, and decision rules in Markdown-based agent skills and context files. The resulting system helps analysts produce draft reports and explore follow-up questions, while humans remain responsible for validating results and making final decisions. ## Repetitive Analysis Is an Ideal Automation Target - Recommendation analysis often begins with simple questions about CTR changes, experiments, or user-group anomalies. - Answering them typically requires: - Connecting to the analysis environment - Finding the right tables - Writing and executing queries - Interpreting results - Repeating the process across dimensions such as age, category, and time - Much of the effort lies in data preparation and extraction rather than interpretation. - The initial goal was for the AI to follow these steps and produce a first-pass analysis without requiring users to handle queries directly. ## Teaching the Agent to Use Hadoop - The team did not build a new analytics platform or add an MCP integration layer. - Existing Hadoop access scripts were sufficient; the missing component was documentation explaining how to use them. - These procedures were packaged as Agent Skills—Markdown files such as `SKILL.md` describing: - How to connect to Hadoop - How to submit queries - How to retrieve and organize results - The `hadoop-butler` plugin bundled these skills for internal use. - The main lesson was that existing infrastructure can often be extended by converting undocumented operational knowledge into instructions an agent can follow. ## Context Documents Improve Analytical Accuracy - Access to data does not guarantee correct analysis. - Context files such as `CLAUDE.md` or `AGENTS.md` documented: - Relevant tables and clusters - Feature definitions, such as `watch_length` and `valid_view` - User and session aggregation rules - Standard metric definitions - This prevented the agent from repeatedly guessing which tables, columns, or aggregation rules to use. - The documentation also captured institutional knowledge that could help new team members, not only AI systems. - Output quality was determined by the quality and precision of the available context. ## AI Produces Drafts; Analysts Continue the Investigation - Natural-language analysis was most useful for recurring tasks such as: - Detecting anomalies - Comparing experiments - Reviewing weekly performance - The agent’s first report helped identify areas for deeper investigation. - Analysts could then ask follow-up questions and refine the analysis conversationally. - AI-generated reports were treated as reviewable drafts, not final conclusions. - Query logic, selected columns, metric definitions, and interpretations still required human verification. ## Plausible but Incorrect Results The agent’s most dangerous errors were not syntax failures; they were queries that executed successfully but produced misleading results. - **Semantic errors** - To count users, the correct field was `user_id`. - The agent once selected the similarly named `session_user_id`, which represented a session-oriented identifier. - The query ran normally, but the resulting user count was wrong. - **Performance errors** - The agent combined several `COUNT(DISTINCT ...)` expressions in one Hive query. - Although valid SQL, this could force processing through a single reducer and make the query extremely slow. - The better approach was to split the calculations by column and run them in parallel. ## Documentation and Regression Testing - Explicit rules were added to context files and skills, including: - Which identifier to use for user-level aggregation - Wrapping column names in backticks - Splitting multiple `COUNT(DISTINCT)` operations into separate queries - Because natural-language instructions can break other behaviors when modified, the team tested them like software. - An MLflow-based end-to-end evaluation pipeline: - Defines expected behavior for each skill - Runs the agent headlessly with `claude -p` - Uses an LLM judge to evaluate tool-call order, execution traces, and final output - Runs regression scenarios before deployment - This made it possible to catch unintended behavior changes before release. The recommended architecture combines four elements: an AI model, precise domain context, an existing execution environment such as Hadoop, and a verification loop. Organizations should first document their established procedures and analytical definitions, then connect the agent to existing tools and test its behavior systematically.
Read original(opens in new tab)
