LINE/Database Design

19 posts

line4 min readCurated summary

ID-JAG, a next-generation standard candidate for solving authentication challenges in the AI era

ID-JAG extends enterprise SSO trust to API access between AI agents, applications, and services. It uses an enterprise IdP to centrally evaluate permissions and issue a signed JWT that can be exchanged for a resource-specific access token. This can reduce consent prompts, improve auditing, and limit token sprawl, but organizations should adopt it cautiously while the specification remains an Internet-Draft. ## The Authentication Challenge in the AI Era - AI agents increasingly perform real work, including: - Searching systems - Querying databases - Sending messages - Creating tickets - As the number of connected services grows, authentication and authorization become more complex. - Poorly coordinated integrations can turn AI from a productivity tool into an operational bottleneck. - ID-JAG is being discussed by the IETF OAuth Working Group as a potential solution. ## What ID-JAG Is - ID-JAG, or Identity Assertion JWT Authorization Grant, extends the enterprise IdP’s SSO trust relationship to API access. - The IdP centrally determines: - Which application or agent may access an API - Which user or identity it acts for - Which scopes or permissions are allowed - It combines: - OAuth 2.0 Token Exchange (RFC 8693) - JWT Profile for OAuth 2.0 Authorization Grants (RFC 7523) - The IdP issues a cryptographically verifiable JWT as an “introduction” or authorization assertion. - The target authorization server validates that assertion and issues the final access token. ## The ID-JAG Participants and Flow The model involves four main parties: - **Requesting Agent:** An AI agent or application calling another service’s API - **Enterprise IdP:** Provides SSO and enforces centralized organizational policies - **Authorization Server:** Issues tokens for the target application - **Resource Server:** Hosts the API being accessed The basic five-step flow is: 1. The user signs in to the requesting agent, which obtains an ID token from the IdP. 2. The agent presents the ID token to the IdP and requests an ID-JAG through token exchange. 3. The IdP evaluates organizational policy and issues the ID-JAG if access is allowed. 4. The agent presents the ID-JAG to the target authorization server and receives an access token. 5. The agent uses the access token to call the resource server. The key architectural shift is that authorization decisions move from isolated agent-to-service relationships toward a centrally governed relationship between the enterprise IdP and target authorization servers. ## Benefits for User Experience and Auditing - Centralized IdP policies can reduce repeated consent screens. - This is especially useful when AI agents connect to many tools and services. - ID-JAG claims can record important context, such as: - The user whose authority is being delegated (`sub`) - The requesting agent (`client_id`) - The target authorization server (`aud`) - Approved scopes (`scp`) - Issuer, issue time, expiration, and unique token ID - Centralized issuance logs provide a clearer view of service-to-service relationships. - Security teams can more easily determine which agent accessed which service, on whose behalf, and with what permissions. - The same records can support incident investigation, compliance audits, and accountability. ## Centralized Control and Reduced Token Sprawl - The IdP can help detect and control unauthorized “shadow AI” integrations. - It can evaluate every token exchange using consistent organizational policies. - Requested scopes can be narrowed or overridden according to enterprise security requirements. - Blocking future access can be handled centrally instead of by changing policies across every endpoint. - ID-JAG may reduce token sprawl by avoiding additional long-lived refresh tokens. - The draft recommends that resource authorization servers generally not issue refresh tokens when an ID-JAG is exchanged. - Agents can instead submit a new ID-JAG to obtain another access token, replacing scattered API keys, service credentials, and refresh tokens with dynamic, policy-based trust. ## Adoption Requirements and Risks - ID-JAG is still an IETF Internet-Draft, not a finalized RFC. - Its behavior may change, so systems should avoid tightly coupling their core architecture to the current draft. - Before implementation, organizations need to verify that: - The requesting agent is registered as an OAuth client with both the enterprise IdP and the target authorization server. - Explicit trust relationships exist between the IdP and agent, and between the IdP and authorization server. - The IdP has pre-authorized the agent to act on users’ behalf for the relevant services and scopes. - Deployment also requires coordinated support from agents, enterprise IdPs, authorization servers, and resource servers. Organizations should treat ID-JAG as a promising architectural direction for governing AI-agent access, while isolating its implementation behind adaptable interfaces until the standard stabilizes. Pilot deployments should focus on centralized policy enforcement, detailed audit logging, strict scope control, and minimizing long-lived credentials.

Read original(opens in new tab)
line3 min readCurated summary

ODW #3: Boosting Development Efficiency by Safely Utilizing MCP Servers

LY Corporation is expanding AI use across its engineering organization through MCP servers, which connect AI assistants with internal and external tools through a common protocol. The company combines this flexibility with allowlists, automated security checks, and internal standards to reduce risk. Its Orchestration Development Workshop demonstrates practical applications such as Jira ticket automation and multi-agent code reviews, while emphasizing shared learning and experimentation as AI practices evolve. ## MCP Servers and Their Benefits - MCP servers act as translators between AI assistants and external systems. - Before MCP, each assistant required a separate integration for every tool. - With MCP, a tool can implement one standardized interface and work with multiple compatible assistants. - This improves interoperability, scalability, and the ability to combine different AI tools. ## Security Risks and LY Corporation’s Controls - A 2025 Astrix Security report found that: - More than 5,200 public MCP servers were analyzed. - 53% relied on long-lived static API keys or personal access tokens. - Only 8.5% used newer authentication methods such as OAuth. - LY Corporation manages externally developed MCP servers through: - An allowlist permitting only approved servers. - Automated security verification based on internal standards. - Internal MCP servers for groupware and business systems are built to meet the company’s security requirements. - Centralized infrastructure lets teams focus on applying AI rather than independently rebuilding integrations and controls. ## Workshop Applications The Orchestration Development Workshop taught participants how to understand, configure, and safely apply MCP servers with AI assistants. - Topics included MCP fundamentals, security risks, internal policies, development rules, and configuration in Claude and Cline. - The internal plugin marketplace was introduced as a way to distribute MCP configurations. - Participants practiced using Claude Code with the internal groupware MCP server to: - Generate a Jira ticket title and summary. - Create the ticket automatically. - The exercise showed how AI can remove repetitive administrative work and free time for higher-value tasks. ## Multi-Agent Code Review Demonstration - A demonstration combined Claude Code, Codex CLI, Context7 MCP, and Codex MCP. - A Sonnet-based agent first analyzed a pull request, including: - Technical stack and relevant documentation. - Code changes and repository context. - Security, performance, and code-quality concerns. - GPT-5 then validated the initial review, identifying missed issues and checking the prioritization of findings. - Using different models provided more varied and potentially objective perspectives on the same code. ## Results and Organizational Learning - Around 1,600 people attended the workshop in real time. - 31.5% had already applied related techniques before the event. - Another 55.7% planned to try them soon. - LY also created “Help LY MCP,” a GPTs-based tool that explains internal MCP rules and helps teams assess whether proposed uses are suitable, including for global subsidiaries. - The workshop’s broader purpose was to create a shared understanding of: - What AI and MCP can currently do. - What risks and pitfalls exist. - How to use the technology meaningfully. ## Continuing to Experiment The article concludes that rapidly changing AI technology makes shared experimentation more valuable than simply announcing new tools. MCP may eventually be surpassed by other approaches, such as skills, so teams should continually reassess the best solution. LY recommends creating a culture where employees can safely try small ideas, learn together, and adapt as new practices emerge.

Read original(opens in new tab)
line4 min readCurated summary

Clearing Review Bottlenecks with AI - Transforming Review Culture with PR Review Support and Internal Workshops

Orchestration Guild member Fukuyama describes how Yahoo! Places addressed PR review bottlenecks by combining AI assistance with standardized processes and team culture. Reviews had become concentrated among a few engineers, creating delays and forcing a trade-off between speed and quality. The team introduced Claude Code–based screening reviews, then expanded the approach into a broader system for improving PR creation, review accuracy, and continuous improvement. ## PR Review Bottlenecks - In late 2024, review responsibilities were concentrated on the tech lead and one other engineer. - Reviewers were simultaneously implementing features and reviewing code, causing PR queues to grow. - The main problems were: - Authors could not move to their next tasks while waiting for reviews. - Review work consumed most of the day. - Large PRs had to be reviewed quickly, increasing the risk of missed bugs. - This created a single point of failure and exposed the trade-off between thoroughness and development speed. - The launch of a dedicated frontend team in early 2025 provided an opportunity to redesign the review process. ## Introducing AI Screening Reviews - The team first tried having AI summarize PR changes before review. - Although summaries made changes easier to understand, AI did not sufficiently reduce the work of tracing dependencies or identifying hidden problems. - Manually pasting prompts for every review also made the approach inconvenient, so it was abandoned after about two weeks. - The introduction of Claude Code in summer 2025 changed the situation because reusable custom commands eliminated repetitive prompt preparation. - AI screening reviews now perform an initial inspection before a human reviewer makes the final judgment. - This changes the process from “humans inspect everything” to a two-stage model: - AI analyzes the PR, its impact, coding conventions, and possible risks. - A human reviewer validates the analysis and makes the final decision. ## Claude Code Custom Review Commands The custom command requests that Claude Code: - Summarize the PR and its affected areas. - Explain the before-and-after changes for each file. - Check coding and naming conventions. - Investigate dependent files and broader codebase impact. - Identify potential bugs, security issues, performance problems, code smells, and unintended side effects. - Suggest concise, respectful review comments for the author. - Classify comments with labels such as `[must]`, `[want]`, `[imo]`, `[ask]`, `[nits]`, and `[info]`. - Determine whether additional tests are needed based on existing project practices. The command uses GitHub CLI operations such as: - `gh pr view --json title,body,files,url` - `gh pr diff` - `gh pr view --comments` - GitHub API calls for line-level comments - `gh pr checkout` when the relevant branch is not currently checked out The review procedure is deliberately structured: 1. Confirm the review requirements. 2. Understand the PR’s overall purpose and background. 3. Review each changed file in detail. 4. Investigate dependencies across the codebase. 5. Produce a final assessment and suggested comments. The same screening process can help both reviewers and PR authors. Reviewers use it to reduce preparation time and understand impact, while authors can run it before requesting review to fix likely issues in advance. ## Expanding Beyond AI Screening After seeing benefits from screening reviews, the team created a broader improvement framework spanning technology and team culture. It was organized around four connected goals: - Improving efficiency. - Establishing a foundation for review accuracy. - Building review-oriented team culture. - Creating a mechanism for continuous improvement. The approach treats review optimization as an ongoing cycle rather than a one-time tool deployment. ## Automating PR Creation The team also uses AI to reduce the effort required to create PRs. - Git operations such as branch creation, commits, and PR creation are automated. - AI analyzes the commit diff to generate: - A PR title. - A summary of the changes. - Background and motivation. - Other required PR template fields. - Standardized and more complete PR descriptions provide better context for both human reviewers and AI screening. - Improving PR quality at the creation stage also increases the accuracy and consistency of later reviews. ## Practical Recommendation AI should support—not replace—reviewer judgment. Teams should begin by standardizing the review workflow, encode that workflow in reusable AI commands, and measure whether review time, PR waiting time, and review quality improve. Combining AI screening with better PR context, dependency analysis, clear comment conventions, and continuous process refinement offers a more sustainable solution than relying on individual reviewers.

Read original(opens in new tab)
line3 min readCurated summary

The Key to AI Utilization Lies in 'Organizational Learning' - The Start of the Orchestration Development Workshop

LY Corporation is moving from simply adopting AI tools to building with AI as a collaborative development partner. Its new Orchestration Development Workshop teaches engineers to coordinate multiple AI systems across coding, testing, reviews, incident analysis, and other workflows. The initiative aims not only to improve efficiency but to free engineers from repetitive work so they can focus on more creative, high-value challenges. ## From AI Adoption to AI Collaboration - AI-assisted development and operations are spreading rapidly across LY Corporation. - Engineers use generative AI for code generation and testing, while combining it with non-generative AI for analysis and operational optimization. - Despite broader adoption, employees differ significantly in how deeply they use AI in their daily work. - The workshop was created to help the organization evolve from “using AI” to “creating alongside AI.” ## Orchestration: Coordinating Multiple AI Systems - “Orchestration” refers to combining multiple AIs, along with human input, to produce a complete outcome. - Example workflows include: - Generating code automatically from a Jira ticket. - Having AI run tests, conduct reviews, and create a pull request. - Analyzing a Slack incident report, estimating the cause, and proposing a fix. - The workshop turns these emerging practices into hands-on learning rather than passive demonstrations. ## A Hands-On, Interactive Learning Model - Participants follow instructors in real time and perform the same tasks themselves. - Zoom conversations and Slack questions create two-way communication during the session. - Instructors and representative participants explore solutions to problems as they arise. - The goal is for attendees to gain skills they can reproduce in their own projects, not merely acquire theoretical knowledge. ## Organization-Wide Support Through Guilds and DevRel - The initiative is designed to avoid depending on individual enthusiasm. - Three complementary functions support continuous growth: - **DevRel:** Drives the program and promotes adoption. - **Guilds:** Contribute practical insights from engineering teams. - **TD:** Helps maintain quality and reproducibility. - This structure supports consistent content quality and enables AI knowledge to spread across the company. ## Beyond Efficiency: Unlocking Engineering Creativity - LY Corporation views AI as more than a way to complete tasks faster. - By delegating repetitive work to AI, engineers can spend more time on creative and strategically valuable activities. - The organization aims to move beyond a model where AI writes code and humans only review it. - Instead, engineers should collaborate with AI from the design stage through implementation. ## Future Direction - LY Corporation plans to share lessons from the workshops through external channels such as its technology blog. - Future topics will include both generative and non-generative AI. - The broader goal is to provide practical guidance for engineers building new workflows with AI. The workshop represents a structured way to turn AI experimentation into repeatable organizational practice, helping engineers coordinate multiple AI tools while preserving human creativity and judgment.

Read original(opens in new tab)
line4 min readCurated summary

From Hive to Iceberg: The Secret to 12x Faster Data Reflection

LINE Plus replaced a full-dump ETL pipeline for product data with incremental processing using Apache Iceberg and Apache Flink. The previous HBase/Hive workflow rewrote hundreds of millions of rows for every update, causing high compute costs and delays that left data up to an hour out of date. With the new architecture, update intervals were reduced from 60 minutes to 5 minutes—roughly a 12× improvement—while preserving consistency and fault tolerance. ## Limitations of Full-Data ETL - The existing HBase and Hive pipeline continuously collected CDC data in HDFS but had to merge it with existing data and rewrite the entire table before changes became queryable. - This caused: - High compute and storage costs - Dependence on limited shared Hadoop resources - Delayed updates and stale data - Snapshot-based extraction provides consistency, but large snapshots can take hours and retain old versions through MVCC, increasing system overhead. - Processing only the changed rows would reduce the workload from hundreds of millions of records to tens of thousands, separating update cost from total dataset size. ## Introducing Apache Iceberg - Iceberg manages data through metadata and table snapshots rather than relying solely on directory structures like traditional Hive tables. - It supports row-level `upsert` and `delete` operations. - This allows incremental changes to be written without rewriting the entire table, making much shorter ETL intervals possible. ## Requirements for the Streaming Pipeline The team evaluated Spark and Flink against three essential requirements: - **Data freshness:** Late-arriving compensation or replay data must not overwrite newer records. - **End-to-end exactly-once processing:** Iceberg updates and Kafka status messages must not partially succeed. - **Fault tolerance and state management:** Processing state must survive failures and restarts. A Kafka message indicating that all CDC data through a specific timestamp—such as 13:03—has been applied serves as the signal that a bulk extraction can safely begin. This requires complete confidence that the message accurately represents the Iceberg table’s committed state. ## Why Two-Phase Commit Was Necessary - Iceberg and Kafka are independent systems, so writing to one while failing to write to the other could create inconsistent state. - Two-phase commit (2PC) prevents partial success: - Both systems prepare their writes. - They commit only when all required operations succeed. - Any failure causes the operation to roll back. - Exactly-once processing also prevents duplicate or missing records during retries, network failures, or node restarts. - Together, these guarantees make Kafka status messages a reliable representation of the Iceberg table’s state. ## Choosing Flink over Spark - Spark Structured Streaming uses a micro-batch model, which makes fine-grained event-time and state control more difficult. - Flink provides native event-by-event streaming and better support for the required consistency model. - The team used Flink state to track each record’s `updatedate`: - Older late-arriving events are ignored. - Replayed historical data cannot overwrite newer values. - Flink checkpoints: - Persist streaming state externally. - Enable recovery from the latest consistent point. - Integrate with the Kafka sink’s 2PC mechanism. - Kafka messages remain in a pre-commit state until the Iceberg write and checkpoint both succeed. ## Kubernetes Deployment Options - The team compared: - **Native Kubernetes:** Requires manually configuring roles, service accounts, services, routing, deployments, slots, and jobs. - **Flink Kubernetes Operator:** Represents Flink infrastructure and jobs as custom resources, automating configuration such as routing and the web UI through Helm values. - Although Flink has greater operational complexity and a steeper learning curve than Spark, it was selected because it was the only option that satisfied all three core requirements at the engine level. The recommended architecture is an incremental Iceberg pipeline powered by Flink, with stateful processing, checkpoints, and two-phase commit between Iceberg and Kafka. This approach keeps data current, avoids expensive full-table rewrites, and provides reliable recovery and consistency at a five-minute update interval.

Read original(opens in new tab)
line6 min readCurated summary

How did we build a domain-independent chat platform?

MessagingHub turns chat into a reusable platform rather than rebuilding it for each product domain. It separates domain-specific authentication and business context from common chat capabilities, allowing chatbot, customer-support, direct, and group conversations to share the same infrastructure. Its policy-driven design, modular architecture, and configurable metadata aim to reduce integration complexity while preserving flexibility. ## Why MessagingHub Was Introduced - Chat requirements vary across chatbots, customer support, one-to-one conversations, and group chats. - Building each implementation independently increases integration points, system complexity, development cost, and the impact of small changes. - MessagingHub is designed as a domain-independent platform that can be adopted by multiple services. - The platform focuses on chat itself while absorbing external requirements through generalized, reusable structures. - It is currently used by a Japanese food-delivery service for users, drivers, customer-service agents, and restaurants. ## Supported Chat Types - **Chatbots:** Delivered through a public web URL embedded in a partner service’s webview. Scenarios are created and deployed through an administrative console. - **Inquiry chat:** A user is matched with a customer-service agent. The partner domain supplies contextual information such as user details and previous consultation history. - The platform is also structured to support direct one-to-one and group conversations. ## Core Platform Policies ### Authentication and User Identification - MessagingHub does not manage user accounts or domain authentication. - Partner systems handle login, registration, permissions, and the decision of whether a user may access chat. - After authenticating a user, the partner requests a connection token and passes it to the client. - The client uses the token to establish a WebSocket connection; unauthenticated direct access is not allowed. - A user is identified by a `client_id`, combining the partner domain identifier with the partner’s user identifier. - Display names, profile images, and `pushToken` values are supplied and updated by the partner system. ### Service Contexts and Room Types - A **service context** defines which roles may communicate, such as: - `Driver2CS` - `Consumer2CS` - A **chat room type** defines the conversation structure, such as: - `USER_DIRECT` - `USER_GROUP` - `INQUIRY_CHATBOT` - `INQUIRY_CHAT` - The combination of service context and room type controls room creation, participation, and message permissions. ### Room Lifecycle and Data Retention - General room states progress from `WAIT` or `PENDING`, to `SERVICE`, and eventually to `DISABLE` or `BLOCK`, where sending messages is prohibited. - Messages and potentially identifying data are encrypted at rest. - Data can be deleted immediately when all participants leave a room. - Partners can also configure retention periods for automatic deletion of older data. ## Modular Architecture MessagingHub is not a monolithic chat server. Its components have clearly separated responsibilities and communicate through loosely coupled events. - **`connection-manager`** - Manages WebSocket connections and validates connection tokens. - Tracks user connection status. - Helps identify active chatbot scenario connections during `SOFT STOP` processing. - **`chat-app`** - Implements core chat logic, including message delivery, room creation, state transitions, and read status. - Exposes functionality as commands that can be combined for different chat types. - **`message-router`** - Determines where recipients are connected. - Routes messages from the chat server to the appropriate connection-management component. - **`notification-app`** - Sends push notifications when recipients are offline or the application is in the background. - Uses partner-provided `pushToken` values and room-level notification settings. - **`admin-hub`** - Manages chatbot scenario editing and deployment. - Handles agent accounts, roles, service contexts, events, webhooks, monitoring, and statistics. ## Command-Based Chat Flows - Chat behavior is modeled as composable commands. - Common commands provide functionality shared across chat types. - Chatbot and inquiry-chat features add more specialized commands. - This “building block” approach allows business requirements to be assembled without creating a separate chat implementation for every domain. ## Data Model MessagingHub separates operational data from core chat data: - **`chat` database:** Stores users, rooms, participants, metadata, and messages. - **`chat_operation` database:** Stores operational and administrative information. Important entities include: - `chat_user`: Uniquely identifies users by `client_id`. - `chat_room`: Represents rooms and enforces room uniqueness at the schema level. - `chat_member`: Connects users to rooms. - `chat_room_meta`: Stores participant-specific state, including read position, push settings, input restrictions, and room status. - `chat_log`: Stores encrypted messages in a one-to-many relationship with rooms. - `prev_chat_log_id` preserves message ordering. - Room-level first and last message IDs, together with participant read positions, support unread-count calculation. - Partner metadata such as `system_data`, `search_data`, `user_details`, and `descriptions` is stored as JSON. MessagingHub preserves and forwards it without interpreting its domain meaning. - Scheduling, event, and webhook history are tracked through tables such as `chat_schedule`, `chat_event_record`, and `webhook_event_record`. - `service_context`, `chat_event`, and `webhook` configure allowed role relationships, event-message policies, and webhook behavior. ## Chatbot Scenario Management ### Flexible Scenario Structure - Administrators manage multiple chatbot scenarios through an editing tool. - Scenarios define messages, selectable options, and answers. - Webhooks can dynamically generate response content. - The hierarchical data model supports a broad range of chatbot flows. ### Version Deployment and `SOFT STOP` Chatbot scenarios transition through: `WAIT → SERVICE → SOFT STOP → DISABLE` - A newly deployed scenario becomes `SERVICE`. - The previous scenario moves to `SOFT STOP`. - Existing users can finish conversations using the previous version. - New users are directed to the latest scenario. - A scheduler periodically checks whether any users still have active connections to the old scenario. - Connection information is collected from connection-management servers and stored in a shared resource. - Once no active users remain, the old scenario is disabled and the scheduler stops. - This provides backward compatibility without disrupting users during deployment. ## Inquiry Chat Metadata and Lifecycle ### Partner-Defined Metadata Inquiry chat allows partner domains to provide information that helps agents handle cases effectively: - Search data for finding conversations - User details shown to agents - Custom display data - Event data for surveys or webhooks - Basic consultation descriptions - Room settings such as room names and push-notification titles - Tracking data for identifying and mapping rooms in partner systems ### Room Lifecycle Inquiry rooms generally move through: `PENDING → SERVICE → DISABLE → BLOCK` - `PENDING` represents the period while the user waits for an agent match. - `SERVICE` is the active consultation period. - `DISABLE` indicates that the consultation has ended. - `BLOCK` prevents further messaging after closure. MessagingHub’s overall approach is to keep the platform’s responsibilities narrow and reusable while allowing partner domains to own authentication, user meaning, and business-specific metadata. For organizations supporting multiple chat scenarios, a policy-driven, command-based platform with separated components and explicit data ownership can significantly reduce duplication and integration risk.

Read original(opens in new tab)
line3 min readCurated summary

Applying Spark on Kubernetes to process large-scale advertising data for LINE services

LINE Ads processes tens of billions of advertising events daily and nearly one hundred billion internal data records. As growing numbers of features increased computational demands, its Spark-on-YARN environment suffered from resource contention, inefficient scaling, and Hadoop dependencies. The team migrated to Spark on Kubernetes to achieve infrastructure independence, containerized execution, flexible scaling, and easier operational automation. ## Large-Scale LINE Ads Data Pipelines - The data pipeline supports: - Real-time advertising-event processing - Abuse and validity checks - Machine-learning systems and model training - Analytics and system integration - Advertiser reporting - The platform must handle hundreds of billions of events per day and hundreds of thousands per second. - It must provide low latency, elastic capacity, minimal service impact during failures, and rapid recovery. - The most heavily used table grew to approximately 2.91 times its December 2022 size by December 2025 as more features were added. ## Limitations of Spark on YARN - Hadoop’s storage and compute resources were colocated, causing Spark workloads to compete with HDFS and other Hadoop components. - Scaling compute required adding Hadoop nodes, even when additional storage was unnecessary, increasing cost and wasting capacity. - JVM and Spark versions were difficult to manage independently, limiting access to newer Spark features. - Applications became tightly coupled to the Hadoop infrastructure. ## How Spark on Kubernetes Works - Kubernetes replaces YARN as the cluster manager. - Spark drivers and executors run as separate Kubernetes pods. - In cluster mode: - `spark-submit` requests a driver pod. - Kubernetes schedules the driver on an appropriate node. - The driver creates a `SparkContext`, builds the DAG, and requests executors. - Executors run as independent pods with individually allocated CPU and memory. - The driver divides the DAG into stages and distributes tasks to executors. - Shuffle data is normally tied to executor-pod lifecycles unless an external shuffle service is configured. ## Advantages over YARN - **Containerized execution:** Docker images package application dependencies, improving reproducibility and CI/CD integration. - **Infrastructure independence:** Spark can use HDFS, S3, GCS, or other storage systems without requiring a Hadoop cluster. - **Simpler autoscaling:** Kubernetes can scale pods and integrate with cloud VM autoscalers. - **Unified platform:** Spark, Airflow, machine-learning workloads, and API servers can share a Kubernetes cluster. - **Governance and isolation:** Namespaces, `ResourceQuota`, and RBAC provide flexible team-level controls. - **Operational automation:** Helm, ArgoCD, GitOps, and rolling updates enable more automated application management. ## LINE Ads’ Kubernetes-Based System The platform is organized into four layers: - **Deployment layer** - GitHub Actions runs CI workflows based on repository events. - ArgoCD monitors desired and deployed states and supports easier rollback and synchronization. - **Compute layer** - Kubeflow’s Spark Operator deploys applications through the `SparkApplication` Kubernetes custom resource. - Apache YuniKorn schedules batch jobs and supports resource coordination and gang scheduling. - LogSender forwards pod logs to OpenSearch. - ClusterMonitoring sends Prometheus metrics to the company’s monitoring system. - **Storage layer** - Kafka provides high-throughput, low-latency storage for real-time advertising actions. - Hadoop remains available for large-scale, long-term analysis. - **Monitoring layer** - Kubernetes workers and Spark applications are monitored through exposed Prometheus metrics and centralized logging. The migration to Spark on Kubernetes is recommended for organizations whose Spark workloads are outgrowing tightly coupled Hadoop environments. It separates compute from storage, improves deployment flexibility, and allows data applications to be managed as cloud-native workloads.

Read original(opens in new tab)
line4 min readCurated summary

Image Content Moderation in Large-Scale Service Environments (feat. Multimodal LLM)

Image content moderation has evolved from simple rule-based filtering into an AI-powered decision system capable of handling visual context, text, and policy complexity. At large platforms, the challenge is not only accuracy but also latency, cost, scalability, and adaptability to changing policies. LY Corporation addresses these demands through optimized traditional ML models, a hybrid ML–multimodal LLM pipeline, and modular decision-making that combines OCR, visual analysis, and contextual reasoning. ## The Evolution of Content Moderation - Early systems relied on keyword matching, rule-based filters, and predefined patterns. - Machine learning enabled broader pattern recognition and detection of modified or less explicit violations. - Modern systems combine: - Deep learning for text and image classification - Multimodal models for joint image–text understanding - LLMs for context-sensitive judgments - Separate prediction and policy layers for operational flexibility - Despite these advances, image moderation remains difficult because images lack explicit structure and their meaning often depends on context. ## Why Image Moderation Is Difficult - **Visual complexity:** Backgrounds, objects, people, colors, and composition interact in ways that simple object detection cannot fully interpret. - **Context dependency:** Symbols, gestures, and imagery may have different meanings across cultures; embedded text can also determine whether an image is harmful. - **Evasion and variation:** Memes, composites, partially obscured images, and AI-generated edits continually challenge existing detectors. - **Scale requirements:** Platforms may receive millions or tens of millions of images daily, requiring high accuracy alongside low latency, reliability, and cost efficiency. ## LY Corporation’s Moderation API - LY Corporation operates a monitoring platform designed to process large-scale traffic and enforce diverse content policies. - Its image moderation API detects: - Adult content - Violent or graphic scenes - Offensive or disturbing imagery - Identity documents containing personal information - Social media screenshots and other policy-sensitive images - The system is designed to apply service-specific policies consistently while maintaining high throughput. ## Improving Accuracy, Speed, and Cost ### Traditional ML Model Optimization - A PyTorch-based image classification model was selected with latency, cost, and throughput in mind. - The model was converted to ONNX and optimized with FP16 precision. - ONNX Runtime improved execution efficiency, while FP16 reduced memory usage and inference time. - These changes increased throughput by up to **4.3 times**. ### Hybrid ML and Multimodal LLM Architecture - The traditional classifier acts as a fast first-stage filter. - Clear cases are resolved immediately by the image model. - Ambiguous cases are sent to a multimodal LLM for deeper analysis. - More than 90% of production data could be classified by the traditional model alone. - Since multimodal LLM throughput was over 100 times lower than that of the traditional model, routing every image to the LLM would have significantly increased GPU usage and cost. - The hybrid approach preserves high-quality reasoning where necessary while avoiding unnecessary LLM calls. ### vLLM-Based LLM Optimization The team optimized multimodal LLM serving with vLLM, using characteristics such as repeated prompts, predictable token lengths, and prefill-heavy workloads. - **`enable_prefix_caching`:** Reuses KV-cache blocks for repeated system prompts and templates, reducing prefill computation. - **`max_model_len`:** Limits the maximum input-plus-output length to avoid excessive KV-cache allocation. - **`max_num_seqs`:** Controls concurrent requests, balancing throughput against per-request latency and resource contention. - **`max_num_batched_tokens`:** Sets the token budget per scheduling step; larger values can improve throughput for prefill-heavy workloads. - Regularly updating vLLM is recommended because new releases add improvements such as asynchronous scheduling, CUDA graph support, and broader quantization options. ## Moving Beyond Single-Model Policy Prediction - Earlier end-to-end vision models directly predicted final policy categories from images. - This worked for visually obvious violations, such as detecting smoking, but struggled with complex behaviors such as tobacco sales. - Sales-related judgments may require combining: - Product presence - Prices - Sales language - Contact information - Encouragement to purchase - Directly learning every combination of national regulations, service policies, and exceptions created overly complex output classes. - It also made the model harder to extend and maintain, while limiting the use of text embedded in images. ## Hybrid Decision-Making with OCR and Multimodal Reasoning - The redesigned system separates visual and textual information rather than forcing one model to learn every policy combination. - OCR extracts text from images when relevant. - Extracted text helps identify policy-violating behavior or intent. - Visual signals and textual evidence are then combined with a multimodal LLM. - This allows the system to reason about context and intent beyond simple object detection, while making policy logic more modular and adaptable. The practical recommendation is to avoid routing all traffic through expensive general-purpose models. Use fast specialized models for clear cases, reserve multimodal LLMs for ambiguity, optimize serving according to workload characteristics, and separate content understanding from policy decisions so the system can evolve as requirements change.

Read original(opens in new tab)
line4 min readCurated summary

How We Built an SRE Bot That Reduced Our Team’s Repetitive Work by 90%

LINE Home DevOps created an SRE bot to reduce the repetitive work caused by growing services, Flava cloud migration, and increasing developer requests. By making Slack the central interface and automating Jira, Confluence, and workflow updates, the team reduced deployment-request handling from roughly 30 minutes to under one minute. The bot also improved tracking, consistency, and response speed, helping SREs move away from constant firefighting. ## Repetitive SRE Work and Its Costs - Developers frequently asked how to inspect Flava pod logs, request permissions, interpret errors, and access staging environments. - Deployment requests required manual movement between Slack, Confluence, and Jira: - Finding release checklists - Copying information into Jira - Creating missing Fix Versions - Linking Epics and active sprints - Sharing ticket links and deployment documentation - Each deployment request previously took about 30 minutes to an hour. - Manual processing caused omissions and mistakes, especially during urgent releases. - General requests were buried in Slack mentions, making ownership and completion status difficult to track. - Measurement showed that each SRE spent nearly half a day per week on repetitive work. ## Slack-Centered Automation The team adopted the principle that developers should only need Slack, while SREs should be able to manage work with a few clicks. - **Slack as the single source of truth:** Requests begin and remain trackable in Slack. - **Zero manual work:** Rule-based Jira and documentation tasks are automated. - **Immediate visibility:** Status changes and results are posted to Slack in real time. - **Permission control:** Only authorized SRE members can claim or complete requests. ## Key Technical Decisions ### Slack Workflows Instead of Slash Commands - Slash commands are easy to implement but depend on users entering correctly formatted text. - Slack Workflows provide structured forms with required-field validation. - Because Workflows are native Slack functionality, the team avoided building a separate user interface. - The lower usage barrier made adoption more likely. ### Asynchronous Processing - Slack requires event responses within three seconds. - Sequential calls to Jira, Confluence, and other APIs could exceed that limit. - The bot immediately acknowledges the request, then performs external work in the background. - Successes and failures are reported in the Slack thread, keeping processing transparent. ### Redis-Based State Management - In-memory state would be lost whenever the bot restarted. - Slack metadata APIs were considered too slow for real-time interactions such as emoji clicks. - Redis was selected for sub-100-millisecond lookups and persistent state. - A 30-day TTL limits stale data. - Redis transactions using `WATCH/MULTI/EXEC` ensure consistent updates when multiple SREs interact simultaneously. ### Hexagonal Architecture - The bot uses ports and adapters to isolate business logic from external systems. - The architecture separates: - Inbound Slack event adapters - Application use cases and business logic - Outbound Jira, Confluence, and Redis adapters - External API or SDK changes can be handled without modifying core business logic. - This structure also makes testing and future feature development easier. ## Automated Request Scenarios ### Deployment Requests - Developers submit required project, release-version, checklist, and other details through a Slack Workflow. - The bot automatically: - Creates a missing Jira Fix Version - Creates and configures the Jira ticket - Links the Epic - Adds the ticket to the active sprint - Finds the relevant deployment manual - Posts the result to the Slack thread - An SRE can click 👀 to claim the work. - Clicking ✅ completes the Jira ticket and posts a completion notification. - SRE effort falls from about 30 minutes to under one minute, with minimal risk of missing required fields. ### Emergency Deployments - Selecting an urgent request automatically sets Jira Priority to `Highest`. - The bot immediately announces the request in Slack. - An SRE can claim it with 👀, perform the deployment, and complete it with ✅. - The process reduces delays from roughly 30–40 minutes to about one minute. ### General SRE Requests - Requests such as production-access permissions are submitted through a structured Slack Workflow. - The bot creates a Jira ticket, links the Epic, assigns the active sprint, and sets an appropriate priority. - Slack retains the ticket link and status, eliminating the need to search through message history later. - SREs claim and complete the request using the same emoji-based workflow. The main recommendation is to automate repetitive, rule-based operations at the point where requests already occur. A Slack-centered, asynchronous bot with durable state and clean system boundaries can reduce manual effort while making ownership, progress, and completion visible to everyone.

Read original(opens in new tab)
line4 min readCurated summary

Internalizing without specifications: Proving equivalence through validation logic

The article describes how LINE Plus safely internalized black-box e-commerce systems without specifications or source code. The team built an automated equivalence-testing loop using Kafka, CDC, OpenSearch, and ksqlDB to compare legacy and new behavior at massive scale. By repeatedly identifying differences, fixing logic, and rechecking results, they could reduce discrepancies toward zero while also measuring performance and protecting production stability. ## Domain: Products, Catalogs, and Data Ingestion - **Products** are individual seller-listed items, potentially with different prices and shipping conditions. - **Catalogs** group products representing the same model or product type and provide derived value such as: - Real-time lowest prices - Unit-price metrics such as price per 100 ml - **Ingestion** receives large product files from sellers, validates and transforms them into internal formats, and updates product and catalog data. - Because the platform contains tens of millions of catalogs and hundreds of millions of products, small logic differences can affect the entire service. ## The Verification Loop - The goal was not merely to find errors, but to help developers understand and correct them quickly. - Inputs had to be identical for both systems, such as: - The same IDs - The same time-based snapshot - The same product files - Outputs were compared according to system type: - API response objects - Database update values - Final registered product data - The general loop consisted of: - **Trigger:** Database changes, developer requests, or file arrivals - **Execution:** Send identical inputs to legacy and new systems - **Comparison:** Apply logic suited to reads, updates, or end-to-end flows - **Processing:** Store detailed differences and produce real-time statistics - **Action:** Developers inspect dashboards or Slack alerts, fix the implementation, and repeat ## Query Logic Verification - The catalog API was difficult to reproduce because it had over 100 response fields, complex filters, undocumented defaults, and unknown sorting behavior. - CDC streamed database binary-log changes into Kafka, allowing verification to begin from many real catalog states. - The verifier made dual API calls and compared legacy and new responses field by field. - Responses were converted into `Map<String, Object>` structures and compared recursively, avoiding the need to model every response class. - If values differed only because list ordering varied, the verifier sorted serialized values and performed a second comparison. - This helped distinguish real implementation defects from harmless ordering differences. - Kafka isolated verification traffic from production services while handling large event volumes. - Difference events were written to Kafka topics and indexed in OpenSearch for detailed investigation. - ksqlDB aggregated streaming discrepancies and sent Slack notifications when abnormal patterns appeared. - Rate limiting restricted repeated errors, such as those from the same field, to a manageable sample per minute. - Because both APIs were called in parallel, the same pipeline also measured and compared their response times. ## Update Logic Verification - The second case involved recalculating catalog statistics whenever product or catalog data changed. - Unlike read verification, this process tested state transitions and asynchronous updates. - When CDC detected a relevant change: - The new statistics logic calculated an expected result. - The verifier compared it with the result actually written by the legacy logic. - Recursive Map-based comparison checked deeply nested statistics fields. - To avoid wasting resources, verification was triggered only for updates related to the catalog-statistics module. ## Handling Asynchronous Lag - Kafka-based processing caused timing gaps: the verifier could read the database before the legacy update had completed. - The team introduced an **N-attempt retry queue**: - Temporarily inconsistent events were requeued. - Only differences that remained after several retries were treated as genuine defects. - The verifier remained a separate process rather than being embedded in the production statistics stream. - This avoided adding load or latency to the existing processing pipeline while preserving independent verification. ## ETL Batch Verification for Missing Triggers - Real-time comparison could detect incorrect results, but not cases where an update should have happened and never occurred. - During refactoring, a complex combination of product and catalog field changes contained a missing trigger condition. - As a result, some statistics remained stale without generating any comparison event. - To detect these silent omissions, the team designed a separate batch-verification process using ETL data alongside the real-time stream checks. The practical recommendation is to treat system internalization as an evidence-building process: define identical inputs and observable outputs, compare legacy and replacement systems continuously, isolate verification through event streams, and supplement real-time checks with batch validation for silent or missing updates.

Read original(opens in new tab)
line4 min readCurated summary

Utilizing SLI/SLO to Improve Reliability Part 1: SLI/SLO Framework and the Development Story of Service Status Check Tool LINE Status

Repeated SLI/SLO adoption revealed a common process that could be standardized across services. The team turned that process into a reusable framework and built “LINE Status,” an internal tool that automatically presents service health according to user experience rather than raw alerts. Together, these initiatives create a shared organizational language for understanding reliability and its impact on users. ## A Reusable SLI/SLO Framework After applying SLI/SLOs to several platforms and services, the SRE team identified recurring patterns independent of service type. They organized these patterns into a five-stage framework: - **Select critical user journeys (CUJs) and define SLIs** - Identify the experiences most important to users. - Define measurable SLIs that represent those experiences. - **Design instrumentation and metrics** - Build or adapt metrics suitable for each CUJ. - Use standardized naming based on Prometheus or OpenTelemetry. - **Create dashboards and recording rules** - Provide Grafana dashboards for quickly assessing SLO achievement. - Precompute complex PromQL operations to improve query performance. - **Set SLOs and alerts** - Begin with flexible targets, such as 99.9% availability over a 28-day rolling window, allowing roughly 40 minutes of downtime. - Define runbooks for responding to alerts. - Refine targets after operational data and experience accumulate. - **Establish error-budget governance** - Balance release speed against reliability. - Review objectives monthly or quarterly. - Adjust SLOs and processes as needed. The framework is currently distributed as a Confluence template containing guidance and FAQs, reducing the communication effort required from SREs during initial adoption. ## Moving from Alerts to User-Centered Service Status As more services adopted SLI/SLOs, the team wanted a consistent way to understand the health of services they did not directly operate. - The existing public LINE Status API page focused on external users and was updated manually during major incidents. - The new internal tool was intended to: - Represent the status of individual service components. - Update automatically from SLI/SLO alerts and outage data. - Show whether user experience was being affected. - Rather than simply reflecting whether an alert or outage existed, status was based on CUJ-related SLI performance and SLO achievement. - Only representative, high-value CUJs were exposed, avoiding unnecessary technical detail. ## LINE Status Architecture and Interface LINE Status was designed as more than an alert list. It collects events through webhooks, stores them in a separate database, and uses that data to track both current status and historical changes. - Technical SLI/SLO terms are translated into user-facing functions such as “Message Sending” or “Read Receipts.” - Status colors provide an immediate overview: - Green: normal - Yellow: event detected - Red: outage - The main page provides: - An overview of all services. - CUJ status within each service card. - AI-generated one-line summaries. - Service detail pages provide: - Recently affected items near the top. - Timeline-based event displays. - Monthly historical events. - The history page shows: - The scope of impact for each service during an event. - Past events organized by month. The initial implementation took about a month and was refined through colleague feedback. The author also used AI-assisted “vibe coding” for the frontend, emphasizing that clear, detailed requirements were more important than the development tool itself. ## Connecting the Framework and LINE Status Once a service adopts SLI/SLOs through the framework, it can be registered in LINE Status. This connects the definition of reliability objectives with an organization-wide view of service health. - Developers and operators can use the same CUJ-based standards. - Teams can focus on whether users are affected instead of interpreting isolated alerts. - During incidents, the tool helps identify impacted experiences quickly. - Over time, the approach may improve decision-making speed and cross-team communication. The team plans to refine CUJs, SLIs, and status-transition rules through continued operational experience. The practical goal is to make SLI/SLOs a common language for describing service health, enabling reliability practices to scale without depending heavily on individual teams or specialists.

Read original(opens in new tab)
line4 min readCurated summary

Large-scale iOS Settings System Unraveled through AttributedString Structure

LINE’s “Service Configuration” system lets teams deploy features dynamically without waiting for LINE’s two-week app release cycle. As the iOS app grew to roughly 700 configuration keys across 60 modules, its monolithic design created dependency, usability, concurrency, testing, and QA problems. The article argues that the original design was reasonable at small scale but needed to evolve, beginning with lessons from Foundation’s type-safe `AttributedString` design. ## What Service Configuration Provides - Service operators modify values through an administration page. - The server notifies LINE clients, which fetch updated values. - Values are selected based on factors such as: - User region - Device - OS version - The system supports: - Feature flags - Rollbacks - A/B tests - Error-reporting sample rates - UI behavior policies - Configuration is delivered as a string-to-string dictionary, for example: - `"function.media.image_medium": "1280,70"` - `"function.media.message.flow.v2.image": "Y"` ## Problems Caused by the Monolithic Design The original implementation required every key to be declared in one roughly 7,000-line file. Although this was simple initially, growth in teams and modules made the structure increasingly costly. ### Circular Dependencies and Weak Typing - Configuration values were exposed as raw strings because the configuration module could not depend on feature-specific modules. - For example, `"1280,70"` represented image dimensions and JPEG quality, but callers had to parse it into an `ImageTransferQuality` value themselves. - Defining `ImageTransferQuality` in the configuration module avoided repeated parsing but polluted unrelated modules with photo-specific types. - Defining it in the photo module preserved separation of concerns but created an impossible reverse dependency. ### Incomplete and Confusing Abstractions - Developers had to understand server-specific encoding rules and implementation details. - Boolean values were sent as `"Y"` and `"N"`, requiring a custom `decodeBoolIfPresent(forKey:)` method. - The custom decoder’s name resembled Swift’s standard decoding API, making incorrect implementations easy to write and review. - Decoding failures could silently fall back to defaults, making the underlying problem difficult to diagnose. - The same default value often had to be declared three times: - A property-group default - A decoding fallback - A global `defaultConfiguration` entry - These duplicated defaults served subtly different purposes, although the distinctions were generally unnecessary. ### Lack of Thread Safety - Configuration groups were lazily decoded and replaced when new server values arrived. - Multiple services could read configuration values concurrently on different threads. - This caused use-after-free crashes— reportedly hundreds per day—leading to bug tickets and hotfix releases. - As the number of services and concurrent operations increased, this became a systemic issue rather than an occasional edge case. ### No Built-in Debug Overrides - QA frequently needed to temporarily change configuration values. - Because the system had no override mechanism, each feature required custom: - Persistent storage - Debug-menu UI - Value-display text - Implementing this repeatedly required edits across several files and modules. ### Fragmented Test Doubles - Since `LineConfigurationManager` was a singleton, modules created narrow protocols and custom mocks for the settings they used. - This resulted in dozens of duplicated protocols and test doubles. - These had to be updated alongside configuration keys and could fall out of sync. - Differences between mocks and production behavior could allow bugs to escape tests or create false failures. ## Looking to Established Designs The team first distilled the required properties of a replacement: - Type-safe access to a large number of key-value pairs - Independent key definitions by each module - Safe behavior under concurrency They identified Foundation’s `AttributedString` as a useful precedent because it manages many typed attributes while allowing UIKit, AppKit, SwiftUI, and other frameworks to define their own attributes independently. The article presents this as the starting point for redesigning Service Configuration around a more modular and type-safe architecture.

Read original(opens in new tab)
line4 min readCurated summary

Journey Toward Perfect AI Guardrails

NeurIPS 2025 research shows that AI safety is moving beyond simple post-training alignment and output filtering toward system-level, modular defenses. New approaches intervene in reasoning, multimodal interpretation, policy enforcement, and continuous evaluation to balance safety with latency and usefulness. The central conclusion is that deployable AI requires adaptable guardrails designed for real-world systems, not isolated attack benchmarks. ## The Shift Toward Practical AI Safety - Guardrails protect AI services from harmful instructions, privacy leaks, confidential-data exposure, bias, prompt injection, and other failures. - NeurIPS 2025 reflects a broader shift: - From post-training safety tuning to intervention in reasoning mechanisms. - From text-only LLMs to VLMs, RAG systems, and reasoning models. - From laboratory attack scenarios to the practical balance between utility and safety. - The article focuses on guardrail frameworks, multimodal moderation, prompt injection and jailbreaks, hallucinations, and over-refusal. ## Modular Guardrail Frameworks **PRIME Guardrails: A General, Low-Latency Safety Framework for Generative AI** addresses the trade-off between rigorous safety checks and response latency through a modular architecture: - **Policy specification:** Declarative, human-readable rules separate policies from model parameters, allowing legal or policy teams to control behavior. - **Risk sensing and scoring:** Asynchronous detectors combine lexical rules, semantic similarity, and lightweight classifiers. Early exit blocks obvious attacks quickly while allowing domain-specific calibration. - **Intervention router:** A deterministic controller chooses whether to allow, rewrite, or reject an interaction based on policies and risk scores. - **Monitoring and memory:** Lightweight records preserve decisions and rejection reasons for predictability and auditing. - **Evaluation and evolution:** Red-team recipes and automated vulnerability testing help the system adapt to new attack methods. The framework supports defense in depth without running every expensive safety mechanism sequentially. Its modularity, auditing capabilities, and continuous-evaluation loop make it suitable for production environments. ## Turning Governance Policies into Code **Policy-as-Prompt: Turning AI Governance Rules into Guardrails for AI Agents** converts informal organizational materials into runtime-enforceable controls. - The framework analyzes sources such as PRDs, technical design documents, regulations, and source code. - It builds a **source-linked policy tree** connecting individual rules to their original documents. - The policies are compiled into lightweight prompt-based classifiers. - When an agent rejects a request, the system can trace the decision back to its legal or organizational basis. - The approach helps enforce: - Least-privilege access. - Data minimization. - Restrictions on out-of-scope tasks. - Protection against prompt injection. - It may be especially valuable in regulated industries such as finance and healthcare, where frequently changing policies create substantial technical debt. ## Multimodal Safety and VLM Reasoning Vision-language models create new safety challenges because harmful meaning can emerge from interactions between images and text. **GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning** trains models to reason about combined modalities rather than classifying each input independently. - It addresses cases where harmless text obscures harmful visual content, such as an image of a bloodied knife paired with “cooking.” - Its GRPO-based training process includes: - **Safety-aware data concatenation** to create difficult examples containing hidden or mixed harmful content. - **Dynamic clipping** that encourages exploration early in training and tighter refinement later. - **Length-aware safety rewards** that reward concise conclusions supported by reasoning. - The method aims to detect subtle harms such as hate speech hidden in memes and visual metaphors. ## Hidden Vulnerabilities in Multimodal Training Data **VLMs can Aggregate Scattered Training Patches** demonstrates that filtering training images may not be sufficient. - A harmful image can be divided into individually innocuous patches and included in training. - A VLM may reconstruct the harmful concept by associating patches that share the same text label. - The paper calls this behavior **visual stitching**, related to cross-sample reasoning and inductive out-of-context reasoning. - Text labels such as “safe” or “unsafe” can help the model connect fragmented visual information and infer the original image-level meaning. - This suggests that safety evaluations must inspect not only final outputs but also: - Input-processing pipelines. - Cross-sample interactions. - Internal or latent representations. The available article ends while introducing research on distorted safety perception, so that section cannot be summarized further from the provided text. In practice, organizations should combine modular, low-latency enforcement with traceable policy management and multimodal evaluations that test hidden interactions—not just obvious harmful prompts or images.

Read original(opens in new tab)
line4 min readCurated summary

Building an Enterprise LLM Service Part

FAA achieves a 96.1% response rate by favoring simple, maintainable techniques over complex AI architectures. Its design choices were to use RAG instead of knowledge-focused fine-tuning, retrieve complete documents before cutting them into question-relevant sections, and rely on a basic ReAct agent loop rather than elaborate workflows or multiple agents. The article concludes that improving documentation is more valuable than adding complexity when unanswered questions mainly result from missing source material. ## RAG Instead of Fine-Tuning - Fine-tuning was rejected as the primary method for injecting enterprise knowledge. - Research cited in the article found that fine-tuning was highly effective for changing a model’s style—about 97% success—but achieved only about 11% accuracy when teaching new factual knowledge. - FAA’s experiment with approximately 40 examples showed that the model answered the exact training question correctly but failed when the wording changed slightly. - Maintaining larger fine-tuning datasets would require experts to create, verify, and continuously update training examples whenever product documentation changes. - RAG is better suited to frequently changing product information because only the source documents need to be updated. - Fine-tuning may still be useful for domain-specific terminology or reasoning patterns, but not for keeping FAA’s product knowledge current. ## Retrieving Whole Documents Instead of Pre-Chunking - Conventional RAG systems split documents into small chunks before embedding them, improving semantic search precision. - Pre-chunking can remove essential context, especially when references such as “this case” or “the following settings” are separated from the text they depend on. - FAA’s documents are generally short, well-structured, focused on one product and topic, making whole-document retrieval practical. - Instead of chunking before search, FAA embeds and retrieves complete documents, then splits them after the relevant document is known. - The post-split process has two stages: - Split the document by Markdown headers into meaningful sections. - Use a lightweight LLM to select only the sections relevant to the user’s question. - For a question about creating and deleting a VM, the main model might receive only the “VM creation” and “VM deletion” sections. - This extra filtering call remains inexpensive because the lightweight model outputs only section indexes rather than generating a full response. - The key advantage is that splitting happens after the system understands the question, preserving context while delivering only the necessary information. ## ReAct Instead of Complex Agent Workflows - FAA tested plan-and-execute workflows, in which the model first creates a multi-step plan and then carries it out. - Planning and replanning increased system complexity without producing a noticeable improvement in answer quality. - With well-designed tools and carefully filtered context, the model was able to determine tool order on its own. - FAA therefore uses ReAct: the model reasons, takes an action, observes the result, and decides what to do next. - This approach allowed the agent to handle troubleshooting questions without a separate planning layer. ## Rejecting Multi-Agent Architectures - The team also tested specialized agents, such as separate VM and Kubernetes experts. - Delegating questions and assembling the results required additional LLM calls, increasing response time from roughly 9 seconds to 14 seconds in one test. - Multi-agent routing performed poorly for cross-domain questions, such as moving data from a VM to object storage. - Specialists could miss information outside their assigned domain, whereas a single agent could maintain the complete context. - FAA therefore kept one agent with access to progressively disclosed tools and relevant documentation. ## Documentation as the Main Bottleneck - Analysis of unanswered questions showed that about 50% were caused by a documentation gap: no reference document existed. - Other failures were mostly temporary API issues or questions outside FAA’s intended scope. - This suggests the core retrieval and agent system performs well when documentation is available. - The team shares missing questions with product teams, whose updated documents are then re-embedded and incorporated into future evaluations. The practical recommendation is to start with the simplest architecture that fits the data: use RAG for changing knowledge, preserve document context during retrieval, and let a capable model operate through a ReAct loop. In enterprise systems, improving the underlying documentation may produce greater gains than adopting more sophisticated AI frameworks.

Read original(opens in new tab)
line3 min readCurated summary

On-Device Image Model Training for Mess

This post describes an on-device image captioning system for mobile messenger apps. Because autoregressive vision-language models took more than five seconds to generate captions, the team replaced them with a non-autoregressive decoder, reducing latency to roughly 200–400 ms. They then used LLM-based acceptance evaluation, caption re-generation, and multi-stage knowledge distillation to improve quality while keeping the model at 172 MB. ## Why Conventional Captioning Was Unsuitable - Models such as BLIP-2, MobileVLM, PaliGemma, and MiniCPM were too large or slow for mobile deployment. - BLIP-1 was selected as a practical baseline because of its smaller size and clear licensing, but still required more than five seconds after quantization. - Autoregressive decoding generates tokens sequentially, requiring one decoder pass per token. - On a Samsung Galaxy Fold 4, the initial model required about 142 ms per token, or approximately 2.8 seconds for 20 tokens. - Mobile UX required stable latency in the hundreds of milliseconds, including cold-start and variable-device conditions, so simple model compression was insufficient. ## Non-Autoregressive Caption Generation - The system predicts all caption tokens in parallel using a fixed set of learnable query tokens. - This changes the decoding cost from roughly O(T) for autoregressive generation to near O(1) through parallel processing. - The architecture consists of: - An image encoder reused from the previous system - Image embeddings injected as a prefix, following the ClipCap approach - A 66.4-million-parameter Transformer-based text decoder - Twenty learnable query tokens for short captions - Query-CTC loss addresses the alignment problem caused by predicting tokens simultaneously. - The resulting model generated captions in about 200 ms, achieving the required speed improvement. ## Speed Improved, but Caption Quality Declined - Standard metrics such as CIDEr and CLIPScore appeared acceptable. - Manual inspection revealed frequent: - Repeated words, such as “a desk with a computer on a desk” - Spelling errors, such as “a people ons” - Grammatical problems - Incomplete captions, such as “a” - These defects made the model unsuitable for direct use in a messaging product. ## LLM-Based Acceptance Evaluation - The team introduced an “accept ratio” based on GPT-4o mini judgments. - Captions were classified as either `accept` or `non-accept`. - The evaluation checked for: - Duplicate content - Errors - Clarity and grammatical correctness - This better reflected production usability than conventional image-captioning benchmarks. - The low acceptance rate confirmed that CIDEr and CLIPScore alone could not measure whether captions were appropriate for users. ## Data Quality and Knowledge Distillation - Analysis showed that the training data contained inconsistent and noisy captions: - A mixture of very short and overly long descriptions - Unnecessary OCR-like attempts to describe text in images - Uneven language quality - The smaller 66.4-million-parameter model also had less representational capacity than BLIP-1’s 110 million parameters. - Generating an entire sentence in one pass was especially difficult for the compact non-autoregressive model. - The training pipeline was redesigned as an iterative quality-improvement loop: - Train a baseline using the original data - Identify failures with LLM-based acceptance evaluation - Re-caption poor-quality training examples - Distill knowledge from a larger teacher model into the student model - Replace or refine rejected samples and repeat - Architectural scaling and metric optimization did not consistently improve acceptance rates, while re-captioning and knowledge distillation produced more meaningful gains. The practical recommendation is to design on-device captioning around the actual product experience: prioritize parallel decoding for latency, measure quality with production-oriented acceptance criteria, and use carefully curated data plus knowledge distillation to make compact models reliable.

Read original(opens in new tab)