Chatbot

6 posts

toss4 min readCurated summary

6. Beyond Tools: Standards and Responsibility

Toss’s commerce domain found that reliable organizational knowledge cannot be created by writing more documents or adding automation alone. Sustainable knowledge management requires clear standards for what should be documented, who owns it, how it is maintained, and which sources can be trusted. The proposed solution combines AI-assisted documentation with domain-level responsibility and company-wide governance. ## The Limits of Writing Alone - A commerce wiki consolidated terminology, onboarding material, code references, and policy documents. - This reduced confusion over terms such as “seller” and “store” and gave teams a shared starting point. - However, product and policy changes happened faster than one Technical Writer could document them. - Important knowledge also appeared in policy changes, temporary experiments, and chat discussions that were difficult to track manually. ## Why Culture and Participation Were Not Enough - The team promoted documentation through: - A weekly “Commerce Wiki News” newsletter - A policy-question channel and bot - AI documentation workshops - A documentation guild - These efforts increased requests, wiki usage, and adoption of official terminology. - Participation rarely continued beyond an individual’s first document because documentation was not part of normal work priorities. - Writers lacked guidance on: - What information to preserve - How much detail to include - Which audience to target - How to verify whether a document was correct - Documentation became sustainable only when it was treated as a team responsibility embedded in existing workflows. ## AI Automation Reveals the Governance Problem - AI now creates draft documents nightly from two signals: - Product deployment and policy-change announcements - Questions that the commerce Q&A bot cannot answer - AI gathers supporting context and produces drafts, while humans verify the evidence and approve them. - This removes the burden of starting documents from a blank page. - Automation also exposed new problems: - Duplicate or overlapping documents - Unclear authoritative sources - Outdated policies being used in bot answers - Difficulty distinguishing current policies from completed experiments - Automation can collect and draft information, but it cannot decide who owns a policy or whether a document should still be trusted. ## Knowledge Standards and Governance - The focus shifted from “How do we create more documents?” to “How do we create knowledge people can trust?” - Toss’s knowledge-management standards state that teams should: - Preserve recurring questions, important decisions, and information needed by newcomers. - Organize knowledge so both people and AI can find it. - Connect documents to work tools such as Q&A bots and GitHub. - Assign owners and review cycles to keep information accurate and current. - Possible classification systems include: - **Technical layers** for teams with clear data or system flows - **Service domains** for teams responsible for multiple service areas - **Functional units** for systems with distinct feature boundaries - Information becomes organizational knowledge only when it helps people understand situations and make better decisions, with sufficient context and verification. ## The Role of the Knowledge Committee - The Knowledge Committee defines and maintains company-wide documentation standards and resolves conflicts between organizational rules. - Unlike a voluntary guild, it has designated members with decision-making authority. - Governance operates at two levels: - The Technical Writing Chapter manages shared standards for sources, ownership, document status, and lifecycle. - Individual domains decide how those standards apply locally, including ownership, update schedules, and retirement rules. - This balance prevents both inconsistent practices across teams and overly centralized rules that ignore local realities. - For example, commerce teams may need separate handling for permanent deployments and temporary experiments so expired policies do not remain authoritative. The practical recommendation is to treat knowledge management as an operating system for the organization, not a documentation project. AI can reduce the effort of capturing knowledge, but clear ownership, review processes, lifecycle rules, and governance are necessary to keep that knowledge reliable and useful.

Read original(opens in new tab)
toss3 min readCurated summary

Why Does Our Team’s Documentation Fail? (2)

Documentation succeeds not through individual resolve, but through centralized knowledge, clear purpose, and systems that reduce the fear of being wrong. Interviews across a commerce domain and a server-development chapter show that documentation strategies must match an organization’s existing maturity, audience, and work patterns. AI lowers the cost of writing and sharing knowledge, while also requiring more deliberate document organization. ## Lessons from Early Attempts - The commerce wiki was initially designed as a self-service platform supported by workshops and guilds. - Workshops could encourage a first contribution, but sustaining second, third, and later contributions was difficult. - In the Ads domain, documentation was already strong, so the better approach was to respect existing conventions and help people locate knowledge rather than create another system. - Organizations with little documentation need to build foundational knowledge; organizations with mature documentation need better discovery and maintenance. ## Reducing the Fear of Asking and Writing - Developers often avoid asking questions because doing so publicly reveals what they do not know. - They may also hesitate to publish documentation because they fear their knowledge could be inaccurate. - The team addresses this through: - **“Lee’s Development Consultation Week,”** which normalizes questions and encourages teammates to answer questions Lee cannot address. - **A daily knowledge bot, “Ha,”** which shares short server-development tips automatically. - Correcting or adding to an existing shared post feels easier than writing a complete document from scratch. ## How AI Has Changed Documentation - AI makes it faster to create initial drafts and distribute knowledge through chatbots. - It also enables measurement of knowledge flow, including: - The number of questions asked. - Whether teammates provide answers. - The quality and content of those answers. - The number of new documents and week-over-week growth. - These metrics reveal knowledge gaps and recurring questions without manually reviewing every channel. - AI also creates a need for more detailed internal context than human readers typically require. - Commerce therefore separates: - Central, human-friendly documentation managed by technical writers. - Team-repository documentation containing detailed, team-specific context useful to AI but unnecessary for everyone else. ## Shared Principles and Organizational Differences - Both domains and chapters should centralize knowledge and avoid spreading it across too many channels. - Domains typically: - Connect documentation to products and code. - Change rapidly. - Serve a broad and varied audience. - Chapters typically: - Document conventions, working methods, and professional knowledge. - Change more slowly. - Focus on productivity and capability development. - Have a clearer audience, such as a specific role or discipline. - Domain documentation should be understandable even to non-developers, since developers may also lack context outside their specialties. - Separating guides, capability-based policies, glossaries, and metrics helps different readers find documents suited to their needs. ## Where to Begin - First diagnose the organization’s current documentation maturity by asking what people do when they get stuck: - **Ask coworkers or search chat:** foundational documentation is largely missing. - **Search documents:** assess whether information is easy to find; fill gaps if searches fail. - **Ask an AI or bot:** evaluate whether answers are accurate and whether the underlying documents are complete, current, centralized, and sufficiently contextualized. - Define the specific problem driving documentation, rather than starting with a vague goal. - Begin with a focused need, such as creating a glossary for inconsistent terminology or building references for sharing knowledge with other teams. The practical recommendation is to centralize knowledge, identify the organization’s biggest documentation gap, and build a low-friction system where people can contribute, correct, and consume information without relying solely on personal effort.

Read original(opens in new tab)
toss3 min readCurated summary

Creating a Role That Never Existed in the World

Technical Writing at Toss has evolved from producing documents to designing organizational knowledge systems. The core argument is that code captures outcomes but not the context, decisions, and history behind them—information that both employees and AI need. Toss’s Technical Writing Chapter therefore aims to make knowledge discoverable, structured, and increasingly self-sustaining, with the long-term goal of making the chapter unnecessary. ### Why Code Alone Is Not a Sufficient Source of Truth - Code records what a system does, but rarely explains: - Why it was designed that way - Which alternatives were considered - What organizational history shaped the decision - Employees often reconstruct context from colleagues’ memories or years-old messenger threads. - AI faces the same problem: it understands general knowledge but lacks an organization’s internal context. - A true Single Source of Truth requires both the code and the surrounding knowledge that explains it. ### From Writing Documents to Bringing Knowledge to People - Early work focused on creating onboarding documentation for the frontend chapter. - The team recognized that well-written documents are ineffective if people do not actively read or find them. - Toss created “Parkssi,” a chatbot integrated into messaging tools and IDEs: - Users ask questions conversationally. - Answers are based on existing documents. - Sources are provided for verification. - Instead of requiring employees to search for documents, the knowledge reaches them where they work. - Teams that previously ignored documentation began requesting similar systems to reduce repetitive questions and capture implicit knowledge. ### From Documents to Knowledge Systems - A knowledge system collects scattered information from code, conversations, deployments, and other sources. - It structures that information so both humans and AI can understand and use it through questions and automation. - Properly designed systems: - Increase productivity across the organization - Reduce communication costs as the company grows - Help employees adapt quickly and consistently - Toss defines knowledge as verified information that helps people understand situations, make better decisions, and act within a specific context. ### The Four Responsibilities of Toss’s Technical Writing Chapter - **Build products:** The chapter develops and operates “Todok,” an internal knowledge-management platform. - **Lead organizational documentation:** TWs work directly with teams to collect and organize knowledge according to each group’s needs. - **Automate Technical Writing:** AI workflows and automation help employees create and review documentation with consistent quality. - **Shape documentation culture:** The chapter runs company-wide sessions, documentation guilds, and knowledge committees focused on making knowledge easier for AI and people to use. ### A Broader Definition of the Technical Writer - Although the job title remains “Technical Writer,” the role now resembles a knowledge-infrastructure designer. - The role expanded by following organizational problems rather than staying within a fixed professional boundary. - The chapter’s ultimate goal is for teams to create and manage knowledge independently, eliminating the need for a dedicated TW chapter. - The series will explore Toss’s knowledge product, failed documentation approaches, automation of TW work, and the path toward self-sufficient organizations. Technical writers can create greater value by designing systems that make knowledge continuously available—not merely by writing more documents. As AI reshapes every profession, expanding a role around the problems it must solve is presented as an opportunity for any discipline.

Read original(opens in new tab)
toss3 min readCurated summary

How AI Has Changed the Product Design Process

Toss redesigned its customer-service chatbot after finding that 60% of users abandoned the existing menu-based experience. Instead of forcing customers to identify the correct category, the team used AI to interpret natural-language requests and connect users directly to solutions. The project’s key insight was to prototype and validate experiences early, then improve the system through reusable rules rather than endlessly editing individual scenarios. ## Replacing Menu Navigation with Natural Language - The customer center serves roughly 600,000 visitors monthly, including about 170,000 chat users. - Users often know their problem—such as a failed payment or mistaken transfer—but not how Toss categorizes it. - The team therefore aimed to build a chatbot that understood intent and guided users directly to the appropriate resolution. ## Generating Scenario Drafts from Real 상담 Data - AI created initial scenarios for the 20 most common customer-service issues. - Personal information was removed and data was anonymized or pseudonymized before use. - Instead of relying primarily on policy documents, the team trained the AI on real consultation data. - This revealed: - The language customers actually use - The questions agents ask to narrow down causes - The order in which problems are diagnosed and resolved - Ways to explain complex policies in understandable language - As a result, the chatbot behaved more like a human 상담사 than a system that simply listed policies. ## Validating Many Situations with a Scenario Hub - A single request can hide dozens of possible conditions. For example, membership cancellation may depend on whether: - The monthly fee was already paid - Benefits were used - Cancellation was already scheduled - The team built a scenario hub where these conditions could be saved and selected. - Each selection immediately opened a prototype conversation with the chosen conditions applied. - This enabled rapid testing after scenario changes, new branches, or rule updates. - The prototype became an experimentation environment rather than a static mockup, allowing the team to judge realistic conversations instead of relying on assumptions. ## Replacing Individual Fixes with General Rules - Testing revealed recurring problems, including: - Asking for information the chatbot already knew - Repeating the same explanation - Guessing when it lacked reliable information - Rather than fixing each scenario separately, the team created general rules such as: - Resolve the customer’s problem before giving explanations - Admit uncertainty instead of guessing - Escalate to a human agent only under defined conditions - Clearly limit the AI’s authority - Improving one scenario helped only that scenario, while improving a rule improved many scenarios at once. ## Designing the Experience Before the System - The team first envisioned the desired customer experience, then worked backward to determine the required: - Data structures - APIs - Operational tools - System architecture - This reversed the conventional process of designing infrastructure first and fitting the user experience onto it. - Within approximately three weeks, the team completed analysis, experience design, scenario generation, prototyping, testing, and refinement. - AI changed not only productivity but also the order of product design: designers could validate more possibilities before committing to implementation. ## Practical Lessons - Build an imperfect prototype early instead of waiting for a complete design. - Study real user data before relying solely on guides or documentation. - When similar problems recur, identify a shared rule instead of repeatedly patching individual cases. The broader recommendation is to use AI as a fast experimentation tool. Its greatest value is not simply producing artifacts, but making it cheaper and faster to test which experiences and design principles actually work.

Read original(opens in new tab)
line6 min readCurated summary

How did we build a domain-independent chat platform?

MessagingHub turns chat into a reusable platform rather than rebuilding it for each product domain. It separates domain-specific authentication and business context from common chat capabilities, allowing chatbot, customer-support, direct, and group conversations to share the same infrastructure. Its policy-driven design, modular architecture, and configurable metadata aim to reduce integration complexity while preserving flexibility. ## Why MessagingHub Was Introduced - Chat requirements vary across chatbots, customer support, one-to-one conversations, and group chats. - Building each implementation independently increases integration points, system complexity, development cost, and the impact of small changes. - MessagingHub is designed as a domain-independent platform that can be adopted by multiple services. - The platform focuses on chat itself while absorbing external requirements through generalized, reusable structures. - It is currently used by a Japanese food-delivery service for users, drivers, customer-service agents, and restaurants. ## Supported Chat Types - **Chatbots:** Delivered through a public web URL embedded in a partner service’s webview. Scenarios are created and deployed through an administrative console. - **Inquiry chat:** A user is matched with a customer-service agent. The partner domain supplies contextual information such as user details and previous consultation history. - The platform is also structured to support direct one-to-one and group conversations. ## Core Platform Policies ### Authentication and User Identification - MessagingHub does not manage user accounts or domain authentication. - Partner systems handle login, registration, permissions, and the decision of whether a user may access chat. - After authenticating a user, the partner requests a connection token and passes it to the client. - The client uses the token to establish a WebSocket connection; unauthenticated direct access is not allowed. - A user is identified by a `client_id`, combining the partner domain identifier with the partner’s user identifier. - Display names, profile images, and `pushToken` values are supplied and updated by the partner system. ### Service Contexts and Room Types - A **service context** defines which roles may communicate, such as: - `Driver2CS` - `Consumer2CS` - A **chat room type** defines the conversation structure, such as: - `USER_DIRECT` - `USER_GROUP` - `INQUIRY_CHATBOT` - `INQUIRY_CHAT` - The combination of service context and room type controls room creation, participation, and message permissions. ### Room Lifecycle and Data Retention - General room states progress from `WAIT` or `PENDING`, to `SERVICE`, and eventually to `DISABLE` or `BLOCK`, where sending messages is prohibited. - Messages and potentially identifying data are encrypted at rest. - Data can be deleted immediately when all participants leave a room. - Partners can also configure retention periods for automatic deletion of older data. ## Modular Architecture MessagingHub is not a monolithic chat server. Its components have clearly separated responsibilities and communicate through loosely coupled events. - **`connection-manager`** - Manages WebSocket connections and validates connection tokens. - Tracks user connection status. - Helps identify active chatbot scenario connections during `SOFT STOP` processing. - **`chat-app`** - Implements core chat logic, including message delivery, room creation, state transitions, and read status. - Exposes functionality as commands that can be combined for different chat types. - **`message-router`** - Determines where recipients are connected. - Routes messages from the chat server to the appropriate connection-management component. - **`notification-app`** - Sends push notifications when recipients are offline or the application is in the background. - Uses partner-provided `pushToken` values and room-level notification settings. - **`admin-hub`** - Manages chatbot scenario editing and deployment. - Handles agent accounts, roles, service contexts, events, webhooks, monitoring, and statistics. ## Command-Based Chat Flows - Chat behavior is modeled as composable commands. - Common commands provide functionality shared across chat types. - Chatbot and inquiry-chat features add more specialized commands. - This “building block” approach allows business requirements to be assembled without creating a separate chat implementation for every domain. ## Data Model MessagingHub separates operational data from core chat data: - **`chat` database:** Stores users, rooms, participants, metadata, and messages. - **`chat_operation` database:** Stores operational and administrative information. Important entities include: - `chat_user`: Uniquely identifies users by `client_id`. - `chat_room`: Represents rooms and enforces room uniqueness at the schema level. - `chat_member`: Connects users to rooms. - `chat_room_meta`: Stores participant-specific state, including read position, push settings, input restrictions, and room status. - `chat_log`: Stores encrypted messages in a one-to-many relationship with rooms. - `prev_chat_log_id` preserves message ordering. - Room-level first and last message IDs, together with participant read positions, support unread-count calculation. - Partner metadata such as `system_data`, `search_data`, `user_details`, and `descriptions` is stored as JSON. MessagingHub preserves and forwards it without interpreting its domain meaning. - Scheduling, event, and webhook history are tracked through tables such as `chat_schedule`, `chat_event_record`, and `webhook_event_record`. - `service_context`, `chat_event`, and `webhook` configure allowed role relationships, event-message policies, and webhook behavior. ## Chatbot Scenario Management ### Flexible Scenario Structure - Administrators manage multiple chatbot scenarios through an editing tool. - Scenarios define messages, selectable options, and answers. - Webhooks can dynamically generate response content. - The hierarchical data model supports a broad range of chatbot flows. ### Version Deployment and `SOFT STOP` Chatbot scenarios transition through: `WAIT → SERVICE → SOFT STOP → DISABLE` - A newly deployed scenario becomes `SERVICE`. - The previous scenario moves to `SOFT STOP`. - Existing users can finish conversations using the previous version. - New users are directed to the latest scenario. - A scheduler periodically checks whether any users still have active connections to the old scenario. - Connection information is collected from connection-management servers and stored in a shared resource. - Once no active users remain, the old scenario is disabled and the scheduler stops. - This provides backward compatibility without disrupting users during deployment. ## Inquiry Chat Metadata and Lifecycle ### Partner-Defined Metadata Inquiry chat allows partner domains to provide information that helps agents handle cases effectively: - Search data for finding conversations - User details shown to agents - Custom display data - Event data for surveys or webhooks - Basic consultation descriptions - Room settings such as room names and push-notification titles - Tracking data for identifying and mapping rooms in partner systems ### Room Lifecycle Inquiry rooms generally move through: `PENDING → SERVICE → DISABLE → BLOCK` - `PENDING` represents the period while the user waits for an agent match. - `SERVICE` is the active consultation period. - `DISABLE` indicates that the consultation has ended. - `BLOCK` prevents further messaging after closure. MessagingHub’s overall approach is to keep the platform’s responsibilities narrow and reusable while allowing partner domains to own authentication, user meaning, and business-specific metadata. For organizations supporting multiple chat scenarios, a policy-driven, command-based platform with separated components and explicit data ownership can significantly reduce duplication and integration risk.

Read original(opens in new tab)
google3 min readCurated summary

Exploring the feasibility of conversational diagnostic AI in a real-world clinical study

The study evaluated Google’s conversational medical AI, AMIE, in a real-world primary care workflow rather than simulated cases. In a prospective, IRB-approved study at Beth Israel Deaconess Medical Center, AMIE conducted supervised pre-visit history-taking with 100 patients. Results suggested that supervised deployment was feasible and conversationally safe, while AMIE’s diagnostic and management-plan quality was broadly comparable to that of primary care physicians, with physicians performing better on practicality and cost effectiveness. ## Study Design and Clinical Workflow - Patients with new, non-emergency, episodic complaints used AMIE through a secure web link before an in-person or telehealth appointment. - A physician supervised each AI-patient interaction through live video and screen-sharing. - AMIE produced a transcript and summary for the patient’s primary care physician. - Independent clinical evaluators assessed: - The quality of the AMIE conversation - AMIE’s differential diagnoses - AMIE’s management plans - Comparable outputs from physicians - The study was prospective, single-center, single-arm, pre-registered, and IRB approved. ## Participants - 100 adults completed the AMIE interaction. - 98 attended their scheduled primary care appointments. - Participants represented varied ages, racial and ethnic groups, health literacy, technology literacy, and prior chatbot experience. - Compared with all 1,452 urgent care visits during the study period, participants tended to be younger, although the sample reflected the broader population’s female and white demographic skew. ## Safety Oversight - Human supervisors could stop an interaction if they observed: - Immediate risk of harm to the patient or others - Significant emotional distress related to the AI interaction - Potential clinical harm - A patient’s explicit request to end the session - No safety stops were required across the study. - The authors interpret this as evidence that supervised AMIE interactions were conversationally safe in this setting. ## Clinical Reasoning Performance - Three independent clinical evaluators reviewed each case using blinded, randomized assessments. - AMIE and physicians showed similar overall quality for: - Differential diagnoses - Management plans - Management-plan appropriateness and safety - Physicians performed better on the practicality and cost effectiveness of management plans. - AMIE’s differential-diagnosis accuracy was reported as high, including cases where the final diagnosis was confirmed through diagnostic testing. ## Patient and Clinician Experience - The study measured trust, perceptions, and acceptance among both patients and clinicians. - Patient trust in AI increased after interacting with AMIE. - Overall findings indicated that the system was well received within the supervised pre-visit workflow. The study supports cautious, supervised testing of conversational diagnostic AI in clinical environments. It does not establish that AMIE can independently replace clinicians; rather, it suggests that pre-visit information gathering may be a practical early use case, provided rigorous oversight, safety protocols, and further evaluation in larger and more diverse settings.

Read original(opens in new tab)