Toss’s Speed and Quality: Are Commercial Tools Enough? — Tossion (opens in new tab)
Toss’s QA Platform team built Tossion to replace a rigid commercial test case management system with a platform they could continuously adapt. It unifies test cases, manual and automated results, evidence, history, and release snapshots while preserving a clear record of what was tested at each point in time. The platform later expanded into AI-assisted PR analysis, test case generation, and real-device regression testing, enabling QA processes to evolve at Toss’s speed. ## Why Tossion Was Needed - Previous QA information was scattered across automation results, manual test results, test cases, and decision evidence. - Answering questions such as “How did this test perform last time?” required checking multiple systems. - The commercial TCM could not be modified quickly, and requests for new features often stalled. - Toss needed a platform that could be changed directly as new QA requirements emerged. ## Preserving Test History - Tossion organizes work as: - Project - Suite - Section - Test Case - Test cases continue to change as products evolve, but test runs must preserve historical reality. - When a test run is created, it copies the test case’s assignee, steps, and description instead of merely referencing the current test case. - Status changes create history entries showing who made the decision, when, and against which version. - Closed test runs store snapshots of test cases, comments, and automation results, so reports remain available even if the original test cases are later deleted. - Manually selected test cases override Type or Platform filters. ## Collaborative Test Execution - Test runs progress through `Active → Completed → Closed`. - Assignee-based charts show each person’s remaining work at a glance. - Fields such as Status, Type, Assignee, Version, Platform, RNR, and History are added or removed based on actual usage. - Multiple users can work in the same test run simultaneously: - Active viewers appear as avatars. - Editing locks prevent conflicts. - Locks are automatically released when users leave or disconnect. - Status changes are synchronized without requiring a refresh. - The main benefit is short feedback loops: requested improvements can be built and deployed immediately. ## AI-Assisted Release and PR Analysis - Toss planned to use AI for test case creation, PR analysis, regression automation, and execution. - Tossion analyzes every PR included in a release, separating those with QA labels from those marked as not requiring QA. - The goal is not merely to summarize changes, but to audit whether “no QA needed” classifications are correct. - An agent running on a QA server: - Registers with Tossion. - Polls for work. - Executes the AI already authenticated on the server. - Hundreds of PRs are divided into smaller batches and analyzed in parallel for deeper review. - Results are checked for vague or unusable content, such as: - Missing screens or conditions. - Repeated PR titles. - Raw function names. - Missing reproduction steps, expected results, failure symptoms, or reasoning. - Invalid analyses are retried, and the number of merged PRs is compared with the number of analyzed PRs to detect omissions. - Historical incident data increases risk when a new PR touches files associated with past outages. - Results are uploaded after each batch, allowing interrupted jobs to resume without repeating completed analysis. - The resulting “must-test” list defines the scope of the Sprint test run. ## AI-Generated Test Cases - AI generates test cases for new features, but Tossion controls their placement in the test case tree. - AI returns paths such as `Assets > Account Connection > Select Bank`; Tossion converts them into actual sections, reusing existing sections or creating missing ones. - Reliability is checked in three layers: - AI reviews its own output for missing branches, error cases, and boundary values. - Scripts validate naming, numbering, screen coverage, and requirements coverage. - A separate AI creates an independent test plan based on ISTQB and ISO/IEC 29119 practices. - The plan and generated cases are compared: - Planned but missing cases indicate omissions. - Cases outside the plan indicate unplanned scope. - This catches state-based scenarios that screen-oriented test generation might miss. - Generated cases are uploaded through a portable CLI rather than direct UI integration, reducing dependency on local packages, runtimes, and paths. - Stable cases can later become regression tests. ## Running Regression Tests on Real Devices - A Tossion execution modal specifies: - Device - Build - Test scope - Target test run - Runners connected to Android and iOS physical devices register themselves with Tossion but require administrator approval before receiving work. - Runners report device status every 30 seconds. - Tossion supplies the correct build for installation, ensuring results are tied to a known version. - Users can run the full regression suite or a selected section. - Progress is streamed as scenarios finish, including duration and failure messages. - Results are stored at the step level: - Status - Duration - Error message - Screenshot - Scenario-level video - Automation results can be attached directly to the relevant Sprint test run, making them part of the same record used for manual testing. ## Linking Automation Results to Test Cases - Aggregate reports such as “200 regression tests, 3 failures” do not identify which manual test case rows were covered. - Tossion aims to connect test cases and automation in both directions: - Generate automation code from test cases. - Write automation results back into individual test cases. - This removes the need for QA engineers to manually reconcile separate reports with test case lists. Tossion’s central value is not just test management, but ownership and adaptability. By combining immutable test history, collaborative execution, AI validation, and real-device automation in one extensible platform, Toss’s QA team can adjust its tools and processes as quickly as product requirements change.
