Rubric Design and System Implementation for Skill Quality Management
Toss’s AI DX Team created a 30-item rubric to improve the quality of internal Skills used by coding agents. The central conclusion is that deterministic defects should be checked with rules, while semantic questions—especially whether a Skill will be triggered—should be evaluated by an LLM. This separation improves accuracy, cost efficiency, and developer feedback. ## Why Skill Evaluation Is Difficult Skills are artifacts that are both invoked and read by LLMs, so they lack the compiler and test-based validation available for code. - Defects can accumulate silently: - A Skill may never be invoked. - It may be invoked but have little practical effect. - Two especially common problems are: - **Trigger failure:** Trigger conditions are placed in the Skill body instead of its description. Agents inspect the description when deciding whether to invoke a Skill; the body is read only afterward. - **Format failure:** Invalid naming conventions, mismatched folder names, or malformed metadata can prevent the agent from recognizing the Skill at all. ## Rules for Deterministic and Semantic Checks The rubric explicitly separates the 30 checks into: - **17 rule-based checks** - Use regular expressions, counts, and AST parsing. - Handle objective issues consistently and cheaply. - **13 model-based checks** - Use an LLM for meaning-dependent judgments. - Evaluate questions such as whether a description adequately communicates when the Skill should be used. Mixing the two approaches causes problems: - LLMs may overlook clear format violations. - Regular expressions produce false positives when trying to understand varied natural-language intent. - Rule checks can run on every pull request at nearly no cost. - Model checks run only after structural blockers have passed, reducing LLM expenses. ## Rubric Structure and Severity The rubric contains six sections and 30 evaluation items. - Each item is classified as: - **BLOCKER** - **MAJOR** - **MINOR** - Results are summarized using grades from **S to F**. - Any single BLOCKER automatically produces an **F**. - The grade is primarily a compact signal for authors; merge eligibility is simplified to whether the result is F or not-F. ## Validity: Does the Skill Need to Exist? The validity section contains three MAJOR checks. - It asks whether the Skill: - Has a legitimate reason to exist. - Provides recurring or reusable value. - Offers something more useful than simply asking the coding agent to perform the task directly. - This section is intended to identify Skills that should not have been created in the first place. ## Structure: Catching Format Errors The structure section has eight checks, including five BLOCKER-level checks. The rule-based implementation verifies items such as: - Presence and parseability of YAML frontmatter. - `name` following lowercase kebab-case. - Consistency between the Skill’s `name` and its folder name. - Description length between 1 and 1,024 characters. - Absence of XML tags in the body. The checks collect all failures and return them together so authors can fix multiple problems from a single pull-request comment. Only an unrecoverable frontmatter parsing failure causes an immediate return. ## Trigger Design: Making Skills Discoverable The trigger section contains six checks, including one BLOCKER. - A description must communicate both: - **WHAT** the Skill does. - **WHEN** it should be used. - A detailed “when to use” section in the body is insufficient because the agent cannot see the body during invocation selection. - The team initially tried regular expressions to detect trigger signals such as: - “when” - “use when” - Korean expressions meaning “when using” or “at the time of.” - This produced failures because trigger intent can be expressed through emojis, indirect wording, and many other forms that keyword lists cannot cover. - The final approach assigns the semantic question—whether the description covers the body’s trigger conditions—to an LLM. The resulting design favors simple, repeatable rule checks for formal correctness and model-based evaluation only where natural-language meaning is unavoidable.
Read original(opens in new tab)
