techlist.io
KREN
toss

Rubric Design and System Implementation for Skill Quality Management (opens in new tab)

llmpythonyamlcoding-agentsregular-expressionsast-parsingfrontmatter

Toss’s AI DX Team created a 30-item rubric to improve the quality of internal Skills used by coding agents. The central conclusion is that deterministic defects should be checked with rules, while semantic questions—especially whether a Skill will be triggered—should be evaluated by an LLM. This separation improves accuracy, cost efficiency, and developer feedback.

Why Skill Evaluation Is Difficult

Skills are artifacts that are both invoked and read by LLMs, so they lack the compiler and test-based validation available for code.

  • Defects can accumulate silently:
    • A Skill may never be invoked.
    • It may be invoked but have little practical effect.
  • Two especially common problems are:
    • Trigger failure: Trigger conditions are placed in the Skill body instead of its description. Agents inspect the description when deciding whether to invoke a Skill; the body is read only afterward.
    • Format failure: Invalid naming conventions, mismatched folder names, or malformed metadata can prevent the agent from recognizing the Skill at all.

Rules for Deterministic and Semantic Checks

The rubric explicitly separates the 30 checks into:

  • 17 rule-based checks
    • Use regular expressions, counts, and AST parsing.
    • Handle objective issues consistently and cheaply.
  • 13 model-based checks
    • Use an LLM for meaning-dependent judgments.
    • Evaluate questions such as whether a description adequately communicates when the Skill should be used.

Mixing the two approaches causes problems:

  • LLMs may overlook clear format violations.
  • Regular expressions produce false positives when trying to understand varied natural-language intent.
  • Rule checks can run on every pull request at nearly no cost.
  • Model checks run only after structural blockers have passed, reducing LLM expenses.

Rubric Structure and Severity

The rubric contains six sections and 30 evaluation items.

  • Each item is classified as:
    • BLOCKER
    • MAJOR
    • MINOR
  • Results are summarized using grades from S to F.
  • Any single BLOCKER automatically produces an F.
  • The grade is primarily a compact signal for authors; merge eligibility is simplified to whether the result is F or not-F.

Validity: Does the Skill Need to Exist?

The validity section contains three MAJOR checks.

  • It asks whether the Skill:
    • Has a legitimate reason to exist.
    • Provides recurring or reusable value.
    • Offers something more useful than simply asking the coding agent to perform the task directly.
  • This section is intended to identify Skills that should not have been created in the first place.

Structure: Catching Format Errors

The structure section has eight checks, including five BLOCKER-level checks.

The rule-based implementation verifies items such as:

  • Presence and parseability of YAML frontmatter.
  • name following lowercase kebab-case.
  • Consistency between the Skill’s name and its folder name.
  • Description length between 1 and 1,024 characters.
  • Absence of XML tags in the body.

The checks collect all failures and return them together so authors can fix multiple problems from a single pull-request comment. Only an unrecoverable frontmatter parsing failure causes an immediate return.

Trigger Design: Making Skills Discoverable

The trigger section contains six checks, including one BLOCKER.

  • A description must communicate both:
    • WHAT the Skill does.
    • WHEN it should be used.
  • A detailed “when to use” section in the body is insufficient because the agent cannot see the body during invocation selection.
  • The team initially tried regular expressions to detect trigger signals such as:
    • “when”
    • “use when”
    • Korean expressions meaning “when using” or “at the time of.”
  • This produced failures because trigger intent can be expressed through emojis, indirect wording, and many other forms that keyword lists cannot cover.
  • The final approach assigns the semantic question—whether the description covers the body’s trigger conditions—to an LLM.

The resulting design favors simple, repeatable rule checks for formal correctness and model-based evaluation only where natural-language meaning is unavoidable.