Developing a Model to Assess Harmfulness from Open Chat Names and Descriptions (opens in new tab)

The AI Services Lab developed a model to automatically detect harmful LINE OpenChat names and descriptions, reducing the need for manual review. The project improved an existing moderation system by cleaning inconsistent labels, selecting a lightweight safety-tuned decoder model, and adapting it to predict both penalty levels and reasons. Granite Guardian 3.1 2B was ultimately fine-tuned with LoRA and deployed using token-probability-based inference.

OpenChat Monitoring

• Users must provide an OpenChat name and may add a description.
• Names and descriptions are reviewed whenever they are created or modified.
• LINE processes a large volume of global OpenChats, making fully manual moderation impractical.
• The project aimed to:
  • Expand automated moderation to countries requiring more detailed judgments.
  • Improve accuracy in regions already using automation.
  • Reduce the amount of content requiring human review.

Data Cleansing

• Training data consisted of previously manually reviewed OpenChat names and descriptions.
• Only records reviewed under the current moderation guidelines were used.
• Identical name-description pairs sometimes had conflicting penalty outcomes.
• Labels were consolidated using these rules:
  • Select the most severe penalty if it appeared at least twice.
  • If it appeared only once, treat it as possible noise and select the second-most-severe penalty.
  • Choose the most frequent penalty reason.
  • If reasons were tied, choose the globally rarer reason, following a TF-IDF-like principle that rarer reasons may be more specific.
• This process produced a single, consistent label for each identical input.

Selecting the Pretrained Model

The team evaluated models according to four requirements:

• Decoder-based architecture.
• Fine-tuned for safety moderation.
• Approximately 2 billion parameters.
• Apache license for commercial use.

Granite Guardian 3.1 2B was selected because:

• It is designed to classify harmfulness through the probabilities of “Yes” and “No” tokens.
• Restricting predictions to predefined tokens avoids unpredictable free-form responses.
• Token probabilities provide confidence scores that can be thresholded for operational needs.
• Its relatively small size supports lower serving costs and faster responses.

Fine-Tuning for Penalty Prediction

• A simple harmful/not-harmful classification was insufficient because moderation decisions include different penalty levels and reasons.
• The model was trained to produce structured responses containing:
  • An Action penalty code.
  • A Reason penalty reason.
• Cross-entropy loss was calculated only over the assistant’s response tokens, not the entire prompt.
• This focuses training on predicting moderation decisions rather than reproducing the input text.
• LoRA was used instead of full-parameter fine-tuning:
  • The base model parameters remained frozen.
  • Only small trainable matrices representing parameter updates were optimized.
  • This reduced memory and training costs while preserving pretrained capabilities.

Inference Design

• During inference, the model calculates logits for all possible next tokens.
• The system extracts only the logits corresponding to valid penalty-code tokens, converts them to probabilities, and selects the highest-scoring code.
• It then predicts the penalty reason in a second step.
• Existing operational codes consisted of arbitrary letters and numbers that tokenized into multiple pieces.
• To simplify probability calculations, penalty codes and reasons were mapped to meaningful natural-language tokens, each represented by a single tokenizer token.
• KV caching was used between the penalty-code and penalty-reason predictions to improve efficiency.

The resulting approach combines cleaned moderation labels, lightweight decoder-model fine-tuning, structured output targets, and constrained token-level inference. It is intended to broaden automated OpenChat moderation while maintaining the accuracy and response speed required for real-time LINE operations.