Developing a Model to Assess Harmfulness from Open Chat Names and Descriptions (opens in new tab)
The AI Services Lab developed a model to automatically detect harmful LINE OpenChat names and descriptions, reducing the need for manual review. The project improved an existing moderation system by cleaning inconsistent labels, selecting a lightweight safety-tuned decoder model, and adapting it to predict both penalty levels and reasons. Granite Guardian 3.1 2B was ultimately fine-tuned with LoRA and deployed using token-probability-based inference. ## OpenChat Monitoring - Users must provide an OpenChat name and may add a description. - Names and descriptions are reviewed whenever they are created or modified. - LINE processes a large volume of global OpenChats, making fully manual moderation impractical. - The project aimed to: - Expand automated moderation to countries requiring more detailed judgments. - Improve accuracy in regions already using automation. - Reduce the amount of content requiring human review. ## Data Cleansing - Training data consisted of previously manually reviewed OpenChat names and descriptions. - Only records reviewed under the current moderation guidelines were used. - Identical name-description pairs sometimes had conflicting penalty outcomes. - Labels were consolidated using these rules: - Select the most severe penalty if it appeared at least twice. - If it appeared only once, treat it as possible noise and select the second-most-severe penalty. - Choose the most frequent penalty reason. - If reasons were tied, choose the globally rarer reason, following a TF-IDF-like principle that rarer reasons may be more specific. - This process produced a single, consistent label for each identical input. ## Selecting the Pretrained Model The team evaluated models according to four requirements: - Decoder-based architecture. - Fine-tuned for safety moderation. - Approximately 2 billion parameters. - Apache license for commercial use. Granite Guardian 3.1 2B was selected because: - It is designed to classify harmfulness through the probabilities of “Yes” and “No” tokens. - Restricting predictions to predefined tokens avoids unpredictable free-form responses. - Token probabilities provide confidence scores that can be thresholded for operational needs. - Its relatively small size supports lower serving costs and faster responses. ## Fine-Tuning for Penalty Prediction - A simple harmful/not-harmful classification was insufficient because moderation decisions include different penalty levels and reasons. - The model was trained to produce structured responses containing: - An `Action` penalty code. - A `Reason` penalty reason. - Cross-entropy loss was calculated only over the assistant’s response tokens, not the entire prompt. - This focuses training on predicting moderation decisions rather than reproducing the input text. - LoRA was used instead of full-parameter fine-tuning: - The base model parameters remained frozen. - Only small trainable matrices representing parameter updates were optimized. - This reduced memory and training costs while preserving pretrained capabilities. ## Inference Design - During inference, the model calculates logits for all possible next tokens. - The system extracts only the logits corresponding to valid penalty-code tokens, converts them to probabilities, and selects the highest-scoring code. - It then predicts the penalty reason in a second step. - Existing operational codes consisted of arbitrary letters and numbers that tokenized into multiple pieces. - To simplify probability calculations, penalty codes and reasons were mapped to meaningful natural-language tokens, each represented by a single tokenizer token. - KV caching was used between the penalty-code and penalty-reason predictions to improve efficiency. The resulting approach combines cleaned moderation labels, lightweight decoder-model fine-tuning, structured output targets, and constrained token-level inference. It is intended to broaden automated OpenChat moderation while maintaining the accuracy and response speed required for real-time LINE operations.