tokenization

Developing a Model to Assess Harmfulness from Open Chat Names and Descriptions (opens in new tab)

The AI Services Lab developed a model to automatically detect harmful LINE OpenChat names and descriptions, reducing the need for manual review. The project improved an existing moderation system by cleaning inconsistent labels, selecting a lightweight safety-tuned decoder model, and adapting it to predict both penalty levels and reasons. Granite Guardian 3.1 2B was ultimately fine-tuned with LoRA and deployed using token-probability-based inference. ## OpenChat Monitoring - Users must provide an OpenChat name and may add a description. - Names and descriptions are reviewed whenever they are created or modified. - LINE processes a large volume of global OpenChats, making fully manual moderation impractical. - The project aimed to: - Expand automated moderation to countries requiring more detailed judgments. - Improve accuracy in regions already using automation. - Reduce the amount of content requiring human review. ## Data Cleansing - Training data consisted of previously manually reviewed OpenChat names and descriptions. - Only records reviewed under the current moderation guidelines were used. - Identical name-description pairs sometimes had conflicting penalty outcomes. - Labels were consolidated using these rules: - Select the most severe penalty if it appeared at least twice. - If it appeared only once, treat it as possible noise and select the second-most-severe penalty. - Choose the most frequent penalty reason. - If reasons were tied, choose the globally rarer reason, following a TF-IDF-like principle that rarer reasons may be more specific. - This process produced a single, consistent label for each identical input. ## Selecting the Pretrained Model The team evaluated models according to four requirements: - Decoder-based architecture. - Fine-tuned for safety moderation. - Approximately 2 billion parameters. - Apache license for commercial use. Granite Guardian 3.1 2B was selected because: - It is designed to classify harmfulness through the probabilities of “Yes” and “No” tokens. - Restricting predictions to predefined tokens avoids unpredictable free-form responses. - Token probabilities provide confidence scores that can be thresholded for operational needs. - Its relatively small size supports lower serving costs and faster responses. ## Fine-Tuning for Penalty Prediction - A simple harmful/not-harmful classification was insufficient because moderation decisions include different penalty levels and reasons. - The model was trained to produce structured responses containing: - An `Action` penalty code. - A `Reason` penalty reason. - Cross-entropy loss was calculated only over the assistant’s response tokens, not the entire prompt. - This focuses training on predicting moderation decisions rather than reproducing the input text. - LoRA was used instead of full-parameter fine-tuning: - The base model parameters remained frozen. - Only small trainable matrices representing parameter updates were optimized. - This reduced memory and training costs while preserving pretrained capabilities. ## Inference Design - During inference, the model calculates logits for all possible next tokens. - The system extracts only the logits corresponding to valid penalty-code tokens, converts them to probabilities, and selects the highest-scoring code. - It then predicts the penalty reason in a second step. - Existing operational codes consisted of arbitrary letters and numbers that tokenized into multiple pieces. - To simplify probability calculations, penalty codes and reasons were mapped to meaningful natural-language tokens, each represented by a single tokenizer token. - KV caching was used between the penalty-code and penalty-reason predictions to improve efficiency. The resulting approach combines cleaned moderation labels, lightweight decoder-model fine-tuning, structured output targets, and constrained token-level inference. It is intended to broaden automated OpenChat moderation while maintaining the accuracy and response speed required for real-time LINE operations.

Designing a Semantic Context OS: Beyond Token Stuffing in Agent Systems (opens in new tab)

The article argues that larger LLM context windows do not automatically produce better software-engineering agents. In long-running workflows, indiscriminately filling the context window can cause attention dilution, context rot, reasoning failures, and potential data exposure. It proposes a “Semantic Context OS,” a local runtime layer that actively governs context as a finite, structured system resource rather than treating it as an unmanaged text stream. ## The Context Window Is Not RAM - The article uses the “Karpathy metaphor”: - The LLM acts like a CPU: a largely stateless inference engine driven by pretrained parameters. - The context window acts like RAM: volatile working memory containing current state, instructions, telemetry, and runtime data. - Unlike physical RAM, LLM context is probabilistic rather than deterministic: - Traditional RAM provides precise address-based retrieval with predictable performance. - LLM retrieval depends on attention weights across Q, K, and V matrices. - Increasing capacity from 32K tokens to 1M or 2M tokens therefore does not guarantee proportionally better retrieval. Larger sequences also increase computational cost and structural noise. ## Attention Dilution and Long-Context Failure - Large codebases and logs contain substantial irrelevant material, including: - Boilerplate definitions - Unused imports - Duplicate syntax - Repeated utilities and naming patterns - As sequence length grows, the attention calculation `QKᵀ` accumulates entropy and background noise. - Softmax then spreads attention energy across more tokens, weakening the sharp attention peaks needed to retrieve important facts. - This contributes to the “lost in the middle” effect: - Information near the beginning and end of a prompt is often retrieved more reliably. - Retrieval accuracy can fall sharply across the middle portion of the context. - The article considers relying on massive, unmanaged contexts an architectural anti-pattern for tasks such as large-scale code review, dependency tracing, and automated refactoring. ## Context Rot in Long-Running Agents The article defines “context rot” as the degradation of an agent’s working context during extended autonomous tasks. - **Context poisoning** - Raw logs, obsolete errors, and previous execution data accumulate over multiple turns. - The model may treat temporary historical failures as current architectural constraints. - **Context distraction** - Monorepos often contain similar names, overloaded methods, and duplicated helper code. - Broad retrieval can overwhelm the model with structurally similar but logically irrelevant code. - **Context clash** - Old instructions may remain after the plan has evolved. - Contradictory directives can cause indecision, infinite reasoning loops, timeouts, or hallucinations. - The article claims that, without active management, failure rates increase nonlinearly with context depth and may reach roughly 40% in deeply nested codebases. ## Semantic Context OS as an AI Kernel The proposed Semantic Context OS sits between agent application logic and external foundation-model APIs, operating as a localhost loopback proxy at `localhost:8080`. Its responsibilities include: - Treating context as a finite hardware-like resource. - Tracking token lifecycles and state access. - Filtering and isolating data before it reaches the model. - Separating physical token limits from semantic governance. - Protecting downstream inference engines from structural noise and helping prevent intellectual-property leakage. The architecture includes: - A POSIX-like virtual file system for managing state topology. - A proprietary “PathAlign” stage for AST-based code-tree pruning. - An asynchronous “sawtooth” memory model for runtime token optimization. ## MVC: Minimum Viable Context The core MVC pipeline—described as “minimum viable context”—aims to provide only the smallest dense set of information required for the agent’s current reasoning step. Its processing stages include: - **Collection and token mapping** - Gather source files, dependency graphs, and runtime logs. - Map them using the target model’s tokenizer, such as `cl100k_base` or `o200k_base`. - **Structural pruning** - Use static analysis and structural rules to remove compiler comments, unused imports, boilerplate, and unrelated utilities. - The broader design replaces passive string concatenation with active context selection, lifecycle management, and bounded transmission policies. The article concludes that reliable enterprise agents require active context orchestration rather than larger prompts alone. A dedicated governance layer should prune, isolate, and refresh context throughout execution so that models receive minimal, relevant, and internally consistent information.