discordMar 17, 2026

How ROOST is Advancing Online Safety (opens in new tab)

Discord argues that online safety should be built through shared, open-source infrastructure rather than isolated corporate systems. Its donated rules engine, Osprey, lets platforms detect suspicious behavior and harmful activity in real time, while ROOST develops and maintains tools for broad industry adoption. Early adoption, including by Bluesky, suggests this model can raise baseline safety standards across the internet. ## The Need for Shared Safety Tools - Nearly 100 million people use Discord daily, generating hundreds of millions of events that must be evaluated for threats. - Generative AI has increased the scale and sophistication of phishing, deepfakes, and coordinated abuse. - Smaller platforms often lack the resources to build effective trust-and-safety systems from scratch. - ROOST aims to make proven safety technologies open, shared, and auditable. ## How Osprey Works - Osprey is a real-time rules engine for event processing and behavioral analysis. - It can evaluate logins, messages, account creation, content posts, and platform-specific actions. - Safety teams write rules in a simple language and deploy them without engineering dependencies. - The engine produces transparent decisions indicating whether activity is safe, suspicious, or malicious. - Discord runs thousands of rules across hundreds of action types. - Investigation findings feed new rules, while enforcement generates additional signals for future detection. - The open-source release is based on Discord’s production system rather than a reduced version; improvements from ROOST were later reintegrated into Discord. ## ROOST’s Collaborative Model - ROOST builds on earlier cross-industry efforts such as image hashing for child-safety work, the Tech Coalition’s Lantern program, GIFCT incident response, and shared ISO safety standards. - Unlike organizations that primarily steward open-source projects, ROOST also develops and maintains a suite of public-interest safety tools. - Its projects include Osprey and Coop, a comprehensive review tool. - Open-source tools can raise the minimum level of protection available to smaller platforms and reduce the spread of threats across services. - The model also enables companies to build managed services around free tools, similar to businesses built around Linux. ## Adoption and Industry Impact - Musubi announced a managed Coop offering, while Zentropi integrated its labeling engine with Coop. - Osprey v1 was introduced at FOSDEM, prompting collaboration among engineers from multiple organizations and protocols. - Platforms such as Bluesky are already using Osprey. - More than 360 million users across participating platforms are now covered by open-source safety tooling. - ROOST continues development through public contributor and adopter working-group meetings held every two weeks. ROOST’s approach suggests that open, production-grade safety infrastructure can help platforms respond faster to emerging threats while creating a shared foundation for industry-wide improvement.

trust-and-safetygen-aiopen-sourcebehavioral-analysis+4

discordFeb 18, 2026

Osprey: Open Sourcing our Rule Engine (opens in new tab)

Discord is open-sourcing Osprey, a rule engine designed to help platforms detect and respond to emerging safety threats in real time. Built with ROOST and internet.dev, it processes platform events, evaluates configurable rules, and produces actionable verdicts with minimal engineering effort. Osprey emphasizes scale, rapid rule deployment, transparency, extensibility, and continuous improvement. ## Goals for a Modern Rule Engine Osprey was designed around several requirements: - Process thousands of events per second in real time. - Let teams create and deploy expressive rules within minutes. - Return clear verdicts indicating whether activity is safe, suspicious, or malicious. - Explain how rules were executed and expose errors for investigation and debugging. - Support feedback loops that improve future detection rules. - Remain extensible enough to address new attack patterns. ## Osprey’s Processing Model Osprey accepts platform events called **Actions** through either: - Synchronous gRPC requests. - Asynchronous message queues. The engine evaluates these actions using rules written in SML, a Python-based rule language. Rules can use Python UDFs, Features, and Effects, while synchronous requests can return Verdict effects directly to callers. Outputs are sent to Apache Druid, which powers investigation and analysis tools. ## Actions Actions are JSON-like events submitted to Osprey. - Each action type has a unique name and schema. - Callers can customize the payload with relevant platform data. - Example data includes login attempts, user IDs, usernames, email addresses, and IP addresses. - Rules extract and evaluate values from these action payloads. ## Rules and SML Rules are the central mechanism for detecting suspicious behavior. - SML uses a Python-inspired syntax intended to be accessible to less-technical rule authors. - Rules can reference other rules and extracted data. - Static validation enforces consistent rule-writing practices. - Validation can be extended with Python, from naming conventions to more complex domain-specific checks. - Example rules identify a known spammer by email and apply a `spammer` label to the associated user entity. ## User-Defined Functions UDFs are regular Python functions that extend Osprey’s rule language and standard library. - Built-in capabilities such as `Rule`, `WhenRules`, and `JsonData` are implemented as UDFs. - Teams can add their own UDFs when integrating Osprey into other products. - UDFs can retrieve information from external services, including machine-learning models. - They can be configured for asynchronous execution and access external-service providers through the execution context. - A sample UDF obtains a link-spam score from an external prediction service. ## Features and Entities Features are globally named variables produced during Osprey executions. - Features are exported to Apache Druid for later querying and investigation. - Prefixing a variable name with `_` keeps it local instead of exporting it. - Examples include `UserId` and `UserEmail`, extracted from JSON action data. - Entities are a specialized type of Feature representing persistent objects such as users, servers, or email addresses. - Entities can receive effects such as labels, classifications, and signals. - Entity types determine which effects are valid through static validation. - The Osprey interface provides dedicated Entity Views for examining an entity’s history. ## Effects Effects are outcomes triggered when rules evaluate as true. - They are validated and processed in aggregate after execution. - Effects can modify or annotate entities with labels, classifications, or signals. - Verdict effects can be returned synchronously to inform the requesting service of a safety determination. Osprey’s open-source release gives platforms a reusable foundation for real-time trust and safety enforcement. Teams interested in adopting it can explore the repository at [github.com/roostorg/osprey](https://github.com/roostorg/osprey).

trust-and-safetypythongrpcapache-druid+3

pinterestDec 8, 2025

How Pinterest Built a Real‑Time Radar for Violative Content using AI (opens in new tab)

Pinterest built an AI-assisted prevalence measurement system to estimate how often users actually see policy-violating content, rather than relying only on user reports. The system samples daily impressions, uses production risk scores to improve efficiency, labels content with a multimodal LLM, and applies statistical reweighting to preserve unbiased estimates. This enables daily, segmented monitoring with substantially lower cost and latency than human-only review. ## Why Prevalence Matters - User reports miss important harms because: - Some sensitive issues, such as self-harm, are under-reported. - Users seeking harmful content may not report it. - Rare policy categories provide too few reports for reliable trend detection. - Human review of reports is expensive and slow. - Prevalence measures exposure: the share of total views directed to violating content. - This helps Pinterest identify under-reported harms, evaluate interventions, and detect changes earlier. - Human-only prevalence studies were previously conducted only about every six months and required multiple reviewers plus adjudication. ## What Pinterest Measures - Daily prevalence is calculated as: - **Views of content violating a policy ÷ total views** - For example, 10 violating views in a sample of 100,000 produces an estimated prevalence of 0.01%. - Results include 95% confidence intervals to communicate statistical precision. - Metrics can be segmented by: - Policy area, such as Adult Content, Self-harm, or Graphic Violence - Sub-policy, such as nudity versus explicit sexual content - Surface, including Homefeed, Search, and Related Pins - Content age, geography, and user-age groups where relevant ## Risk-Aware, Unbiased Sampling - Pinterest samples from the daily user-impressions stream. - Production enforcement risk scores are used to prioritize likely high-risk and high-exposure content, but they are not treated as labels or eligibility rules. - Missing scores are replaced with the day’s median so that new content remains eligible. - Weighted reservoir sampling approximates probability-proportional-to-size sampling, considering impressions and risk scores. - Inverse-probability weighting removes the bias introduced by risk-based sampling, ensuring estimates represent impressions rather than model thresholds. - Pinterest uses Hansen–Hurwitz ratio estimators for sampling with replacement and Horvitz–Thompson ratio estimators for sampling without replacement. - Pure random sampling is also available for validation studies. ## LLM-Based Labeling - A multimodal LLM analyzes sampled content using both images and text. - Prompts are reviewed by policy subject-matter experts and can return structured label hierarchies such as `safe`, `not_safe`, and `unsure`. - Each decision records: - The label and brief rationale - Policy version - Prompt and model identifiers - Token usage and run cost - Human validation is performed on strategically selected samples to identify edge cases and AI blind spots. - The LLM is tested against human-reviewed gold sets before launch and periodically afterward to detect drift. - The workflow is reportedly 15 times faster and far cheaper than human-only labeling while maintaining comparable decision quality and statistical governance. ## Production System and Monitoring - Inputs include entity-by-day engagement data such as impressions, clicks, hides, and reports, alongside current production risk scores. - The system stores prevalence estimates, sampling weights, labels, diagnostics, and lineage for audits. - Dashboards display: - Daily prevalence and 95% confidence intervals - Confidence-interval width and effective sample size - Sample positive rate - Risk-score distributions - Prompt, model, taxonomy, and metric versions - Teams can pivot results by policy, sub-policy, and surface. - Validation samples and run-health information help monitor both statistical quality and operational reliability. Pinterest’s approach combines probability sampling, inverse-probability estimation, and continuously calibrated multimodal AI labeling to create a daily radar for harmful exposure. The practical recommendation is to use AI to scale measurement, but retain rigorous sampling, human validation, confidence intervals, and full model and policy lineage so that faster estimates remain trustworthy.

trust-and-safetyllmmachine-learningmultimodal-ai+3

discordJul 20, 2025

ROOST Announces “Coop” and “Osprey”: Free, Open-Source Trust and Safety Infrastructure for the AI Era (opens in new tab)

ROOST, a non-profit dedicated to digital safety, has launched two open-source tools, Coop and Osprey, to provide enterprise-grade content moderation and threat investigation capabilities to organizations of all sizes. By open-sourcing technology previously developed by industry leaders like Discord and Cove, ROOST aims to democratize access to the infrastructure required to detect, triage, and respond to online harms. This initiative shifts Trust and Safety from a proprietary competitive advantage to a shared public resource, enabling platforms to prioritize user protection without the burden of expensive enterprise software. ### Content Review and Compliance with Coop Built on technology acquired from Cove and utilized by platforms like Notion, Coop focuses on the human-in-the-loop aspect of content moderation. * The platform provides robust tools for content review, allowing teams to route specific cases to subject-matter experts for deeper analysis. * It includes built-in integration with the National Center for Missing & Exploited Children’s (NCMEC) API, automating the mandatory reporting process for child sexual abuse material (CSAM). * The interface is designed to surface relevant context and metadata, ensuring moderators can make informed decisions and take immediate action against policy violations. ### Incident Response and Investigation with Osprey Osprey is a lightweight investigation tool originally developed by Discord to manage large-scale safety incidents and platform-wide threats. * It serves as a foundation for incident response, helping safety teams understand platform trends and investigate coordinated threats like phishing or harassment campaigns. * The tool is designed to be user-friendly and accessible for grassroots communities while remaining powerful enough for established platforms. * Early adopters, including the decentralized social network Bluesky, are implementing Osprey to demonstrate that effective safety infrastructure can be scalable and resource-efficient. ### A Collaborative Model for Safety Infrastructure The launch of these tools represents a strategic shift toward a collaborative "public-interest" model for digital defense. * ROOST acquired the intellectual property of Cove and received the donation of Osprey from Discord to ensure these tools remain available as a public good. * The initiative is backed by philanthropic funding and legal support from Perkins Coie, removing the financial barriers that often prevent smaller platforms from implementing high-level safety measures. * Major industry players like Notion and Bluesky are championing the move, signaling an industry-wide push to share safety innovations rather than silo them. Platforms and developers should prepare to integrate these tools into their safety stacks as they become publicly available in the coming months. By adopting open-source infrastructure for routine tasks like NCMEC reporting and incident triage, organizations can focus their internal resources on platform-specific innovations while maintaining a high standard of digital safety.

trust-and-safetyopen-sourceapi-integrationincident-response+3