online-safety

2 posts

discord

How ROOST is Advancing Online Safety (opens in new tab)

Discord argues that online safety should be built through shared, open-source infrastructure rather than isolated corporate systems. Its donated rules engine, Osprey, lets platforms detect suspicious behavior and harmful activity in real time, while ROOST develops and maintains tools for broad industry adoption. Early adoption, including by Bluesky, suggests this model can raise baseline safety standards across the internet. ## The Need for Shared Safety Tools - Nearly 100 million people use Discord daily, generating hundreds of millions of events that must be evaluated for threats. - Generative AI has increased the scale and sophistication of phishing, deepfakes, and coordinated abuse. - Smaller platforms often lack the resources to build effective trust-and-safety systems from scratch. - ROOST aims to make proven safety technologies open, shared, and auditable. ## How Osprey Works - Osprey is a real-time rules engine for event processing and behavioral analysis. - It can evaluate logins, messages, account creation, content posts, and platform-specific actions. - Safety teams write rules in a simple language and deploy them without engineering dependencies. - The engine produces transparent decisions indicating whether activity is safe, suspicious, or malicious. - Discord runs thousands of rules across hundreds of action types. - Investigation findings feed new rules, while enforcement generates additional signals for future detection. - The open-source release is based on Discord’s production system rather than a reduced version; improvements from ROOST were later reintegrated into Discord. ## ROOST’s Collaborative Model - ROOST builds on earlier cross-industry efforts such as image hashing for child-safety work, the Tech Coalition’s Lantern program, GIFCT incident response, and shared ISO safety standards. - Unlike organizations that primarily steward open-source projects, ROOST also develops and maintains a suite of public-interest safety tools. - Its projects include Osprey and Coop, a comprehensive review tool. - Open-source tools can raise the minimum level of protection available to smaller platforms and reduce the spread of threats across services. - The model also enables companies to build managed services around free tools, similar to businesses built around Linux. ## Adoption and Industry Impact - Musubi announced a managed Coop offering, while Zentropi integrated its labeling engine with Coop. - Osprey v1 was introduced at FOSDEM, prompting collaboration among engineers from multiple organizations and protocols. - Platforms such as Bluesky are already using Osprey. - More than 360 million users across participating platforms are now covered by open-source safety tooling. - ROOST continues development through public contributor and adopter working-group meetings held every two weeks. ROOST’s approach suggests that open, production-grade safety infrastructure can help platforms respond faster to emerging threats while creating a shared foundation for industry-wide improvement.

discord

Getting Global Age Assurance Right: What We Got Wrong and What's Changing (opens in new tab)

Discord’s CTO says the company mishandled communication around its global age-assurance rollout, leading users to believe that face scans or ID uploads would be required for everyone. Discord’s stated goal is to protect teens and restrict age-sensitive content while preserving the normal experience for most users and avoiding collection of users’ identities. The company plans to use internal signals for most age determinations and privacy-focused third-party verification only when necessary. ## Where Discord Says It Fell Short - Discord did not clearly explain how age assurance would work or who would be affected. - Many users mistakenly believed that everyone would need to submit a face scan or government ID. - The company acknowledges that skepticism about technology companies collecting personal data is justified. - Discord says the rollout was not intended to create a pretext for gathering more personal information. ## The Goal of Age-Appropriate Experiences - More than 90% of users are expected to continue using Discord without verifying their age. - Discord wants teenagers to receive stronger safeguards while allowing adults to access the full range of content. - The approach is being influenced by laws already taking effect in the UK and Australia, with Brazil, Europe, and several US states following. - Discord says building its own system could demonstrate that age can be verified without identifying users. ## Internal Age Determination - Discord’s systems may determine age using account-level signals, including: - Account age - Whether a payment method is attached - Server membership patterns - General account activity - The system will not read private messages, analyze conversations, or inspect posted content. - Discord compares this process to existing safety systems used to detect spam, raids, and coordinated abuse. - The company plans to publish its methodology before the global launch. ## What Happens If Verification Is Required - Fewer than 10% of users are expected to need additional verification. - Users who do not verify can keep: - Their accounts - Servers - Friends lists - Direct messages - Voice chat access - They will only lose access to age-restricted content and the ability to change certain default safety settings intended to protect teens. - A user’s age group will remain private and will not be visible to other Discord users. ## Third-Party Verification Partners - Discord plans to use vendors when its internal systems cannot confirm that a user is an adult. - Vendors are intended to return only an age group, not the user’s identity. - Discord says vendors cannot connect verification information back to a Discord account, and Discord cannot use the process to learn the user’s identity. - Partners undergo security and privacy reviews, contractual data-use restrictions, and retention and deletion requirements. - Verification data is generally deleted immediately after it is no longer needed. - Discord clarified that the vendor involved in a previous customer-service security incident is not used for age assurance and is no longer a partner. - The company tested Persona in the UK in January but decided not to proceed with that vendor. Discord’s revised approach emphasizes minimal data collection, multiple verification choices, and transparency about how age estimation works. Publishing the methodology and clearly explaining vendor safeguards will be essential to rebuilding user trust.