Osprey: Open Sourcing our Rule Engine (opens in new tab)
Discord is open-sourcing Osprey, a rule engine designed to help platforms detect and respond to emerging safety threats in real time. Built with ROOST and internet.dev, it processes platform events, evaluates configurable rules, and produces actionable verdicts with minimal engineering effort. Osprey emphasizes scale, rapid rule deployment, transparency, extensibility, and continuous improvement. ## Goals for a Modern Rule Engine Osprey was designed around several requirements: - Process thousands of events per second in real time. - Let teams create and deploy expressive rules within minutes. - Return clear verdicts indicating whether activity is safe, suspicious, or malicious. - Explain how rules were executed and expose errors for investigation and debugging. - Support feedback loops that improve future detection rules. - Remain extensible enough to address new attack patterns. ## Osprey’s Processing Model Osprey accepts platform events called **Actions** through either: - Synchronous gRPC requests. - Asynchronous message queues. The engine evaluates these actions using rules written in SML, a Python-based rule language. Rules can use Python UDFs, Features, and Effects, while synchronous requests can return Verdict effects directly to callers. Outputs are sent to Apache Druid, which powers investigation and analysis tools. ## Actions Actions are JSON-like events submitted to Osprey. - Each action type has a unique name and schema. - Callers can customize the payload with relevant platform data. - Example data includes login attempts, user IDs, usernames, email addresses, and IP addresses. - Rules extract and evaluate values from these action payloads. ## Rules and SML Rules are the central mechanism for detecting suspicious behavior. - SML uses a Python-inspired syntax intended to be accessible to less-technical rule authors. - Rules can reference other rules and extracted data. - Static validation enforces consistent rule-writing practices. - Validation can be extended with Python, from naming conventions to more complex domain-specific checks. - Example rules identify a known spammer by email and apply a `spammer` label to the associated user entity. ## User-Defined Functions UDFs are regular Python functions that extend Osprey’s rule language and standard library. - Built-in capabilities such as `Rule`, `WhenRules`, and `JsonData` are implemented as UDFs. - Teams can add their own UDFs when integrating Osprey into other products. - UDFs can retrieve information from external services, including machine-learning models. - They can be configured for asynchronous execution and access external-service providers through the execution context. - A sample UDF obtains a link-spam score from an external prediction service. ## Features and Entities Features are globally named variables produced during Osprey executions. - Features are exported to Apache Druid for later querying and investigation. - Prefixing a variable name with `_` keeps it local instead of exporting it. - Examples include `UserId` and `UserEmail`, extracted from JSON action data. - Entities are a specialized type of Feature representing persistent objects such as users, servers, or email addresses. - Entities can receive effects such as labels, classifications, and signals. - Entity types determine which effects are valid through static validation. - The Osprey interface provides dedicated Entity Views for examining an entity’s history. ## Effects Effects are outcomes triggered when rules evaluate as true. - They are validated and processed in aggregate after execution. - Effects can modify or annotate entities with labels, classifications, or signals. - Verdict effects can be returned synchronously to inform the requesting service of a safety determination. Osprey’s open-source release gives platforms a reusable foundation for real-time trust and safety enforcement. Teams interested in adopting it can explore the repository at [github.com/roostorg/osprey](https://github.com/roostorg/osprey).