Streamlining Security Investigations with Agents (opens in new tab)
Slack’s Security Engineering team uses a multi-agent AI system to investigate security alerts across billions of daily events. After finding that a single prompt produced inconsistent results, the team replaced it with a controlled workflow of specialized model invocations, structured outputs, and application-level orchestration. The resulting system improves consistency, enables evidence review, and allocates more capable models only where they add the most value. ## From Prompt Prototype to Controlled Workflow - The initial prototype consisted of a roughly 300-word prompt defining: - The analyst’s role - Available data sources - Investigation methodology - Report formatting - Response classifications - A stdio-based MCP server safely exposed selected security data sources to the model. - A coding-agent CLI served as the prototype execution environment. - Results varied significantly: - Sometimes the model cross-referenced evidence effectively. - Other times it reached convenient or unsupported conclusions too quickly. - Prompt refinements helped somewhat, but prompts were ultimately too limited for fine-grained process control. ## Structured Investigation Tasks - Slack decomposed the investigation into a sequence of model invocations. - Each invocation performs one well-defined task and returns a structured output. - The application chains these tasks together and passes only the necessary context between stages. - Structured outputs use JSON schemas to constrain model responses. - They improve predictability, but can still fail when schemas are too complex and remain vulnerable to hallucination or attempts to circumvent constraints. - Guidance such as “question your evidence” became an explicit workflow step rather than merely an instruction in a prompt. ## Persona-Based Agent Architecture - Slack drew inspiration from research on meta-prompting, multi-persona collaboration, and security tabletop exercises. - Instead of representing multiple personas within one model call, Slack implemented each persona as an independent model invocation. - Every agent/task pair has: - A defined responsibility - A carefully designed output structure - Specific prompts and instructions - Potentially different models and tools - The application orchestrates the agents and controls how knowledge moves through the investigation. ## The Investigation Loop - **Director agent** - Guides the investigation from beginning to end. - Forms questions for domain experts. - Uses a journaling tool to plan and organize progress. - Decides how to continue based on reviewed findings and timelines. - **Expert agents** - Investigate questions using specialized knowledge and data sources. - Slack currently uses four domains: - **Access:** Authentication, authorization, and perimeter services - **Cloud:** Infrastructure, compute, orchestration, and networking - **Code:** Source code and configuration management - **Threat:** Threat intelligence and analysis - **Critic agent** - Acts as a meta-expert reviewing domain findings. - Applies a defined rubric to assess quality. - Adds analysis and credibility scores to individual findings. - Helps identify reliable evidence and reduce hallucinations. - Returns its conclusions to the Director, closing the investigation loop. - The Critic’s mildly adversarial role provides an independent challenge to expert conclusions. ## The Knowledge Pyramid - Different stages use different model capabilities and costs. - Domain experts operate at the base: - They query complex data sources. - They may make many tool calls. - Processing their results can consume substantial tokens. - The Critic reviews the larger set of expert findings and identifies the most valuable or credible evidence. - Higher-level reasoning can therefore use a smaller, more focused context. - Because each stage is independent, Slack can select different model versions, prompts, tools, and output formats for different tasks. Slack’s main recommendation is to treat complex agent behavior as an explicitly orchestrated workflow rather than relying on one increasingly elaborate prompt. Specialized agents, structured outputs, independent critique, and tiered model usage provide more predictable investigations while preserving the flexibility of AI-assisted analysis.