Slack AI: The Path to Multi-Cloud (opens in new tab)

Slack’s AI infrastructure evolved from self-managed SageMaker deployments to managed Amazon Bedrock as Slack sought enterprise-grade security, reliability, lower operational overhead, and faster access to new models. SageMaker provided strong isolation and compliance but required extensive capacity and regional management, while Bedrock simplified operations through managed throughput and on-demand options. Slack’s carefully staged migration achieved zero customer-facing incidents and established a practice of measuring parity, shifting traffic gradually, and monitoring continuously.

Phase 1: The SageMaker Era

• SageMaker offered:
  • Security and FedRAMP compliance
  • Control over model availability
  • An escrow VPC strategy that kept Slack data private while preventing access to providers’ model weights
• Slack deployed model containers across multiple AWS regions to support global availability.
• Engineering teams had to manage:
  • Cross-region IAM roles
  • Routing across model endpoints
  • Capacity planning
  • Auto-scaling
• The main operational challenges were:
  • Scaling latency: New instances could not start instantly.
  • GPU scarcity: A100 and H100 capacity was often unavailable.
  • Over-provisioning: Idle resources had to be maintained for peak demand.
• On-Demand Capacity Reservations and scheduled scaling reduced some problems, but required substantial manual coordination.
• SageMaker also created model feature lag because AWS prioritized releasing newer Anthropic models and optimizations through Bedrock.

Phase 2: Migrating to Amazon Bedrock

By mid-2024, Bedrock had matured enough to meet Slack’s security and FedRAMP requirements.

• The migration provided:
  • Managed infrastructure instead of individual GPU instances
  • Faster access to newly released LLMs
  • Flexible capacity options for different workload patterns
• Slack used:
  • Provisioned Throughput (PT) for predictable, latency-sensitive features such as channel summaries
  • On Demand (OD) for bursty scheduled workloads such as Recap
• Bedrock measured capacity in Model Units (MUs), letting Slack plan around token throughput rather than hardware details.

Executing a Zero-Incident Migration

Slack used a gradual, validation-heavy process:

• Obtained Legal, Security, and FedRAMP approval before moving production traffic.
• Performed load testing to determine the number of MUs needed to match SageMaker capacity.
• Compared model quality and latency through A/B tests and evaluation frameworks.
• Used feature flags for incremental traffic shifting and immediate rollback.
• Ran shadow requests and extensive tests to verify behavioral and performance parity.

This approach allowed Slack to move live production traffic without customer-facing incidents.

Operational Improvements and Remaining Gaps

• Bedrock reduced infrastructure maintenance, allowing engineers to focus more on model quality and product features.
• Slack could adopt new models weeks or months earlier, improving features such as AI Search with higher-reasoning models.
• Capacity planning shifted from reactive scaling to forecasting demand several weeks ahead.
• Provisioned Throughput still required maintaining a high baseline of MUs to handle large regional workday surges, creating an over-provisioning challenge for traffic that varies significantly throughout the day.

Slack’s migration demonstrates that managed AI infrastructure can improve agility and reliability, but success depends on careful capacity modeling, comprehensive parity testing, gradual rollout, and continuous monitoring.