Amazon SQS turns 20: Two decades of reliable messaging at scale | Amazon Web Services
Amazon SQS has spent two decades helping distributed systems communicate asynchronously without tightly coupling services. While its core purpose remains unchanged—decoupling producers and consumers, buffering traffic, and isolating failures—its scale, security, integrations, and workload support have expanded significantly. Recent improvements also make SQS suitable for high-throughput, multi-tenant, and AI-driven architectures. ## SQS’s Core Role in Distributed Systems - Producers place messages in queues and continue processing without waiting for consumers. - Consumers process messages when they are ready, preventing slow or unavailable services from causing cascading failures. - Customers use SQS to: - Decouple application components - Absorb traffic bursts - Improve resilience when individual services fail - Coordinate independent services and AI agents ## Higher Throughput for FIFO Queues - High-throughput FIFO mode launched in 2021 at up to 3,000 transactions per second per API action. - Capacity increased progressively to: - 6,000 TPS in 2022 - 9,000 TPS in 2023 - 18,000 TPS later in 2023 - Up to 70,000 TPS per API action in select Regions - The FIFO in-flight message limit grew from 20,000 to 120,000 in 2024, enabling more concurrent processing. ## Stronger Security and Access Controls - SSE-SQS launched in 2021, providing server-side encryption with AWS-managed keys and eliminating customer key-management requirements. - Encryption became the default for newly created queues in 2022. - Attribute-based access control was introduced in 2022, allowing permissions to be based on queue tags rather than static resource policies. ## Improved Message Recovery and Integration - Dead-letter queue redrive became available in the SQS console in 2021. - SDK and CLI APIs—including `StartMessageMoveTask`, `CancelMessageMoveTask`, and `ListMessageMoveTasks`—followed in 2023. - FIFO queue redrive support was added later that year. - JSON protocol support reduced processing latency by up to 23% for 5 KB payloads while lowering client CPU and memory use. - SQS queues can connect directly to EventBridge Pipes, enabling routing to many AWS services without custom integration code. ## Larger Messages and Fairer Queuing - The Extended Client Library for Python allows payloads up to 2 GB by storing message data in Amazon S3 and sending a reference through SQS. - In 2025, the native maximum message size increased from 256 KiB to 1 MiB for standard and FIFO queues. - Fair queues help prevent one tenant in a shared standard queue from delaying others. Producers provide a message group ID, while consumers require no changes. ## SQS for AI Workloads - SQS can buffer requests to large language models and regulate inference throughput. - Queues also help coordinate autonomous AI agents that operate as separate services. - These use cases apply the same established messaging model to more complex, distributed AI systems. Amazon SQS’s recommendation remains straightforward: use asynchronous queues when systems need loose coupling, burst management, and resilience. Its newer throughput, security, recovery, integration, and fairness features extend that pattern to larger and more demanding applications.
Read original(opens in new tab)