meta·Feb 24, 2026·3 min readCurated summary

RCCLX: Innovating GPU communications on AMD platforms

RCCLX is Meta’s open-source enhancement of RCCL for AMD GPUs, integrated with Torchcomms to support portable distributed AI workloads. It introduces Direct Data Access (DDA) and low-precision collectives, targeting communication bottlenecks in inference and training. On AMD MI300X systems, these optimizations deliver lower latency and higher throughput while maintaining acceptable accuracy. ## RCCLX and Torchcomms Integration - RCCLX is based on RCCL and tested on Meta’s internal workloads. - It integrates CTran transport technology for AMD platforms. - CTran enables features such as `AllToAllvDynamic`, a GPU-resident collective; additional CTran capabilities are planned for future releases. - Through Torchcomms, applications can use a common communication API across AMD, NVIDIA, and other backends without major code changes. - RCCLX is intended to achieve feature parity with Meta’s NCCLX backend for NVIDIA systems. ## Direct Data Access for Intra-Node Collectives - LLM inference has two distinct phases: - **Prefill** is compute-bound and generates the model’s key-value cache. - **Decoding** is memory-bound and generates tokens incrementally. - Tensor parallelism can make AllReduce responsible for up to 30% of end-to-end latency. - RCCLX introduces two DDA algorithms: - **DDA flat** lets each rank directly read other ranks’ memory and perform local reductions. It reduces latency from O(N) to O(1) for small messages by increasing data exchange from O(n) to O(n²). - **DDA tree** divides AllReduce into reduce-scatter and all-gather phases, retaining ring-like data movement while reducing latency for somewhat larger messages. - On AMD MI300X GPUs, DDA improves over RCCL by: - 10–50% for decode workloads. - 10–30% for prefill workloads. - Approximately 10% lower time-to-incremental-token. ## Low-Precision Collectives - RCCLX provides optimized low-precision versions of AllReduce, AllGather, AlltoAll, and ReduceScatter. - These target AMD Instinct MI300 and MI350 GPUs and support FP32 and BF16 inputs. - FP8 quantization provides up to 4:1 compression, reducing communication overhead for messages of at least 16 MB. - Parallel peer-to-peer mesh communication uses AMD Infinity Fabric for bandwidth and low latency. - Computation remains in FP32 to improve numerical stability. - Users can enable the feature with: ```bash RCCL_LOW_PRECISION_ENABLE=1 ``` - Internal evaluations showed: - About a 0.3% change on GSM8K accuracy evaluations. - 9–10% lower latency. - Approximately 7% higher throughput. - The current implementation is tuned for single-node deployments. ## Getting Started - Install Torchcomms with the RCCLX backend. - Create an RCCLX communicator through Torchcomms using the `"rcclx"` backend and a HIP device. - Existing Torchcomms operations such as `allreduce` can then run without backend-specific API changes. - Distributed initialization uses standard `torchrun` environment variables such as `MASTER_ADDR`, `MASTER_PORT`, `RANK`, and `WORLD_SIZE`. RCCLX is positioned as a practical way to improve AMD-based AI training and inference without requiring applications to adopt a new communication API. Teams can use DDA for lower inference latency and selectively enable low-precision collectives for higher throughput, while evaluating numerical accuracy for their own workloads.

Read original(opens in new tab)

Llm InferenceGpu AccelerationDistributed ComputingRcclx+4

google·Sep 10, 2025Original article

Speculative cascades — A hybrid approach for smarter, faster LLM inference (opens in new tab)

Speculative cascades represent a hybrid inference method that integrates the cost-efficiency of model cascades with the latency-reducing benefits of speculative decoding. By utilizing a smaller drafter model to generate token sequences that are verified in parallel by a larger expert model, this approach allows for high-speed generation while maintaining flexible quality standards. The result is a system that achieves superior cost-quality trade-offs and higher speed-ups than either traditional cascading or standard speculative decoding alone. ### Limitations of Cascades and Speculative Decoding * **Sequential Bottlenecks in Cascades:** Traditional cascades use a deferral rule to decide if a small model can handle a prompt. If the small model is not confident, the system waits for it to finish before starting the large model from scratch, wasting significant time. * **Strict Matching in Speculative Decoding:** This method requires the large model to verify the small model’s tokens. Even if the small model produces a factually correct and high-quality response, the large model will reject the entire draft if the tokens do not match its own preferred output exactly. * **Trade-off Divergence:** Cascades prioritize reducing computational costs but suffer from latency when deferring, while speculative decoding prioritizes speed but often performs redundant work because it mandates identical output to the larger model. ### The Speculative Cascades Mechanism * **Parallel Verification with Deferral:** Speculative cascades use the parallel processing of speculative decoding but introduce a flexible decision rule. The system can choose to accept the smaller model’s draft even if it differs from the larger model’s prediction, provided it meets a confidence threshold. * **Flexible Token Matching:** Unlike standard speculative decoding, which often relies on strict token-by-token matching, speculative cascades allow for "probabilistic matches" or quality-based acceptance to prevent unnecessary rejections. * **Resource Optimization:** By strategically deferring to the smaller model for certain segments of the generation, the system reduces the total work required from the expensive expert model without losing the speed of parallel execution. ### Empirical Results and Performance * **Model Testing:** The approach was validated using Gemma and T5 models across diverse language tasks, including reasoning, coding, translation, and question answering. * **Superior Trade-offs:** Testing showed that speculative cascades consistently outperformed baselines in cost-quality metrics, providing faster inference without the strict "all-or-nothing" quality constraints of speculative decoding. * **Task Versatility:** The hybrid method proved effective across both creative tasks (like summarization) and factual tasks (like math or coding), where different levels of "correctness" are acceptable. Speculative cascades offer a practical path for scaling LLM deployments by balancing the high cost of large models with the need for low-latency user experiences. Developers looking to optimize inference should consider this hybrid approach to capture the efficiency of small models while retaining the oversight of larger, more capable ones.

Llm InferenceAILarge Language ModelsGemma+5