techlist.io
KREN
meta

RCCLX: Innovating GPU communications on AMD platforms (opens in new tab)

gpu-accelerationdistributed-computingllm-inferencercclxamd-rocmtorchcommstensor-parallelismlow-precision-collectives

RCCLX is Meta’s open-source enhancement of RCCL for AMD GPUs, integrated with Torchcomms to support portable distributed AI workloads. It introduces Direct Data Access (DDA) and low-precision collectives, targeting communication bottlenecks in inference and training. On AMD MI300X systems, these optimizations deliver lower latency and higher throughput while maintaining acceptable accuracy.

RCCLX and Torchcomms Integration

  • RCCLX is based on RCCL and tested on Meta’s internal workloads.
  • It integrates CTran transport technology for AMD platforms.
  • CTran enables features such as AllToAllvDynamic, a GPU-resident collective; additional CTran capabilities are planned for future releases.
  • Through Torchcomms, applications can use a common communication API across AMD, NVIDIA, and other backends without major code changes.
  • RCCLX is intended to achieve feature parity with Meta’s NCCLX backend for NVIDIA systems.

Direct Data Access for Intra-Node Collectives

  • LLM inference has two distinct phases:
    • Prefill is compute-bound and generates the model’s key-value cache.
    • Decoding is memory-bound and generates tokens incrementally.
  • Tensor parallelism can make AllReduce responsible for up to 30% of end-to-end latency.
  • RCCLX introduces two DDA algorithms:
    • DDA flat lets each rank directly read other ranks’ memory and perform local reductions. It reduces latency from O(N) to O(1) for small messages by increasing data exchange from O(n) to O(n²).
    • DDA tree divides AllReduce into reduce-scatter and all-gather phases, retaining ring-like data movement while reducing latency for somewhat larger messages.
  • On AMD MI300X GPUs, DDA improves over RCCL by:
    • 10–50% for decode workloads.
    • 10–30% for prefill workloads.
    • Approximately 10% lower time-to-incremental-token.

Low-Precision Collectives

  • RCCLX provides optimized low-precision versions of AllReduce, AllGather, AlltoAll, and ReduceScatter.
  • These target AMD Instinct MI300 and MI350 GPUs and support FP32 and BF16 inputs.
  • FP8 quantization provides up to 4:1 compression, reducing communication overhead for messages of at least 16 MB.
  • Parallel peer-to-peer mesh communication uses AMD Infinity Fabric for bandwidth and low latency.
  • Computation remains in FP32 to improve numerical stability.
  • Users can enable the feature with:
RCCL_LOW_PRECISION_ENABLE=1
  • Internal evaluations showed:
    • About a 0.3% change on GSM8K accuracy evaluations.
    • 9–10% lower latency.
    • Approximately 7% higher throughput.
  • The current implementation is tuned for single-node deployments.

Getting Started

  • Install Torchcomms with the RCCLX backend.
  • Create an RCCLX communicator through Torchcomms using the "rcclx" backend and a HIP device.
  • Existing Torchcomms operations such as allreduce can then run without backend-specific API changes.
  • Distributed initialization uses standard torchrun environment variables such as MASTER_ADDR, MASTER_PORT, RANK, and WORLD_SIZE.

RCCLX is positioned as a practical way to improve AMD-based AI training and inference without requiring applications to adopt a new communication API. Teams can use DDA for lower inference latency and selectively enable low-precision collectives for higher throughput, while evaluating numerical accuracy for their own workloads.