Announcing Amazon SageMaker Inference for custom Amazon Nova models | Amazon Web Services

Amazon SageMaker Inference now generally supports deploying and scaling full-rank customized Amazon Nova models. The feature gives production workloads more control over instance types, autoscaling, context length, concurrency, and batch settings while improving cost efficiency through optimized GPU utilization. Customers can train Nova Micro, Nova Lite, and Nova 2 Lite models with SageMaker Training Jobs or HyperPod, then deploy them as managed real-time or asynchronous endpoints.

Custom Nova Model Support

• Supports customized Nova Micro, Nova Lite, and Nova 2 Lite models.
• Models can use:
  • Continued pre-training
  • Supervised fine-tuning
  • Reinforcement fine-tuning
• Custom models can be trained through Amazon SageMaker Training Jobs or Amazon HyperPod.
• SageMaker Inference provides managed deployment, scaling, and HTTPS access for production workloads.
• GPU utilization and inference costs can be optimized with Amazon EC2 G5 and G6 instances instead of relying exclusively on P5 instances.
• Autoscaling can respond to five-minute usage patterns.
• Configurable context length, concurrency, and batch size help balance latency, cost, and accuracy.

Deploying Through SageMaker Studio

• In SageMaker Studio, users select a trained Nova model from the Models menu.
• Choosing Deploy, SageMaker AI, and Create new endpoint starts deployment.
• Deployment settings include:
  • Endpoint name
  • Instance type
  • Initial and maximum instance counts
  • Permissions
  • Networking configuration
• Supported launch instance types vary by model:
  • Nova Micro: G5, G6, and P5 options, including g5.12xlarge through g6.48xlarge and p5.48xlarge
  • Nova Lite: g5.48xlarge, g6.48xlarge, and p5.48xlarge
  • Nova 2 Lite: p5.48xlarge
• Provisioning takes time because SageMaker must create infrastructure, download model artifacts, and initialize the inference container.
• Once the endpoint is InService, users can test it in the Studio Playground using chat prompts.

Deploying with the SageMaker SDK

• Deployment requires two SageMaker resources:
  • A model object referencing the Nova artifacts and inference container
  • An endpoint configuration specifying the instance type and count
• Model artifacts can be stored in Amazon S3 and referenced with an S3 prefix.
• Environment variables configure inference behavior, including:
  • CONTEXT_LENGTH
  • MAX_CONCURRENCY
  • DEFAULT_TEMPERATURE
  • DEFAULT_TOP_P
• The endpoint configuration creates a real-time endpoint, such as one using an ml.g5.12xlarge instance.
• SageMaker supports network isolation and execution roles for secure deployment.

Inference and Request Configuration

• Endpoints support synchronous real-time inference in streaming or non-streaming modes.
• Asynchronous endpoints are available for batch-style processing.
• Requests can configure:
  • Maximum output tokens
  • Temperature
  • Top-p and top-k sampling
  • Log probabilities
  • Streaming usage statistics
  • Reasoning effort, with low and high options
• The example request asks the model to compare quarterly spending against budget and identify variances above 10 percent.

SageMaker Inference provides a complete path from Nova customization to production deployment. Teams should select instance types and tune context length, concurrency, batching, and sampling parameters based on their workload’s latency, cost, and accuracy requirements.