Article · Original
Serving a model from more than one AWS Region is a well-established pattern for highly available inference: when one Region runs short of GPU capacity, requests move to another. The pattern is simple to describe and hard to operate. Until now, if you served models on Amazon SageMaker AI managed endpoints, cross-Region routing was a piece you still had to build yourself. Today, we’re introducing cross-Region inference routing in Amazon SageMaker AI. It uses a new resource, the inference router, that gives your application one invocation path across the endpoints and inference components you already run, in one or more AWS Regions and AWS accounts. You register the destinations that serve a model, and your application invokes the router instead of a specific endpoint. The router keeps requests in its own Region while the local destination is healthy, shifts new requests to another registered destination when it isn’t, and returns traffic gradually after recovery. In this post, we explain how cross-Region inference routing decides where each request runs, walk through an example with one model in two Regions, and cover monitoring, limits, and what you pay. For a cross-account deployment that serves several models, see How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI. When one Region runs short of capacity A SageMaker AI endpoint runs in one AWS Region and one account. When autoscaling can’t add instances of the type your endpoint needs, requests throttle and time out, even when the same model has spare capacity in another Region. To use that capacity, your application must detect a struggling endpoint and switch Regions, decide which errors are safe to retry without running the model twice, move traffic back after recovery without flapping, and manage access to endpoints in other accounts. That code sits on the critical path of every request, and every team serving models across Regions maintains its own version of it. How cross-Region inference routing works Cross-Region inference routing is based on a few core ideas: You decide where your model might run – For each model, you choose which endpoints and inference components the router might use. SageMaker AI decides which destination serves each request – The router prefers its own Region and shifts new requests to another destination only when the local one can’t serve reliably. Your endpoints stay the same – You deploy endpoints as you do today, with the same models, configurations, instance pools, and autoscaling. Direct InvokeEndpoint calls keep working, and the router passes InferenceId and CustomAttributes through, as InvokeEndpoint does. You can see every decision – Each real-time response tells you which destination served the request and why. A few terms describe the pieces: Inference router – A Regional SageMaker AI resource in your account that your application invokes. Routing target – A name for one logical model on a router, such as fraud-model. A router can hold many routing targets, and each fails over independently of the others. Destination – A real-time endpoint, inference component, or asynchronous endpoint registered under a routing target, identified by its full Amazon Resource Name (ARN), which tells the router its Region and account. A routing target holds either real-time destinations, which can mix endpoints and inference components across Regions and accounts, or asynchronous endpoints only. Local destination – A destination in the router’s Region, preferred while it’s healthy. Routing reason – A value returned with each real-time response that explains where the request went and why. The Region you create the router in is its local Region. To prefer a different Region, create a router there. The router doesn’t create capacity, scale your endpoints, deploy models, or copy model artifacts and container images. Each destination must already be running in its Region. Figure 1: Each request goes to a healthy destination, with the router preferring destinations in its own Region How routing decisions are made The router considers only destinations that are registered on the routing target and that it’s authorized to invoke. Among those, it prefers destinations in its own Region, and when more than one local destination is healthy, it spreads requests across them evenly. Requests leave the Region only when local destinations can’t serve reliably. To detect that, SageMaker AI calculates a health score for each destination from its recent requests, measured in the destination’s own Region. Server errors (5xx), throttles, and timeouts lower the score. Errors from the SageMaker AI platform (for example, failed health check) lower it the most. Errors caused by the request itself, such as invalid input, don’t lower it, because the same request would fail in any Region. Model latency isn’t an input, so an endpoint that keeps returning successful responses keeps its traffic. Signals such as failed instance launches or an endpoint running fewer instances than its autoscaling target make the router react faster, but only after the destination starts failing requests. The router evaluates each destination’s health and publishes it to Amazon CloudWatch as one of three states. A healthy destination serves normally. An overloaded destination still serves, and the router moves part of its traffic away. A degraded destination is failing or can’t serve, so the router moves most of its traffic away and keeps a small share to detect recovery. Figure 2: How each destination health state changes the traffic the router sends to it SageMaker AI manages failover thresholds and timing, so you don’t need to tune any thresholds. If health information for remote destinations goes stale, the router keeps traffic in its own Region, so a gap in monitoring never moves traffic on its own. When traffic moves, and when it comes back The more requests fail, the more traffic moves, so a destination need not fail outright first. Traffic starts to move within about a minute of sustained errors, or longer for a destination with little traffic. Traffic that leaves the local Region is spread across the healthy remote destinations in proportion to their health. You can’t set weights or a fallback order. After recovery, traffic returns over several minutes, more slowly than it left. Requests in progress aren’t interrupted. When a destination fails a real-time request without running the model, the router retries that request once on a different destination. This covers, for example, a throttle, a 503 error, a model that isn’t ready, a rejection by a cross-Region traffic cap, or access denied by the destination. The router never retries timeouts, 500 errors, or model errors. For asynchronous requests, the router tries each eligible destination once, local destinations first, until one accepts the request. Figure 3: The router retries a real-time request once on another destination, and only when the model didn’t run Figure 4 shows where requests were served in one of our tests with a test model. Figure 4: Where requests were served as the local endpoint was overloaded, a destination in another account lost and regained access, and the local endpoint recovered What you can route The following use cases show common ways to use cross-Region inference routing in Amazon SageMaker AI. Keep serving when one Region runs short of GPU capacity. When your G-series instance type runs short in US East (Ohio) during peak traffic and requests start to throttle, the router shifts new requests to the same model in US West (Oregon), where autoscaling absorbs the overflow, and moves traffic back gradually after US East (Ohio) recovers. Route across endpoints owned by separate AWS accounts. When a central platform account owns the router and other accounts own the endpoints, each endpoint owner grants access to one named router, and requests reach those endpoints only through that router. We cover the setup after this list. Serve many models from one router. Create one router with one routing target per model, each listing that model’s destinations, and pass the routing target name on each request. Each model fails over independently. Put the router behind your own gateway. If you operate an API or model gateway, you can point its SageMaker AI integration at the router, so SageMaker AI handles Region selection for SageMaker-hosted models while your gateway decides when to use a different provider. For example, see the Salesforce post. Route asynchronous requests. Asynchronous endpoints, in the router’s account or in other accounts, can be destinations. Your application calls InvokeInferenceRouterAsync with the routing target name and the Amazon Simple Storage Service (Amazon S3) input location, and gets back an InferenceId and the output and failure locations, as with InvokeEndpointAsync. Protect a receiving endpoint from other Regions’ overflow. On a real-time endpoint, you can set an optional cross-Region traffic cap so that requests from other Regions can use only part of its capacity. Your endpoint’s local callers keep the rest. We cover the details later in this section. When a destination belongs to a different account than the router, both accounts must allow each request. On the endpoint side, the owner attaches a resource-based policy that allows the router account to invoke the endpoint, and the sagemaker:SourceInferenceRouterArn condition key limits that access to one named router. The following policy allows only my-router in account 444455556666 to invoke the endpoint. Its second statement has no router condition and lets the router account read the endpoint’s status. Without that statement, the destination shows its status as unknown. { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowOneRouter", "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::444455556666:root"}, "Action": ["sagemaker:InvokeEndpoint", "sagemaker:InvokeEndpointAsync"], "Resource": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint", "Condition": {"ArnEquals": { "sagemaker:SourceInferenceRouterArn": "arn:aws:sagemaker:us-east-2:444455556666:inference-router/my-router"}} }, { "Sid": "AllowStatusVisibility", "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::444455556666:root"}, "Action": "sagemaker:DescribeEndpoint", "Resource": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint" } ] } In the router account, callers need sagemaker:InvokeInferenceRouter (or sagemaker:InvokeInferenceRouterAsync) on the router and sagemaker:InvokeEndpoint (or sagemaker:InvokeEndpointAsync) on every destination, because the router invokes each destination on the caller’s behalf. A router never gives a caller access to an endpoint they couldn’t call directly. For DescribeInferenceRouter to report each destination’s status, callers also need sagemaker:DescribeEndpoint on endpoints and sagemaker:DescribeInferenceComponent on inference components. Applications in other accounts call the router by assuming a role in the router account. For a walkthrough, see How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI and User Guide on cross-account destinations. Figure 5: Controls the endpoint owner keeps: an endpoint policy that allows one named router, and an optional cross-Region traffic cap Protect a receiving endpoint with a cross-Region traffic cap When the router moves requests away from a Region that’s short on capacity, those requests land on endpoints that already serve their own Region’s callers. By default, a receiving endpoint accepts any amount of cross-Region traffic, so a surge of overflow can push it into throttling. With a cross-Region traffic cap, the endpoint owner decides how much of the endpoint’s capacity other Regions might use. You set the cap in CreateEndpointConfig with CrossRegionTrafficConfig, which takes two required values. MaxInvocationsPerSecondPerInstance is the number of invocations per second that one instance can take, which you measure with a load test of your model, instance type, and typical payload. MaxCrossRegionPercentPerEndpoint is the share of that capacity that other Regions might use. SageMaker AI multiplies the per-instance rate by the instances in service and applies the percentage, so the cap follows autoscaling. For example, with 50 invocations per second per instance, a 20 percent cap, and 4 instances in service, the endpoint accepts up to 40 requests per second from other Regions. When autoscaling adds 2 instances, the cap rises to 60 requests per second. sm = boto3.client("sagemaker", region_name="us-west-2") sm.create_endpoint_config( EndpointConfigName="my-model-config-us-west-2", ProductionVariants=[...], # your existing variants, unchanged CrossRegionTrafficConfig={ "MaxInvocationsPerSecondPerInstance": 50, "MaxCrossRegionPercentPerEndpoint": 20, }, ) Only requests from other Regions count against the cap. Requests from the endpoint’s own Region, including direct InvokeEndpoint calls and calls from a router in the same Region, are never throttled by it. When cross-Region requests exceed the cap, the endpoint rejects the extra requests before your model runs, and the router retries each one once on a different destination. If no other destination can take the request, your application gets a throttling error. The cap applies to real-time endpoints, and all inference components on an endpoint share it. For an endpoint that hosts inference components, load test one instance with its usual mix of inference components and payloads to measure MaxInvocationsPerSecondPerInstance, and run the test again when that mix or your payloads change. If you leave out CrossRegionTrafficConfig, the endpoint accepts any amount of cross-Region traffic. Get started: One model in two Regions In this example, we deploy Qwen3-4B with the vLLM container as an inference component in US East (Ohio) and US West (Oregon), in one account. Because the inference router doesn’t support response streaming, the application sends non-streaming requests, and each request must complete within about a minute, the same limit as InvokeEndpoint. We then create a router in US East (Ohio) with one routing target that lists both: sm = boto3.client("sagemaker", region_name="us-east-2") rt = boto3.client("sagemaker-inference-router-runtime", region_name="us-east-2") sm.create_inference_router( InferenceRouterName="my-router", RoutingTargets=[{ "Name": "my-model", "Destinations": [ {"ResourceArn": "arn:aws:sagemaker:us-east-2:111122223333:inference-component/ic-us-east-2"}, {"ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:inference-component/ic-us-west-2"}, ], }], ) Run DescribeInferenceRouter and confirm that each destination’s RoutingStatus is Routable before you send traffic. The application invokes the router with the routing target name and the same payload it would send to InvokeEndpoint, and reads the serving destination and routing reason from the response: resp = rt.invoke_inference_router( InferenceRouterName="my-router", RoutingTargetName="my-model", ContentType="application/json", Body=json.dumps({"messages": [{"role": "user", "content": "Who are you?"}]}), ) print("Served by:", resp["InvokedRoutedResourceArn"]) print("Reason: ", resp["RoutingReason"]) What we observed Step RoutingStatus in DescribeInferenceRouter (us-east-2 / us-west-2) Served by Routing reason Both inference components running Routable / Routable us-east-2 local-preferred us-east-2 inference component removed Unroutable / Routable us-west-2 only-healthy us-east-2 inference component restored Routable / Routable us-west-2 only-healthy Traffic back in us-east-2 Routable / Routable us-east-2 local-preferred Traffic stayed in US East (Ohio) while its inference component ran, moved to US West (Oregon) with no change to the application when we removed it, and came back after we restored it. The complete notebook for this example is available on GitHub. Tip: Use instance pools and cross-Region inference routing together. With a capacity-aware instance pool, SageMaker AI can use another instance type in the same Region at creation and during scale-out. The router moves new requests to another Region when the local endpoint as a whole can’t serve. For details, see Capacity-aware inference: Automatic instance fallback for SageMaker AI endpoints. See where each request went The inference router gives you two views of routing: a record on every real-time response and metrics in CloudWatch. Per-request routing record Every real-time response includes the ARN of the destination that served the request in InvokedRoutedResourceArn, and the routing reason in RoutingReason. The routing reason is one of the following: local-preferred – Served by a healthy destination in the router’s Region. capacity-available – Served by a destination with available capacity. failover – Moved away from a destination that couldn’t serve reliably. only-healthy – Served by the only eligible destination that was healthy. We recommend logging both values with your application’s own request logs. Asynchronous responses don’t include the serving destination. Router metrics in CloudWatch show where asynchronous requests were placed. Router metrics in CloudWatch The router publishes OpenTelemetry metrics to CloudWatch in its own account and Region, labeled with the router, routing target, and destination: aws.sagemaker.inference_router.invocations – Requests per destination, with an error.type label on failures (invocation_4xx, invocation_5xx, or invocation_model_error). aws.sagemaker.inference_router.destination.duration – Time at the destination, including your model. aws.sagemaker.inference_router.overhead.duration – The router’s own work plus network time to and from another Region. aws.sagemaker.inference_router.destination.status – Each destination as healthy, overloaded, or degraded. Alarm on destinations that aren’t healthy and on the error rate across all error types. See Monitor an inference routers with Amazon CloudWatch and AWS CloudTrail. Figure 6: Requests per destination from the router’s CloudWatch invocations metric during the test shown in Figure 4 Each destination endpoint keeps publishing its existing SageMaker AI metrics in its own account and Region. To view them in one place, use CloudWatch cross-account observability and cross-account cross-Region dashboards. AWS CloudTrail records the router’s API calls (create, update, delete, describe, and list) as management events and router invocations as data events. You enable data events on your trail as you do for other SageMaker AI data events. Plan your deployment Keep the following in mind when you design with the inference router: Put the router in the Region where most of your traffic originates. The router always prefers destinations in its own Region. Size each Region for its own traffic plus the overflow you want it to absorb. By default, a receiving endpoint accepts any amount of cross-Region traffic, so configure autoscaling with headroom for the overflow you expect. To protect an endpoint’s local callers, set a cross-Region traffic cap, as described in Protect a receiving endpoint with a cross-Region traffic cap. If every remote destination reaches its cap, requests beyond the combined caps return throttling errors. Keep a low-cost standby Region. You can register an inference component in another Region and keep its cost low, but plan for how it gets capacity. A remote inference component with zero copies receives only a small share of traffic during failover, and those requests fail until it has copies. Keep at least one copy running, or configure autoscaling on copies and confirm that it scales out before you rely on the standby Region. Plan for resilience at the entry point. A router is regional, so if its Region is impaired, calls to it fail. Create routers in two Regions and switch between them on the client side. Register only destinations where your data might be processed. A request can be processed in the router’s Region or in any Region where you’ve registered a destination. For data that must stay in one geography, use a separate router, for example one for US data and one for EU data. You are responsible for configuring your routers to align with your data residency and compliance obligations. Keep your own retries modest. A throttling error means every destination was busy, and a service-unavailable error means no destination could take the request. A service-unavailable error after about a minute means the request timed out and the model might have run, so don’t retry it automatically. Remove deleted destinations from the router. Traffic already avoids a deleted endpoint, but the router never removes a destination on its own. Call the router privately. Applications in a virtual private cloud (VPC) can call the router through an interface VPC endpoint (AWS PrivateLink), which supports IPv4 (IPv6/dual-stack not supported). Also note the following limitations: Response streaming isn’t supported. A real-time request that the destination doesn’t respond to within about a minute fails with a service-unavailable error. Each router holds up to 20 routing targets, each routing target holds up to 10 destinations, and real-time payloads can be up to 6 MB. For default quotas, such as routers per account and request rates, and how to request an increase, see Service Quotas in the Amazon SageMaker AI documentation. Conclusion Cross-Region inference routing in Amazon SageMaker AI gives your application one invocation path across the endpoints and inference components you already run, in multiple AWS Regions and AWS accounts. The router prefers its own Region, shifts new requests when a destination can’t serve reliably, returns traffic gradually after recovery, and shows you where each request ran and why. Your endpoints, models, containers, and autoscaling policies don’t change. Cross-Region inference routing is generally available starting today in most commercial AWS Regions (except me-south-1 and me-central-1), for real-time and asynchronous inference. There is no additional charge for cross-Region inference routing in Amazon SageMaker AI. You pay for the SageMaker AI endpoints and inference components that serve your requests at standard SageMaker AI pricing, whether a request reaches them directly or through an inference router. The inference router publishes metrics to Amazon CloudWatch in the router’s account, and you pay standard CloudWatch rates for those metrics and any queries you run on them. For details, see Amazon SageMaker AI pricing. To get started, see the Amazon SageMaker AI documentation on cross-Region inference routing and explore the sample notebook on GitHub. About the authors Dmitry Soldatkin Dmitry is a Worldwide Leader for Specialist Solutions Architecture, Amazon SageMaker AI Inference at AWS. He helps customers design, build, and optimize generative AI and machine learning solutions, with a focus on inference performance optimization, deep learning, and scaling ML across the enterprise. In previous roles, he led teams at financial services, insurance, and telecommunications companies. Apoorva Chandra Apoorva is a Product Manager on the Amazon SageMaker AI Inference team, where she focuses on the endpoint experience and hosting experience. She works with customers deploying generative AI models in production to make accelerated compute easier to obtain, more cost-efficient to run, and more predictable to operate. Outside of work, Apoorva enjoys hiking, exploring coffee shops, and spending time with friends. Deepti Ragha Deepti is a Senior Software Development Engineer on the Amazon SageMaker AI team, specializing in ML inference infrastructure and model hosting optimization. She builds features that improve deployment performance, availability and reduce inference costs, and make ML accessible to organizations of all sizes. Outside of work, she enjoys traveling, hiking, and gardening. Felipe Lopez Felipe is a Principal AI/ML Specialist Solutions Architect at AWS. Prior to joining AWS, Felipe worked with GE Digital and SLB, where he focused on modeling and optimization products for industrial applications. Barkha Saxena Barkha is a Software Development Engineer at Amazon SageMaker AI Inference team, specializing in ML inference infrastructure and model serving optimization. Her work spans efficient multi-adapter inference, which serves hundreds of fine-tuned LoRA adapters from a single endpoint, and bidirectional streaming for low-latency, real-time applications such as voice agents. Outside work, she enjoys table tennis, painting, and puzzles.