An AWS Machine Learning blog post from October 23 walks through a deployment failure that hits teams running real-time SageMaker AI endpoints through CloudFormation or the AWS CDK. SageMaker requires the Application Auto Scaling scalable target to be deregistered before an endpoint configuration can change, but CloudFormation cannot track auto scaling resources it did not register. The post lists three distinct outcomes. Updating the endpoint while the target is still registered is rejected by SageMaker before CloudFormation reaches the auto scaling resources. Deregistering the target by hand and then changing only the endpoint configuration succeeds, but no auto scaling is re-registered, because CloudFormation sees unchanged auto scaling parameters and the endpoint runs without scaling and without any error. Deregistering by hand while also changing auto scaling parameters makes CloudFormation update a scalable target that no longer exists, and the rollback fails too, leaving the stack in a state that needs manual recovery. The proposed fix replaces the native scalable-target and scaling-policy resources with three custom resources that deregister before the endpoint update and re-register after it, ordered by explicit CDK dependencies, so both changes ship in one deployment.
Our takeThe middle failure mode is the dangerous one: a deployment that reports success while the endpoint quietly runs without scaling, which only surfaces under load. The workaround is sound, but it moves auto scaling out of the stack's model into custom resources, so the deregister and re-register handlers now own the state CloudFormation used to track. That deserves drift tests, and a plan for what happens when the middle custom resource times out with the endpoint half updated.