ECS wait services-stable returns prematurely when leveraging --force-new-deployment rollout

[ohshazbot]

Describe the bug

When deploying leveraging --force-new-deployment to reuse existing task definition and then attempting to use ecs wait services-stable results in the wait returning before the rollout of the new software versions completes.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I would expect ecs wait services-stable to wait until the deployment completes (or times out) before returning

Current Behavior

ecs wait services-stable returns after a subset of instances are updated, resulting in the operation completing while a mix of old and new versions are operational

Reproduction Steps

aws ecs update-service --cluster ${ECS_CLUSTER} --service ${SERVICE_NAME} --force-new-deployment
aws ecs wait services-stable --cluster ${ECS_CLUSTER} --services ${SERVICE_NAME}

And observe (quickly) that the service is still mid-deployment

Possible Solution

No response

Additional Information/Context

No response

CLI version used

aws-cli/2.18.4

Environment details (OS name and version, etc.)

Python/3.12.7 Linux/6.8.0-51-generic source/x86_64.ubuntu.24

[RyanFitzSimmonsAK]

Hi @ohshazbot, thanks for reaching out. This looks like a service issue rather than an SDK issue, and I did find similar reports of inconsistent behavior with this waiter. I've reached out to the ECS service team about this issue. I'll let you know when I have any updates.

Ticket # for internal use : P203308568

[RyanFitzSimmonsAK]

The service team asked for additional detail to verify what's going on. They mentioned that there's some problematic behavior with this waiter for EXTERNAL and CODE_DEPLOY deployment controllers, are you using either of those? They also mentioned that the waiter might be timing out if the service is large (high desiredCount) or ECS tasks with a long startup. Are either of those applicable? Thanks!

[ohshazbot]

@RyanFitzSimmonsAK I'm using an ECS deploymentController and I'm dealing with less than 5 desiredCount.