Deploys in ap-southeast-4 fail with transform error

[J-tt]

Describe the bug
Deploying to ap-southeast-4 fails with:

Fragment returned by transform AWS::Serverless-2016-10-31 is null. Rollback requested by user.

Steps To Reproduce
Steps to reproduce the behavior:

1. Follow the quick start guide in the readme (create config.json and deploy it via CLI to ap-southeast-4), or run this command:

aws cloudformation create-stack \
  --region ap-southeast-4 \
  --output text \
  --stack-name buildkite \
  --template-url "https://s3.amazonaws.com/buildkite-aws-stack/v6.28.0/aws-stack.yml" \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
  --parameters "[{\"ParameterKey\":\"BuildkiteAgentToken\",\"ParameterValue\":\"TokenGoesHere\"},{\"ParameterKey\":\"KeyName\",\"ParameterValue\":\"default\"},{\"ParameterKey\":\"ImageId\",\"ParameterValue\":\"\"}]"

2. Navigate to the CloudFormation dashboard
3. Deploy fails with above error

Expected behavior
Deployment succeeds

Actual behaviour
Deployment failed: "ROLLBACK_COMPLETE":

Fragment returned by transform AWS::Serverless-2016-10-31 is null. Rollback requested by user.

Stack parameters (please complete the following information):

• AWS Region: ap-southeast-4
• Version: latest (6.30.0)

Additional context
Not sure what resources are missing in ap-southeast-4 that causes this issue, quite unfamiliar with CloudFormation templates.

[DrJosh9000]

Thanks @J-tt, that's annoying. This is the first we've heard of this. It seems maybe that AWS::Serverless-2016-10-31 transform might not be functioning in ap-southeast-4. This could be because it is not fully supported there, or one of the services it interacts with. Have you raised this in a ticket with AWS?

[J-tt]

Hi @DrJosh9000 I haven't raised it with AWS, as I'm not 100% sure what the root cause is.

I'll look at filing this issue over at https://github.com/aws/serverless-application-model as well

[wolfeidau]

@J-tt I did a bit of digging and https://docs.aws.amazon.com/general/latest/gr/serverlessrepo.html which we use to distribute the autoscaler isn't supported in Melbourne yet. 😞

You can modify the stack to work around this but it is certainly not ideal, so we will raise it with AWS to add our 👍🏻 to getting it deployed there 🤞🏻 cc @DrJosh9000

[steven-geo]

Given this is still a problem ~12 months on and is an issue with other regions. Particularly as new regions come online, with the Serverless Application Repository seemingly a much lower priority for AWS to role out. Do we have some formal guidance on the workaround for this from Buildkite? I'm about to change over the template to work without it, and before I re:invent :) the wheel, if anyone has resolved this successfully and have a process?
Thanks
EDIT: Melbourne Region is now ~34 months old, so this gives us an idea of the delay for other new regions too.

[steven-geo]

OK. Here are the instructions to get around the missing SAR service that worked for me.

NOTE: The below assumes you have already cloned the https://github.com/buildkite/elastic-ci-stack-for-aws repo and are deploying it using that not a direct link.

1. Clone the buildkite-agent-scaler repo: https://github.com/buildkite/buildkite-agent-scaler

2. Run a 'make build' on buildkite-agent-scaler. NOTE: You will now end up with a handler.zip file in the root of the repo.

3. Upload the handler.zip from the root of the /buildkite-agent-scaler to an S3 bucket, it doesn't need to be publicly readable. Copy the Object URI (NOT URL) of the item. Must be in the format s3:////handler.zip

4. Edit the buildkite-agent-scaler repo template.yml file to add in the CoreUri path for handler.zip

Existing

  AutoscalingFunction:
    Type: AWS::Serverless::Function
    Properties:
      Description: !Sub Polls Buildkite and triggers scale out of elastic stack ${RootStackName}
      CodeUri: handler.zip

Change to

  AutoscalingFunction:
    Type: AWS::Serverless::Function
    Properties:
      Description: !Sub Polls Buildkite and triggers scale out of elastic stack ${RootStackName}
      CodeUri: https://<bucketname>.s3.<region>.amazonaws.com/objectprefix/handler.zip

5. Upload the template.yml to the same S3 bucket as you uploaded handler.zip to (doesn't have to be - but keep them together!). Copy the S3 URL (not URI) of this object.

6. Edit the /templates/aws-stack.yml file - in the 'Autoscaling' object change the following.

Existing

  Autoscaling:
    Type: AWS::Serverless::Application
    Condition: HasVariableSize
    Properties:
      Location:
        ApplicationId: !Ref BuildkiteAgentScalerServerlessARN
        SemanticVersion: !Ref BuildkiteAgentScalerVersion

Change to

  Autoscaling:
    Type: AWS::Serverless::Application
    Condition: HasVariableSize
    Properties:
      Location: <S3 Object URL for template.yml>

Save the file and deploy the stack

Enjoy.