Refactor NAT Instance AMIs for Deployment

[rajadain]

These lines consistently fail during deployment:

model-my-watershed/deployment/cfn/vpc.py

Lines 106 to 109 in 69c7252

	filters = {'name': '*ami-vpc-nat-hvm*'}
	nat_ami_id = get_recent_ami(self.aws_profile, filters=filters,
	region=self.region, owner='amazon')

http://civicci01.internal.azavea.com/view/mmw/job/model-my-watershed-staging-deployment/1512/console

python3 mmw_stack.py launch-stacks --aws-profile mmw-stg --mmw-profile staging --mmw-config-path /var/lib/jenkins/.aws/staging.yml --stack-color blue --activate-dns
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/model-my-watershed-staging-deployment/deployment/cfn/vpc.py", line 104, in get_recent_nat_ami
    nat_ami_id = self.get_input('NATInstanceAMI')
  File "/var/lib/jenkins/.local/lib/python3.5/site-packages/majorkirby/majorkirby.py", line 187, in get_input
    raise MKUnresolvableInputError
majorkirby.majorkirby.MKUnresolvableInputError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mmw_stack.py", line 105, in <module>
    main()
  File "mmw_stack.py", line 102, in main
    args.func(mmw_config=mmw_config, **vars(args))
  File "mmw_stack.py", line 16, in launch_stacks
    build_stacks(mmw_config, aws_profile, **kwargs)
  File "/var/lib/jenkins/workspace/model-my-watershed-staging-deployment/deployment/cfn/stacks.py", line 104, in build_stacks
    s3_vpc_endpoint_graph.go()
  File "/var/lib/jenkins/.local/lib/python3.5/site-packages/majorkirby/majorkirby.py", line 277, in go
    self.heartbeat(0)
  File "/var/lib/jenkins/.local/lib/python3.5/site-packages/majorkirby/majorkirby.py", line 386, in heartbeat
    self.input_connections[required].heartbeat(heartbeat_id)
  File "/var/lib/jenkins/.local/lib/python3.5/site-packages/majorkirby/majorkirby.py", line 399, in heartbeat
    self._launch_cfn()
  File "/var/lib/jenkins/.local/lib/python3.5/site-packages/majorkirby/majorkirby.py", line 318, in _launch_cfn
    self.set_up_stack()
  File "/var/lib/jenkins/workspace/model-my-watershed-staging-deployment/deployment/cfn/vpc.py", line 83, in set_up_stack
    'NATInstanceAMI', Type='String', Default=self.get_recent_nat_ami(),
  File "/var/lib/jenkins/workspace/model-my-watershed-staging-deployment/deployment/cfn/vpc.py", line 109, in get_recent_nat_ami
    region=self.region, owner='amazon')
  File "/var/lib/jenkins/workspace/model-my-watershed-staging-deployment/deployment/cfn/utils/cfn.py", line 83, in get_recent_ami
    return sorted(images, key=lambda i: i.creationDate, reverse=True)[0].id
IndexError: list index out of range

This is the output of the equivalent aws-cli command:

aws --profile=mmw-stg ec2 describe-images --filters 'Name=name, Values=*ami-vpc-nat-hvm*' --owners amazon

{
    "Images": []
}

No images.

If we include deprecated images, we get 36!

aws --profile=mmw-stg ec2 describe-images --filters 'Name=name, Values=*ami-vpc-nat-hvm*' --owners amazon --include-deprecated | jq '.Images | length'

36

They were all deprecated at the same time, August 28 2022:

aws --profile=mmw-stg ec2 describe-images --filters 'Name=name, Values=*ami-vpc-nat-hvm*' --owners amazon --include-deprecated | jq '.Images[].DeprecationTime' | uniq

"2022-08-28T23:59:59.000Z"

A workaround has been applied by adding this line to the deployment configuration YAML file:

NATInstanceAMI: 'ami-00a9d4a05375b2763'

Where the AMI id came from this command:

aws --profile=mmw-stg ec2 describe-images --filters 'Name=name, Values=*ami-vpc-nat-hvm*' --owners amazon --include-deprecated | jq '.Images |= sort_by(.CreationDate) | .[] | last | .ImageId'

However, since the AMI is already deprecated, it may be pulled at some point.

Before that happens, we should refactor our deployment pipeline to move away from NAT and to something better supported.