Document boto3 S3 client endpoint resolution behavior when accessing us-east-1 #4696
Description
Describe the issue
We ran into difficulty defining a strategy for accessing S3 from within an internet-isolated VPC. I think there's missing documentation
From an EC2 instance in us-east-1
s3 = boto3.client('s3',config=Config(s3={"us_east_1_regional_endpoint": "regional"}, region_name="us-east-1"))
print(s3.meta.endpoint_url)
s3 = boto3.client('s3',region_name='us-east-1')
print(s3.meta.endpoint_url)
s3 = boto3.client('s3',config=Config(s3={"us_east_1_regional_endpoint": "regional"}))
print(s3.meta.endpoint_url)
Yields:
https://s3.us-east-1.amazonaws.com
https://s3.amazonaws.com
https://s3.amazonaws.com
This matters within an internet-isolated VPC. For example, with this setup:
- VPC in us-east-1
- EC2 instance in VPC
- Gateway endpoint to us-east-1
- Interface endpoint to us-west-2 and eu-west-1
- Security group outbound permissions to the S3 prefix list, and to the regional endpoints
If I'm accessing a Bucket "mybucket" that exists in us-west-2 from within that Instance:
Doing either of these
boto3.client('s3').head_bucket('mybucket')
boto3.client('s3', region_name='us-east-1').head_bucket('mybucket')
boto3.client('s3',config=Config(s3={"us_east_1_regional_endpoint": "regional"})).head_bucket('mybucket')
Boto3 will use the endpoint:
https://mybucket.s3.amazonaws.com
This endpoint points to a public IP address within us-west-2, which is not accessible from within the VPC. So, I get a connection timeout.
Doing this:
boto3.client('s3',config=Config(s3={"us_east_1_regional_endpoint": "regional"}, region_name="us-east-1")).head_bucket('mybucket')
Resolves to:
https://mybucket.s3.us-east-1.amazonaws.com
Which is accessible from within the isolated VPC.
I think boto3 needs documentation that, in order to use the regional endpoint for us-east-1, you have to specify the region_name and the us_east_1_regional_endpoint configuration
Links
https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html