Reduce memory usage from aws-partitions

[alextwoods]

The endpoints (partitions.json) loaded by aws-partitions account for a large amount of memory required when creating/using service clients and post Endpoints2.0, are required only for legacy features.

Benchmarking
code being benchmarked:

require 'aws-sdk-lambda'
client = Aws::Lambda::Client.new
client.list_functions; nil

• Before: memory retained: 8010 kb (69488 objects)
• After: memory retained: 3732 kb (30878 objects)

A decrease of more than 50%.

This PR does the following to reduce memory usage:

• Adds a new partitions metadata module that loads only the partition metadata.
• Change the aws_partitions Endpoints matcher to use the new, metadata only methods.
• Make client.config.endpoint set using an after_initailize and using the configured endpoint resolver

Additional endpoints changes:
Added a create class method to EndpointParameters which takes values from config. Operation endpoint parameter classes now are only generated when there is operation specific values and use the create method to handle setting values from config. Example generated code for S3:

# EndpointParameters#create
    def self.create(config, options={})
      new({
        region: config.region,
        use_fips: config.use_fips_endpoint,
        endpoint: config.regional_endpoint ? nil : config.endpoint.to_s,
        force_path_style: config.force_path_style,
        use_global_endpoint: config.s3_us_east_1_regional_endpoint == 'legacy',
        disable_multi_region_access_points: config.s3_disable_multiregion_access_points,
        use_arn_region: config.s3_use_arn_region,
        disable_s3_express_session_auth: config.disable_s3_express_session_auth,
      }.merge(options))
    end

# Endpoint module
    class ListObjectsV2
      def self.build(context)
        Aws::S3::EndpointParameters.create(
          context.config,
          bucket: context.params[:bucket],
          use_dual_stack: context[:use_dualstack_endpoint],
          accelerate: context[:use_accelerate_endpoint],
          prefix: context.params[:prefix],
        )
      end
    end

Alternatives
Instead of using define_singleton_method to override the endpoint method on the config struct, we could instead:

1. resolve with the new endpoints resolver from config (fallback to legacy if we can’t infer parameters). This is fairly complex code and requires us to make some assumptions about how parameters are bound (but reasonable within all existing AWS services). See demo implementation - this demo does NOT work for S3, but works for most other services.
2. (Slight breaking change) Use a simple templated/pattern resolver instead of the legacy resolver. Requires writing the basic template code, but is relatively simple. It will produce "incorrect" answers in some cases, but these are NOT used when making requests (but may be used by customers).
3. (Breaking change) Set endpoint to nil. Customers that were relying on client.config.endpoint may be broken.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

1. To make sure we include your contribution in the release notes, please make sure to add description entry for your changes in the "unreleased changes" section of the CHANGELOG.md file (at corresponding gem). For the description entry, please make sure it lives in one line and starts with Feature or Issue in the correct format.

2. For generated code changes, please checkout below instructions first:
   https://github.com/aws/aws-sdk-ruby/blob/version-3/CONTRIBUTING.md

Thank you for your contribution!

[mullermp]

Nice. I like this approach better. Just needs some clean up I think.

[mullermp]

There's a lot of nesting here. Can we break this up a bit with either methods and/or guard clauses?

[alextwoods]

yeah, just rescuing exceptions cleans this up a ton.

[github-actions]

Detected 1 possible performance regressions:

• aws-sdk-cloudwatchlogs.filter_log_events_large_ms - z-score regression: 24.96 -> 42.73. Z-score: 22.23

[HeyNonster]

We're wondering if context.config.simple_json needs to be considered here to conditionally generate camel_case or snake_case. cc: @bquorning

[bquorning]

I tried adding this patch:

diff --git i/build_tools/aws-sdk-code-generator/lib/aws-sdk-code-generator/endpoint_parameter.rb w/build_tools/aws-sdk-code-generator/lib/aws-sdk-code-generator/endpoint_parameter.rb
index 36d22c08d..aadeca202 100644
--- i/build_tools/aws-sdk-code-generator/lib/aws-sdk-code-generator/endpoint_parameter.rb
+++ w/build_tools/aws-sdk-code-generator/lib/aws-sdk-code-generator/endpoint_parameter.rb
@@ -174,7 +174,9 @@ module AwsSdkCodeGenerator
 
       return nil unless binding
 
-      "JMESPath.search(\"#{Underscore.underscore_jmespath(binding['path'])}\", context.params)"
+      camel_cased = binding['path']
+      snake_cased = Underscore.underscore_jmespath(binding['path'])
+      "JMESPath.search(context.config.simple_json ? \"#{camel_cased}\" : \"#{snake_cased}\", context.params)"
     end
 
     def static_context_param(operation, param_name)

Which produces the following diff (from c291968) for the generated code:

diff --git i/gems/aws-sdk-dynamodb/lib/aws-sdk-dynamodb/endpoints.rb w/gems/aws-sdk-dynamodb/lib/aws-sdk-dynamodb/endpoints.rb
index 0f9a6b7bb..41359a745 100644
--- i/gems/aws-sdk-dynamodb/lib/aws-sdk-dynamodb/endpoints.rb
+++ w/gems/aws-sdk-dynamodb/lib/aws-sdk-dynamodb/endpoints.rb
@@ -16,7 +16,7 @@ module Aws::DynamoDB
       def self.build(context)
         Aws::DynamoDB::EndpointParameters.create(
           context.config,
-          resource_arn_list: JMESPath.search("keys(request_items)", context.params),
+          resource_arn_list: JMESPath.search(context.config.simple_json ? "keys(RequestItems)" : "keys(request_items)", context.params),
         )
       end
     end
@@ -25,7 +25,7 @@ module Aws::DynamoDB
       def self.build(context)
         Aws::DynamoDB::EndpointParameters.create(
           context.config,
-          resource_arn_list: JMESPath.search("keys(request_items)", context.params),
+          resource_arn_list: JMESPath.search(context.config.simple_json ? "keys(RequestItems)" : "keys(request_items)", context.params),
         )
       end
     end
@@ -241,7 +241,7 @@ module Aws::DynamoDB
       def self.build(context)
         Aws::DynamoDB::EndpointParameters.create(
           context.config,
-          resource_arn: JMESPath.search("table_creation_parameters.table_name", context.params),
+          resource_arn: JMESPath.search(context.config.simple_json ? "TableCreationParameters.TableName" : "table_creation_parameters.table_name", context.params),
         )
       end
     end
@@ -358,7 +358,7 @@ module Aws::DynamoDB
       def self.build(context)
         Aws::DynamoDB::EndpointParameters.create(
           context.config,
-          resource_arn_list: JMESPath.search("transact_items[*].get.table_name", context.params),
+          resource_arn_list: JMESPath.search(context.config.simple_json ? "TransactItems[*].Get.TableName" : "transact_items[*].get.table_name", context.params),
         )
       end
     end
@@ -367,7 +367,7 @@ module Aws::DynamoDB
       def self.build(context)
         Aws::DynamoDB::EndpointParameters.create(
           context.config,
-          resource_arn_list: JMESPath.search("transact_items[*].[condition_check.table_name, put.table_name, delete.table_name, update.table_name][]", context.params),
+          resource_arn_list: JMESPath.search(context.config.simple_json ? "TransactItems[*].[ConditionCheck.TableName, Put.TableName, Delete.TableName, Update.TableName][]" : "transact_items[*].[condition_check.table_name, put.table_name, delete.table_name, update.table_name][]", context.params),
         )
       end
     end

This fixes a problem we’re seeing in an internal library (which uses simple_json: true and fails on the keys(request_items) expressions), but I am not sure if the other changes might break things for others.

[bquorning]

I should add that our problems started on v1.138.0 of the gem, after commit ce5d800.

[mullermp]

I'm just wondering what your use case is for simple json. I was not planning on supporting this for a new major version. If possible, I would move away from it. Can you also be specific about what bug you encountered and can you make a new issue?