get_products of pricing client silently changed its response format (regionCode field is missing) since Mar 24th

[YuzeLiao]

We are using the get_products API to describe RDS instance types. Previously we filter the entries by the instanceType field and the regionCode field. But recently we found that the previously worked API calls now return an empty result. We therefore removed the regionCode filter to see what might be wrong and found that the format of the response could be changed because the regionCode field is missing. Wondering if it's a bug or something else. Thanks!

The previously worked API call now returns an empty list

import boto3
client = boto3.client("pricing", region_name="us-east-1")
region = "us-east-2"
db_instance_class = "db.t2.small"
client.get_products(
        Filters=[
            {
                "Field": "regionCode",
                "Type": "TERM_MATCH",
                "Value": region,
            },
            {
                "Field": "instanceType",
                "Type": "TERM_MATCH",
                "Value": db_instance_class,
            },
        ],
        ServiceCode="AmazonRDS",
        FormatVersion="aws_v1",
        MaxResults=1,
    )

Response:

{'FormatVersion': 'aws_v1', 'PriceList': [], 'ResponseMetadata': {'RequestId': '2569afb7-7029-4159-ad15-02d73e2a355d', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '2569afb7-7029-4159-ad15-02d73e2a355d', 'content-type': 'application/x-amz-json-1.1', 'content-length': '41', 'date': 'Sat, 26 Mar 2022 14:49:02 GMT'}, 'RetryAttempts': 0}}

Removing the regionCode filter

import boto3
client = boto3.client("pricing", region_name="us-east-1")
db_instance_class = "db.t2.small"
client.get_products(
        Filters=[
            {
                "Field": "instanceType",
                "Type": "TERM_MATCH",
                "Value": db_instance_class,
            },
        ],
        ServiceCode="AmazonRDS",
        FormatVersion="aws_v1",
        MaxResults=1,
    )

Response:

{'FormatVersion': 'aws_v1', 'PriceList': ['{"product":{"productFamily":"Database Instance","attributes":{"engineCode":"16","instanceTypeFamily":"T2","memory":"2 GiB","vcpu":"1","instanceType":"db.t2.small","usagetype":"EUW3-InstanceUsage:db.t2.small","locationType":"AWS Region","storage":"EBS Only","normalizationSizeFactor":"1","instanceFamily":"General purpose","databaseEngine":"Aurora MySQL","processorFeatures":"Intel AVX; Intel Turbo","servicecode":"AmazonRDS","physicalProcessor":"Intel Xeon Family","clockSpeed":"Up to 3.3 GHz","licenseModel":"No license required","currentGeneration":"No","networkPerformance":"Low to Moderate","deploymentOption":"Single-AZ","location":"EU (Paris)","servicename":"Amazon Relational Database Service","processorArchitecture":"32-bit or 64-bit","operation":"CreateDBInstance:0016"},"sku":"227M9XKHRSNMBFJQ"},"serviceCode":"AmazonRDS","terms":{"OnDemand":{"227M9XKHRSNMBFJQ.JRTCKXETXF":{"priceDimensions":{"227M9XKHRSNMBFJQ.JRTCKXETXF.6YS6EN2CT7":{"unit":"Hrs","endRange":"Inf","description":"$0.046 per RDS db.t2.small Single-AZ instance hour (or partial hour) running Aurora MySQL","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.JRTCKXETXF.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0460000000"}}},"sku":"227M9XKHRSNMBFJQ","effectiveDate":"2022-03-01T00:00:00Z","offerTermCode":"JRTCKXETXF","termAttributes":{}}},"Reserved":{"227M9XKHRSNMBFJQ.6QCMYABX3D":{"priceDimensions":{"227M9XKHRSNMBFJQ.6QCMYABX3D.6YS6EN2CT7":{"unit":"Hrs","endRange":"Inf","description":"Aurora MySQL, db.t2.small reserved instance applied","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.6QCMYABX3D.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0000000000"}},"227M9XKHRSNMBFJQ.6QCMYABX3D.2TG2D8R56U":{"unit":"Quantity","description":"Upfront Fee","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.6QCMYABX3D.2TG2D8R56U","pricePerUnit":{"USD":"266"}}},"sku":"227M9XKHRSNMBFJQ","effectiveDate":"2017-09-30T23:59:59Z","offerTermCode":"6QCMYABX3D","termAttributes":{"LeaseContractLength":"1yr","OfferingClass":"standard","PurchaseOption":"All Upfront"}},"227M9XKHRSNMBFJQ.NQ3QZPMQV9":{"priceDimensions":{"227M9XKHRSNMBFJQ.NQ3QZPMQV9.6YS6EN2CT7":{"unit":"Hrs","endRange":"Inf","description":"Aurora MySQL, db.t2.small reserved instance applied","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.NQ3QZPMQV9.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0000000000"}},"227M9XKHRSNMBFJQ.NQ3QZPMQV9.2TG2D8R56U":{"unit":"Quantity","description":"Upfront Fee","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.NQ3QZPMQV9.2TG2D8R56U","pricePerUnit":{"USD":"559"}}},"sku":"227M9XKHRSNMBFJQ","effectiveDate":"2017-09-30T23:59:59Z","offerTermCode":"NQ3QZPMQV9","termAttributes":{"LeaseContractLength":"3yr","OfferingClass":"standard","PurchaseOption":"All Upfront"}},"227M9XKHRSNMBFJQ.38NPMPTW36":{"priceDimensions":{"227M9XKHRSNMBFJQ.38NPMPTW36.2TG2D8R56U":{"unit":"Quantity","description":"Upfront Fee","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.38NPMPTW36.2TG2D8R56U","pricePerUnit":{"USD":"285"}},"227M9XKHRSNMBFJQ.38NPMPTW36.6YS6EN2CT7":{"unit":"Hrs","endRange":"Inf","description":"Aurora MySQL, db.t2.small reserved instance applied","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.38NPMPTW36.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0110000000"}}},"sku":"227M9XKHRSNMBFJQ","effectiveDate":"2017-09-30T23:59:59Z","offerTermCode":"38NPMPTW36","termAttributes":{"LeaseContractLength":"3yr","OfferingClass":"standard","PurchaseOption":"Partial Upfront"}},"227M9XKHRSNMBFJQ.4NA7Y494T4":{"priceDimensions":{"227M9XKHRSNMBFJQ.4NA7Y494T4.6YS6EN2CT7":{"unit":"Hrs","endRange":"Inf","description":"Aurora MySQL, db.t2.small reserved instance applied","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.4NA7Y494T4.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0330000000"}}},"sku":"227M9XKHRSNMBFJQ","effectiveDate":"2017-09-30T23:59:59Z","offerTermCode":"4NA7Y494T4","termAttributes":{"LeaseContractLength":"1yr","OfferingClass":"standard","PurchaseOption":"No Upfront"}},"227M9XKHRSNMBFJQ.HU7G6KETJZ":{"priceDimensions":{"227M9XKHRSNMBFJQ.HU7G6KETJZ.2TG2D8R56U":{"unit":"Quantity","description":"Upfront Fee","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.HU7G6KETJZ.2TG2D8R56U","pricePerUnit":{"USD":"136"}},"227M9XKHRSNMBFJQ.HU7G6KETJZ.6YS6EN2CT7":{"unit":"Hrs","endRange":"Inf","description":"Aurora MySQL, db.t2.small reserved instance applied","appliesTo":[],"rateCode":"227M9XKHRSNMBFJQ.HU7G6KETJZ.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.0160000000"}}},"sku":"227M9XKHRSNMBFJQ","effectiveDate":"2017-09-30T23:59:59Z","offerTermCode":"HU7G6KETJZ","termAttributes":{"LeaseContractLength":"1yr","OfferingClass":"standard","PurchaseOption":"Partial Upfront"}}}},"version":"20220325051948","publicationDate":"2022-03-25T05:19:48Z"}'], 'NextToken': 'Slih0xsf4ruXJnqmnU9Ydw==:thakAxqu8HG3LRtLdDZoSo8G9tEZCx1Lz2/+IsmaBQXKJkaEdSpMRENOglxjq5vcNlq0xEMLVsdpD+PKQNwfb7bSEOr+DPFHkPspJY1pHumUQHKd7P3hRkzFMq25H00yJp/gYbRQS7zQEcrNZHIdmKzr6Mzrq60Amk8CGG1EWyqHsaJMehjywnIPTwpXyOyM', 'ResponseMetadata': {'RequestId': 'b761efaf-fde0-4bca-b4ad-f2a1b149259d', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'b761efaf-fde0-4bca-b4ad-f2a1b149259d', 'content-type': 'application/x-amz-json-1.1', 'content-length': '5548', 'date': 'Sat, 26 Mar 2022 14:53:47 GMT'}, 'RetryAttempts': 0}}

[aaoswal]

Hi @YuzeLiao,
Thank you submitting the issue.
On going through the response without using regionCode filter, there does not seem to be a regionCode parameter being returned as a part of the response for ServiceCode=AmazonRDS.

The response from the pricing client might vary for various services:
For example, ServiceCode=AmazonS3 returns the attribute regionCode as a part of the response among others whereas on a contrary services like AmazonEC2 or AmazonRDS do not.

get_products() filters support values which include ServiceCode, and all attribute names.
A good way to list supported attributes for specific services is by using describe_services() to get information specific to that service and then filtering the response using get_products().

If the query including regionCode used to run successfully before, could you please share the debug logs/ response of previous successful cases? This would help analyze and escalate the issue.

Hope this helps!

[YuzeLiao]

Hey @aaoswal,

Thank you for your response! Not sure if the pics below help. The API call did work fine before March 24

[aaoswal]

Hi @YuzeLiao,
Sorry, I am unable to understand the context from the screenshots provided.

Is there a way you could provide the debug logs for the script which worked before it started giving an error?

Also, heads up, I am moving this issue to GitHub discussions as a part of our transition to discussions.

[YuzeLiao]

Hi @aaoswal, the screenshot was from AWS CloudTrail. I shared it just in case you can track the API calls from the requestID / eventID as that did work on Mar 23rd. Other than that, I don't have a log to prove that it worked before. But it does not matter as we don't actually need to filter by region so we removed the filter of regionCode.

I selected your response as the answer. Thank you!

[AykCanDem]

I realized the same issue. I have a notebook running everyday. Before there was no problem but then regionCode is not available in the response. So, you cannot filter on regionCode. Instead of filtering on regionCode, try to filter on location. But, be careful, since the text is not as regionCode.
ex:
regionCode: eu-west-1
should be changed as
location:EU (Ireland)

check here for full location names: https://calculator.aws/#/createCalculator/EC2

[tim-finnigan]

I was contacted by a team member with more background on this issue, and they said there was a bug that caused the removal of the region code columns for services where the Price List files were published between March 24, 2022 and April 6, 2022. This issue should now be fixed but you may need to update to the latest version of boto3.