describe cluster for redshift returns mismatch value

[maybo65]

Hey,
Can you please help with the following issue:
I'm trying to get the size of a cluster using the describe_clusters, but getting mismatch results compare to the UI.
In the UI I can see that the size of my cluster is 320gb, BUT when Im using boto3 to retrive it, I'm getting:
TotalStorageCapacityInMegaBytes=800000.

The nodes are from type dc2.large and there's two nodes in the cluster.
Thank you

[aBurmeseDev]

Hi @maybo65 - thanks for reaching out.
I wasn't able to reproduce this behavior with boto3 version 1.26.44. Are you using recent version? If you could share your code snippet and perhaps debug logs, that could be helpful for us to find out what the issue is.

Best,
John

[maybo65]

Hey @aBurmeseDev, thank you for your response.
I upgraded to version 1.24.59 but the problem keep reproduces. I'm attaching you here a snippet of my code, a capture of the UI, and some of the debug I got.
here is an example of a code that triggers the problem:

def get_redshift_size():
    print(f"boto3 version is: {boto3.__version__}")
    test_session = boto3.session.Session(profile_name='Lab')
    redshift = test_session.client('redshift')

    logging.basicConfig(filename='./boto3.log', level=logging.DEBUG)
    logger = logging.getLogger('botocore')
    logger.setLevel(logging.DEBUG)
    clusters = redshift.describe_clusters()["Clusters"]

    for cluster in clusters:
        print(f"cluster id:{cluster['ClusterIdentifier']}")
        print(f"cluster size in MB:{cluster['TotalStorageCapacityInMegaBytes']}")

here is the output of the code:

boto3 version is: 1.24.59
cluster id:redshift-cluster-1 
cluster size in MB:800000
cluster id:redshift-cluster-2
cluster size in MB:256000000
cluster id:redshift-cluster-3
cluster size in MB:800000 

But, in the UI, we can see that redshift-cluster-1 is from size 320GB (also redshift-cluster-3) :

Im attaching also some of the collected debug. please notice that some of it was masked due to privacy reasons. the debug:

DEBUG:botocore.hooks:Event request-created.redshift.DescribeClusters: calling handler <function add_retry_headers at <adress>
DEBUG:botocore.endpoint:
Sending http request: 
<AWSPreparedRequest stream_output=False, 
method=POST, 
url=https://redshift.us-east-1.amazonaws.com/, 
headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.24.59 Python/3.9.16 Darwin/21.1.0 Botocore/1.27.59', 
'X-Amz-Date': b'20230115T130509Z', 
'X-Amz-Security-Token': <amazon-security-token>, 'Authorization': <auth>, 'amz-sdk-invocation-id': b'9ff9cc35-3736-49de-9236-03fa902302d8', 'amz-sdk-request': b'attempt=1', 'Content-Length': '42'}>
DEBUG:botocore.httpsession:Certificate path: <path>
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): redshift.us-east-1.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://redshift.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 200 10948
DEBUG:botocore.parsers:Response headers: {'x-amzn-RequestId': <request_id>, 'Content-Type': 'text/xml', 'Content-Length': '10948', 'Date': 'Sun, 15 Jan 2023 13:05:10 GMT'} 
DEBUG:botocore.parsers:Response body:
b'<DescribeClustersResponse xmlns="http://redshift.amazonaws.com/doc/2012-12-01/">\n  <DescribeClustersResult>\n  : {a lot of fields (deleted from here becuase some are privare. the result for TotalStorageCapacityInMegaBytes was : <TotalStorageCapacityInMegaBytes>800000</TotalStorageCapacityInMegaBytes>\n )}   
DEBUG:botocore.retryhandler:No retry needed.

Please let me know if you need anything else.
Best regards, and thank you,
May

[aBurmeseDev]

Hi @maybo65 - thanks for sharing these additional details.

Because of the image you provided, I was able to reproduce this behavior with dc2.large cluster. I also tried to reproduce with different types of node cluster and it seems that the only time the value gets mismatched is with that specific node type. This is also reproducible with AWS CLI as this comes from the upstream CreateJob API.

Since this relates to upstream API rather than boto3, I created this issue in our cross-SDK repository and reached out to Redshift team for more investigation. Please feel free to check in on that issue for future updates.

Thank you again,
John

[maybo65]

Thank you for your help!