DynamoDB deserialization throws exception for big numbers

[rowanseymour]

Describe the bug

This was previously reported at #2500 but that issue was closed and the actions bot recommends creating a new issue which references that.

A number like 1234567895171680000000000000000000000000 is valid in DynamoDB and we're happily persisting such numbers using the golang sdk. But boto3 blows up trying to fetch an item with such a number because it treats the zeros as significant digits.

DYNAMODB_CONTEXT = Context(
    Emin=-128,
    Emax=126,
    prec=38,
    traps=[Clamped, Overflow, Inexact, Rounded, Underflow],
)

...

def _deserialize_n(self, value):
    return DYNAMODB_CONTEXT.create_decimal(value)

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Numbers like 1234567895171680000000000000000000000000 are valid in DynamoDB and should be parseable by boto3

Current Behavior

Traceback (most recent call last):
  File "<console>", line 1, in <module>
decimal.Rounded: [<class 'decimal.Rounded'>]

Reproduction Steps

from boto3.dynamodb.types import DYNAMODB_CONTEXT

DYNAMODB_CONTEXT.create_decimal("1234567895171680000000000000000000000000")

Possible Solution

I think it could be fixed using normalize, i.e.

def _deserialize_n(self, value):
    return DYNAMODB_CONTEXT.create_decimal(Decimal(value).normalize())

Additional Information/Context

No response

SDK version used

1.42.23

Environment details (OS name and version, etc.)

MacOS and Ubuntu

[rowanseymour]

Our current monkey patch solution is...

from decimal import Rounded
from boto3.dynamodb.types import DYNAMODB_CONTEXT

DYNAMODB_CONTEXT.traps[Rounded] = False

[jonathan343]

Hey @rowanseymour, thanks for bringing this issue back to our attention.

After investigating, I agree this should be considered a bug. I think the reason this hasn't affected more customers is that both serialization and deserialization perform the same validation via DYNAMODB_CONTEXT.create_decimal in most cases:

Serialization:

boto3/boto3/dynamodb/types.py

Lines 213 to 217 in a6ff277

	def _serialize_n(self, value):
	number = str(DYNAMODB_CONTEXT.create_decimal(value))
	if number in ['Infinity', 'NaN']:
	raise TypeError('Infinity and NaN not supported')
	return number

Deserialization:

boto3/boto3/dynamodb/types.py

Lines 288 to 289 in a6ff277

	def _deserialize_n(self, value):
	return DYNAMODB_CONTEXT.create_decimal(value)

As others have mentioned, the issue surfaces when data is written by other SDKs (Go, Java, etc.), via the low-level client, or using UpdateExpression and ExpressionAttributeValues since they don't enforce this validation.

I think that specifically for deserialization, we should not be performing client side validation. If DynamoDB is storing the value server side, validation was done at some point, so repeating client-side is just additional/unnecessary overhead.

I've opened #4699 to remove this validation. Happy to hear any feedback on the approach.

Thanks!