Decimals - Validation

[mick999]

I have the following:

    schema = pa.DataFrameSchema({'amount' : pa.Column(Decimal)})
    d = {'amount': Decimal('7504.06')}
    print(type(d['amount']))
    df=pd.json_normalize(d)
    print(df)
    new_df = schema.validate(df)
    print(new_df)

Result is

<class 'decimal.Decimal'>

    amount
0  7504.06

pandera.errors.SchemaError: expected series 'amount' to have type Decimal(28, 0):
failure cases:
   index failure_case
0      0      7504.06

How is the Decimal(28,0) being set ? Or how do you set that value to 2 places here ?

[ankitmodak13]

I’m running into the same issue in my repo. Most of the columns I’m working with are of type Decimal. From what I understand, Pandera allows us to define each column using a Pandas column type. The problem is that Pandas itself doesn’t have a native Decimal (Object instead) column type, which seems to be the root cause here.

[cosmicBboy]

you can use the pandas-engine Decimal type: https://pandera.readthedocs.io/en/stable/reference/generated/pandera.engines.pandas_engine.Decimal.html#pandera-engines-pandas-engine-decimal where you can set the precision, scale, and rounding

[ankitmodak13]

import pandas as pd
from decimal import Decimal
import pandera.pandas as pa
import pandera.engines.pandas_engine as pa_engine

df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6],
    'C': [Decimal("100.00"), Decimal("187.51"), Decimal("-501.89")]
})


schema = pa.DataFrameSchema({
    'A': pa.Column(pa_engine.INT64, checks=pa.Check.ge(0)),
    'B': pa.Column(pa_engine.INT64, checks=pa.Check.ge(0)),
    'C': pa.Column(pa_engine.Decimal(scale=2))
})

schema.validate(df)

Thanks! It's working as expected. Hope this example help others.
I found this when I was looking out for solutions. I just didn't know how to use it (lack of examples there).

Few more things:

• Are my imports okay here? What can I do to improve them?
• Is it better to use all dtypes from pandas_engine as I am working majorly with pandas?

[cosmicBboy]

I just didn't know how to use it (lack of examples there).

Feel free to add more examples to the docs! Perhaps on this page

Are my imports okay here? What can I do to improve them?

The imports look correct

Is it better to use all dtypes from pandas_engine as I am working majorly with pandas?

For most cases, I'd recommend using python-native types or pandas-native types, similar to how you would specify dtypes for your pd.DataFrames as usual, but then reaching for pandas_engine when those types won't suffice for your schema spec.