doubt or problem with jsonpredicatehints

[RodrigoSchneiderbr]

Hi.
Sorry if i use the wrong channel.

I ´m trying to use json predicate hints, to filter a some data using date in deltasharing, using pandas, with AWS GLUE 5.0 and 4.0.
Every time the filter created with json predicate hints doesn´t work, the load becames full, and we have memory issues.

the problem is using a json predicate with one filter works, but using and or or doesn´t work. The type is timestamp

I try using Glue with Spark, but Glue is serverless and don´t have hadoop installed, and i don´t find the version.
If someone has a similar problem or a solution, please if possible could share here.

thanks a lot and sorry for using this space.

[GuoyuHao]

There are multiple problems in the json predicate hints implementation. One typical pattern of the problems are handling the NOT operation. For example, when a column A has (min, max) range (5, 10), the evaluator will return TRUE if the query is 'WHERE A = 7'. This is OK, but if the query is 'WHERE A != 7' the evaluator will return FALSE, because it treats 'A != 7' as 'NOT (A = 7)', so 'NOT( TRUE)' means FALSE. The correct behavior is to return UNKNOWN (which means not 100% sure) instead of TRUE.

[sanket-coditas]

Hi @RodrigoSchneiderbr - Were you able to find the solution to this problem? I am also trying very simple filter like product = 'ABC', but in response, I am always receiving the entire data.
I have raised a separate issue for this #744

[RodrigoSchneiderbr]

Hi @sanket-coditas no, i can´t solve, so i change the way a i use. I transform the timestamp in a date interval and trasnform in string.
So i use the json predicate with equal and solve the problem, but the column must partiotioned ins this collumn too that i created, the json only work in partition column in my case.