[VL] TPCDS Bloomfilter fallback all time

[Game-Cruz]

I've upgraded Gluten to support Spark 3.5.3 and tried running TPCDS benchmark queries which use bloom filter. But, it's failing with below mentioned segmentation fault. Can someone who uses Gluten/Velox with Spark 3.5.3 help me with the issue?

Spark Conf used:
spark-shell --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.loadLibFromJar=true --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf "spark.driver.extraJavaOptions"="-Dio.netty.tryReflectionSetAccessible=true -XX:+ErrorFileToStderr" --conf "spark.executor.extraJavaOptions"="-Dio.netty.tryReflectionSetAccessible=true -XX:+ErrorFileToStderr" --conf "spark.plugins"="org.apache.gluten.GlutenPlugin" --conf spark.executor.memory=4g --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=22g --conf spark.sql.autoBroadcastJoinThreshold=10m

TPCDS Query q1:

spark.sql("""WITH customer_total_return AS
(SELECT sr_customer_sk AS ctr_customer_sk, sr_store_sk AS ctr_store_sk,
         sum(sr_return_amt) AS ctr_total_return
    FROM store_returns, date_dim
    WHERE sr_returned_date_sk = d_date_sk AND d_year = 2000
    GROUP BY sr_customer_sk, sr_store_sk)
 SELECT c_customer_id
   FROM customer_total_return ctr1, store, customer
   WHERE ctr1.ctr_total_return >
    (SELECT avg(ctr_total_return)*1.2
      FROM customer_total_return ctr2
       WHERE ctr1.ctr_store_sk = ctr2.ctr_store_sk)
   AND s_store_sk = ctr1.ctr_store_sk
   AND s_state = 'TN'
   AND ctr1.ctr_customer_sk = c_customer_sk
   ORDER BY c_customer_id LIMIT 100
""").collect

Query Failure:

terminate called after throwing an instance of 'facebook::velox::VeloxUserError'
what():  Exception: VeloxUserError
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: (1 vs. 0)
Retriable: False
Expression: kBloomFilterV1 == version
Function: merge
File: /root/velox/velox/common/base/BloomFilter.h
Line: 67
Stack trace:
# 0  facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)
# 1  void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxUserError, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 2  void facebook::velox::exec::SimpleFunctionAdapter<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::BloomFilterMightContainFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, bool, facebook::velox::ConstantChecker<facebook::velox::Varbinary, long>, facebook::velox::Varbinary, long> >::unpackInitialize<1, facebook::velox::StringView>(std::vector<std::shared_ptr<facebook::velox::Type const>, std::allocator<std::shared_ptr<facebook::velox::Type const> > > const&, facebook::velox::core::QueryConfig const&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector> > > const&, facebook::velox::StringView const*) const [clone .isra.0]
# 3  facebook::velox::exec::SimpleFunctionAdapterFactoryImpl<facebook::velox::core::UDFHolder<facebook::velox::functions::sparksql::BloomFilterMightContainFunction<facebook::velox::exec::VectorExec>, facebook::velox::exec::VectorExec, bool, facebook::velox::ConstantChecker<facebook::velox::Varbinary, long>, facebook::velox::Varbinary, long> >::createVectorFunction(std::vector<std::shared_ptr<facebook::velox::Type const>, std::allocator<std::shared_ptr<facebook::velox::Type const> > > const&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector> > > const&, facebook::velox::core::QueryConfig const&) const
# 4  facebook::velox::exec::(anonymous namespace)::compileRewrittenExpression(std::shared_ptr<facebook::velox::core::ITypedExpr const> const&, facebook::velox::exec::(anonymous namespace)::Scope*, facebook::velox::core::QueryConfig const&, facebook::velox::memory::MemoryPool*, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool)
# 5  facebook::velox::exec::(anonymous namespace)::compileExpression(std::shared_ptr<facebook::velox::core::ITypedExpr const> const&, facebook::velox::exec::(anonymous namespace)::Scope*, facebook::velox::core::QueryConfig const&, facebook::velox::memory::MemoryPool*, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool)
# 6  facebook::velox::exec::compileExpressions(std::vector<std::shared_ptr<facebook::velox::core::ITypedExpr const>, std::allocator<std::shared_ptr<facebook::velox::core::ITypedExpr const> > > const&, facebook::velox::core::ExecCtx*, facebook::velox::exec::ExprSet*, bool)
# 7  facebook::velox::exec::ExprSet::ExprSet(std::vector<std::shared_ptr<facebook::velox::core::ITypedExpr const>, std::allocator<std::shared_ptr<facebook::velox::core::ITypedExpr const> > > const&, facebook::velox::core::ExecCtx*, bool)
# 8  gluten::SubstraitToVeloxPlanValidator::validate(substrait::FilterRel const&)
# 9  gluten::SubstraitToVeloxPlanValidator::validate(substrait::Plan const&)
# 10 Java_org_apache_gluten_vectorized_PlanEvaluatorJniWrapper_nativeValidateWithFailureReason
# 11 0x00007f9a3460fa10
# 12 0x00007f9a34609fc0
# 13 0x00007f9a34609fc0
# 14 0x00007f9a34609fc0
# 15 0x00007f9a3460a0a2
# 16 0x00007f9a34609fc0
# 17 0x00007f9a3460a0a2
# 18 0x00007f9a34609fc0
# 19 0x00007f9a34609fc0
# 20 0x00007f9a34609fc0
# 21 0x00007f9a3460a0a2
# 22 0x00007f9a34609fc0
# 23 0x00007f9a34609fc0
# 24 0x00007f9a3460a0a2
# 25 0x00007f9a3460a0a2
# 26 0x00007f9a34609fc0
# 27 0x00007f9a3d1073e0

[Game-Cruz]

When --conf spark.gluten.sql.native.bloomFilter=false is set, getting the below warnings.

25/05/31 16:40:16 WARN FilterExecTransformer: Validation failed with exception for plan: FilterExecTransformer, due to: Not supported to map spark function name to substrait function name: might_contain(Subquery subquery#216, [id=#1111], cast(sr_store_sk#10 as bigint)), class name: BloomFilterMightContain.
25/05/31 16:40:16 WARN GlutenFallbackReporter: Validation failed for plan: Filter[QueryId=1], due to: Not supported to map spark function name to substrait function name: might_contain(Subquery subquery#216, [id=#1111], cast(sr_store_sk#10 as bigint)), class name: BloomFilterMightContain..
25/05/31 16:40:16 WARN RegularHashAggregateExecTransformer: Validation failed with exception for plan: RegularHashAggregateExecTransformer, due to: Could not find a valid substrait mapping name for bloom_filter_agg(_pre_2#284L, 1000000, 8388608, 0, 0).
25/05/31 16:40:16 WARN GlutenFallbackReporter: Validation failed for plan: ObjectHashAggregate[QueryId=1], due to: Could not find a valid substrait mapping name for bloom_filter_agg(_pre_2#284L, 1000000, 8388608, 0, 0).; Could not find a valid substrait mapping name for bloom_filter_agg(_pre_2#284L, 1000000, 8388608, 0, 0)..
25/05/31 16:40:17 WARN RegularHashAggregateExecTransformer: Validation failed with exception for plan: RegularHashAggregateExecTransformer, due to: Could not find a valid substrait mapping name for bloom_filter_agg(cast(s_store_sk#52 as bigint), 1000000, 8388608, 0, 0).
25/05/31 16:40:17 WARN GlutenFallbackReporter: Validation failed for plan: ObjectHashAggregate[QueryId=1], due to: Could not find a valid substrait mapping name for bloom_filter_agg(cast(s_store_sk#52 as bigint), 1000000, 8388608, 0, 0)..