Feature/spark 47230 add posexplode support #52598
Draft
+3,788
−25
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds initial support for schema pruning with posexplode operations. Unlike explode, posexplode maintains Generate nodes in the optimized plan, preventing the standard ScanOperation pattern from matching. Changes: - Added Project → LogicalRelation case to handle Generate nodes - Collect GetStructField expressions and trace through Generate mappings - All existing tests pass (190/190) Work in progress: - posexplode queries still show full schema instead of pruned - Need to debug why tryEnhancedNestedArrayPruning returns None - Tracing logic appears correct but not functioning as expected 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Updated the condition in tryEnhancedNestedArrayPruning to allow pruning when there are GetStructField expressions (not just GetArrayStructFields). This is needed for posexplode queries where field accesses like request.available are GetStructField, not GetArrayStructFields. The condition now checks: - tracedThroughGenerate = true (expressions went through Generate) - AND (arrayStructFields OR structFields present) This maintains backward compatibility with SPARK-34638/SPARK-41961 while enabling support for posexplode struct field accesses. Note: Pruning still not working for posexplode in practice. Further investigation needed into the tracing or schema pruning logic. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
The previous implementation incorrectly pruned top-level columns that didn't have nested field accesses, even when they were directly referenced (e.g., in GROUP BY clauses). This caused invalid query plans where Project expected columns that were removed from the relation. Changes: - Collect all AttributeReference nodes from Project's projectList and filters to identify directly referenced columns - Pass requiredColumns set to tryEnhancedNestedArrayPruning - Modified pruneNestedArraySchema to preserve top-level columns even when they don't have nested accesses Example fix: Before: Relation [pv_requests#342] parquet (missing pv_publisherId) After: Relation [pv_publisherId#330L, pv_requests#342] parquet 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
## Approach
This commit attempts to enable nested column pruning for LATERAL VIEW
queries with explode/posexplode while preserving GetArrayStructFields
ordinal correctness.
GetArrayStructFields uses field ordinals (field.ordinal parameter) to
access array elements. When fields are pruned, ordinals shift, causing
GetArrayStructFields to access invalid memory positions and crash with
SIGSEGV.
## Design
The design introduces field order preservation at the level where
GetArrayStructFields directly accesses fields:
1. **Track GetArrayStructFields paths separately** from GetStructField
paths, as they have different access patterns (ordinal vs name-based)
2. **Filter out GetStructField prefix paths** - If GetArrayStructFields
accesses `request.servedItems.clicked`, don't also track
`request.servedItems` from GetStructField, as it's redundant
3. **Depth-based multi-field checking** - Only block pruning when multiple
fields at the SAME depth are accessed (SPARK-34638/SPARK-41961). This
allows pruning for chained explodes with different depth access patterns:
- request.available (depth 1)
- request.servedItems.clicked (depth 2)
4. **Ordinal preservation via pruneStructPreservingFieldOrder** - When
GetArrayStructFields directly accesses fields at a struct level
(path length == 1), call new function that:
- Keeps ALL fields in the struct (preserves ordinals)
- But recursively prunes nested levels (paths with length > 1)
- This prevents ordinal shifts at the accessed level while still
enabling pruning in deeper nested structures
## Changes
### SchemaPruning.scala (lines 152-494)
1. **Added arrayStructFieldPaths tracking** (lines 155-158)
- Separate map to track GetArrayStructFields paths
- Used for ordinal preservation logic
2. **Enhanced GetArrayStructFields processing** (lines 166-177)
- Track paths in both nestedFieldAccesses and arrayStructFieldPaths
- Added debug logging for path tracking
3. **GetStructField prefix filtering** (lines 181-201)
- Skip GetStructField paths that are prefixes of GetArrayStructFields paths
- Prevents redundant path tracking
4. **Improved multi-field depth checking** (lines 210-227)
- Group paths by depth and check for multiple accesses per depth level
- Only block pruning when multiple fields at same depth
- Added debug logging for depth analysis
5. **Pass arrayStructFieldPaths through pruning chain** (lines 229-239, 333-422)
- Thread arrayStructFieldPaths parameter through:
* pruneNestedArraySchema
* pruneFieldByPaths
- Enables ordinal-aware pruning decisions
6. **New pruneStructPreservingFieldOrder function** (lines 445-494)
- Keeps all fields in struct to preserve ordinals
- Recursively prunes nested levels
- Maps each field:
* If accessed directly (path length 1): keep entirely
* If has nested paths: recursively prune via pruneFieldByPaths
* If not accessed: keep for ordinal preservation
7. **Modified pruneFieldByPaths** (lines 463-484)
- Detects when GetArrayStructFields directly accesses fields (length == 1)
- Calls pruneStructPreservingFieldOrder instead of pruneStructByPaths
- Falls back to normal pruning when safe
## Current Status
**Build**: ✅ Successful
**Schema Preservation**: ✅ All 670 fields preserved when needed
**Query Execution**: ❌ Still crashes with SIGSEGV in GetArrayStructFields
## Known Issues
Despite preserving all fields at the accessed level, queries still crash
with memory access errors in GetArrayStructFields. This suggests that
field preservation alone is insufficient - the issue may require:
1. **Expression tree rewriting** - Update GetArrayStructFields ordinal
parameters after pruning
2. **Complete pruning disablement** - Disable pruning entirely when
GetArrayStructFields detected
3. **Schema compatibility validation** - Ensure pruned schema structure
matches GetArrayStructFields expectations beyond just ordinal positions
This is a work-in-progress commit documenting the ordinal preservation
approach and its limitations.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
…IEWs After 28 attempts, successfully resolved multi-column pruning issues for POSEXPLODE with nested arrays. The solution implements a comprehensive four-pass architecture in GeneratorOrdinalRewriting that ensures all AttributeReferences have correct pruned dataTypes. ## Key Changes: - Added comprehensive AttributeReference update pass (Pass 4) in GeneratorOrdinalRewriting - Fixed InternalError (unsafe memory access) that plagued attempts 1-25 - Fixed Size function error discovered in attempts 26-27 - Ensures schema consistency throughout logical plan transformations ## What Works: - POSEXPLODE with chained LATERAL VIEWs: Fully functional with schema pruning - Complex aggregations (max, sum, conditional) - Nested field access (request.available, servedItem.clicked, etc.) - Performance: 99.5% reduction in fields read (670+ → 3 fields) ## Known Issue: - EXPLODE queries work but don't trigger schema pruning (use full schema) - This will be addressed in a follow-up commit ## Test Results: - Real production queries execute successfully - Query times: 1.6-5.6 seconds - No crashes or errors This represents a major breakthrough in enabling efficient nested array processing in Spark SQL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…aliases This commit enables schema pruning for EXPLODE queries with chained LATERAL VIEWs by teaching SchemaPruning to trace through intermediate _extract_ attributes created by NestedColumnAliasing. Problem: When tracing GetStructField(servedItem, clicked) to root columns: 1. servedItem came from explode(_extract_servedItems) 2. _extract_servedItems was not in generateMappings 3. Tracing stopped, returning None, preventing schema pruning Solution: - Collect _extract_* alias mappings during plan traversal - Pass extractAliases through all tracing functions - When an AttributeReference is not in generateMappings, check extractAliases - If found, recursively trace through the alias child expression - This enables: _extract_servedItems → GetArrayStructFields → trace continues Impact: - EXPLODE now achieves same schema pruning as POSEXPLODE - Field reduction: 573 fields → 6 fields (~99% reduction) - Both EXPLODE and POSEXPLODE variants tested successfully - Comprehensive aggregation queries execute without errors Testing: Verified with real-world nested Parquet data containing 573 fields: - EXPLODE: Prunes to 6 fields, executes successfully - POSEXPLODE: Prunes to 6 fields, executes successfully - ReadSchema correctly shows only accessed nested fields - Ordinal rewriting works correctly (e.g., 15→0, 107→1) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…ields
This commit fixes a bug where case-insensitive queries on nested fields
in exploded arrays would fail at runtime after schema pruning.
Problem:
When users write queries like `SELECT friends.First FROM table` (with
capital F), the GetArrayStructFields expression was storing the user-
provided field name ("First") instead of the resolved schema field name
("first"). After schema pruning, the runtime code generation would try
to lookup "First" in the pruned schema using the case-sensitive
StructType.fieldIndex() method, causing a failure.
Solution:
- Modified ExtractValue.apply() to pass the original StructField from
the schema (with correct case) instead of copying it with the user-
provided name
- This ensures GetArrayStructFields uses the resolved field name for
runtime lookups, enabling correct case-insensitive behavior
Changes:
- complexTypeExtractors.scala: Keep original field from schema
- SchemaPruning.scala: Remove all debug code and cleanup
- SchemaPruningSuite.scala: Update test expectations to reflect new
multi-field pruning capability (SPARK-34638, SPARK-41961)
Test Results:
- All Catalyst tests pass (7,199 tests)
- All ParquetV1SchemaPruningSuite tests pass (190 tests, including
16 previously failing case-sensitivity tests)
- SPARK-34638, SPARK-41961, SPARK-34963 tests now pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Two critical fixes:
1. Remove debug logging from production code
- Removed all 28 SCHEMA_PRUNING_DEBUG log statements
- Production code should not contain debug logging
2. Fix schema validation error when entire generator output is referenced
- When .select("friend") references the entire generator output struct,
skip pruning for that column to preserve all fields
- Previously pruned to only accessed fields (friend.first, friend.middle),
causing schema mismatch when entire struct was also referenced
- Added detection for direct generator output attribute references
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?