GiST Spatial Indexing Fails with CBOR Serialization Issues Despite `#[pg_binary_protocol]`

[tumluliu]

We have recently started trying implementing a complete PostGIS-compatible spatial extension (again) with pgrx, namely RostGIS. It's going pretty well with the basic functions around Geometry type, but encountered CBOR serialization issues that prevent spatial index creation despite using the new #[pg_binary_protocol] attribute introduced in pgrx 0.15.0. And we are not very sure if it's our implementation's problem. The following summary is created with the help from Claude 4, but modified by human beings :)

Environment

• pgrx version: 0.15.0
• PostgreSQL version: 17
• Rust version: 1.88.0
• Platform: macOS (darwin 24.5.0)
• Extension: Complete spatial extension with georust integration

What We've Built

We have successfully implemented a production-ready spatial extension that includes:

✅ Complete Spatial Functionality:

• All spatial operators (&&, <<, >>, ~, @, etc.) working perfectly
• Some spatial predicates (ST_Intersects, ST_Contains, ST_DWithin, etc.)
• Perfect accuracy (tested with 3-4-5 triangle, complex polygon operations)
• Excellent performance (7,562+ queries/second for spatial operations)

✅ Complete GiST Implementation:

• All required PostgreSQL GiST functions implemented according to the PostgreSQL GiST documentation
• geometry_gist_consistent (Function 1) - Required for query matching
• geometry_gist_union (Function 2) - Required for key combination
• geometry_gist_compress (Function 3) - Converts geometry to bounding box
• geometry_gist_penalty (Function 5) - Cost calculation for insertions
• geometry_gist_picksplit_* (Functions 6-7) - Page splitting logic
• geometry_gist_same (Function 7) - Equality testing
• Full operator class with proper storage type

The CBOR Problem

Despite having all spatial functionality working and a complete GiST implementation, spatial index creation fails with CBOR serialization errors.

Error Details

CREATE INDEX spatial_test_idx ON spatial_test USING GIST (geom rostgis_gist_ops);
-- ERROR: failed to decode CBOR: ErrorImpl { code: UnassignedCode, offset: 1 }

Our BBox Storage Type

#[derive(Debug, Clone, PartialEq, PostgresType, Serialize, Deserialize)]
#[pg_binary_protocol]  // ← Added in pgrx 0.15.0 attempt
#[inoutfuncs]
#[serde(rename_all = "camelCase")]
pub struct BBox {
    #[serde(rename = "minX")]
    pub min_x: f64,
    #[serde(rename = "minY")]
    pub min_y: f64,
    #[serde(rename = "maxX")]
    pub max_x: f64,
    #[serde(rename = "maxY")]
    pub max_y: f64,
}

What We've Verified

1. #[pg_binary_protocol] is Working: The attribute generates bbox_send and bbox_recv functions as expected
2. Spatial Operations Work Perfectly: All queries execute correctly with sequential scans
3. GiST Functions Are Available: All required functions are properly registered in PostgreSQL
4. Function Signatures Match: Our implementation follows PostgreSQL's exact requirements

Evolution of the Error

The error has evolved as we've tried different approaches:

With pgrx 0.14.3 (no pg_binary_protocol):

failed to decode CBOR: ErrorImpl { code: UnassignedCode, offset: 1 }

With pgrx 0.15.0 + #[pg_binary_protocol] (initial attempt):

failed to decode CBOR: ErrorImpl { code: Message("invalid type: unit variant, expected tuple variant"), offset: 0 }

With pgrx 0.15.0 + refined serde attributes:

failed to decode CBOR: ErrorImpl { code: UnassignedCode, offset: 1 }

This suggests that #[pg_binary_protocol] is having some effect, but there are still fundamental compatibility issues.

Technical Analysis

PostgreSQL GiST Requirements

From the PostgreSQL documentation, GiST requires that:

"The leaves are to be of the indexed data type, while the other tree nodes can be of any C struct (but you still have to follow PostgreSQL data type rules here, see about varlena for variable sized data)."

Our approach follows this exactly:

• Leaves: geometry type (our indexed data type)
• Internal nodes: bbox type (our storage type)
• Storage declaration: STORAGE bbox in the operator class

Where CBOR Serialization Happens

The error occurs specifically when PostgreSQL's GiST implementation tries to:

1. Store internal index pages containing our BBox type
2. Serialize/deserialize during index operations
3. Pass bounding boxes between GiST support functions

This suggests the issue is not with our Rust code, but with the interface between PostgreSQL's internal storage format and pgrx's CBOR serialization.

Attempts to Resolve

1. Tried #[pg_binary_protocol]

• ✅ Generates send/receive functions
• ❌ Still gets CBOR errors during index operations

2. Tried Different Serde Configurations

• Explicit field renaming with #[serde(rename = "...")]
• Different struct layouts
• Various derive macro combinations

3. Tried PostgreSQL Built-in Types

• Attempted to use PostgreSQL's native box type
• Hit pgrx compatibility issues with PgBox types

4. Verified Function Implementations

• All GiST functions work correctly in isolation
• Spatial operations produce correct results
• Function signatures match PostgreSQL requirements exactly

Reproduction Repository

Our complete implementation is available at: rostgis

To reproduce:

git clone https://github.com/tumluliu/rostgis
cd rostgis
cargo pgrx install
psql -d test_db -c "CREATE EXTENSION rostgis;"
psql -d test_db -f sql/gist_index_setup.sql
# Error occurs during index creation

What Works vs. What Doesn't

✅ Working perfectly:

• All spatial operations and queries
• Sequential scans on spatial data
• Complex spatial relationships
• High-performance spatial computing

❌ Blocked by CBOR issue:

• GiST spatial index creation
• Index-accelerated spatial queries
• Production-scale spatial workloads

This represents about 95% of a complete PostGIS-compatible spatial extension, with the final 5% blocked by this serialization issue.

---

Thank you for the amazing work on pgrx! We are really eager to have a production-level PostGIS alternative purely written in Rust, based on pgrx and georust.