Skip to content

Conversation

@sanketkedia
Copy link
Contributor

@sanketkedia sanketkedia commented Jan 11, 2026

Description of changes

Summarize the changes made by this PR.

  • Improvements & Bug fixes
    • Implements a generic get collection filter that encompasses all possible types of reads of collections
    • Creates indexes for faster lookup
    • Implements e2e GetCollections
  • New functionality
    • ...

Test plan

How are these changes tested?
Added test

  • Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

None

Observability plan

None

Documentation Changes

None

@github-actions
Copy link

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

Copy link
Contributor Author

sanketkedia commented Jan 11, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@sanketkedia sanketkedia marked this pull request as ready for review January 11, 2026 00:16
@propel-code-bot
Copy link
Contributor

propel-code-bot bot commented Jan 11, 2026

Implement sysdb get_collections with filtering, pagination, and Spanner support

Adds a full Rust-side implementation of the SysDB get_collections endpoint. A new CollectionFilter builder and associated GetCollectionsRequest/Response types translate protobuf requests into backend operations, while the Spanner backend now executes dynamic SQL supporting ID/name filters, tenant/database scoping, soft-delete toggles, and pagination. The gRPC server path is wired through assign/run, migrations introduce lookup and listing indexes, and extensive integration tests cover the new functionality.

Key Changes

• Introduced CollectionFilter, GetCollectionsRequest, and GetCollectionsResponse in rust/rust-sysdb/src/types.rs, including proto conversions and backend routing.
• Extended SpannerBackend with a dynamic get_collections query that joins metadata and compaction cursor tables, applies filter predicates, and reconstructs Collection models in order.
• Updated Backend::get_collections and gRPC server wiring in rust/rust-sysdb/src/backend.rs and rust/rust-sysdb/src/server.rs to route requests through the new pipeline.
• Added Spanner migrations 0009-create_collections_lookup_index.spanner.sql and 0010-create_collections_list_index.spanner.sql plus manifest updates.
• Expanded integration test suite in rust/rust-sysdb/src/spanner.rs covering retrieval by ID/name, pagination, multi-tenant isolation, metadata preservation, ordering, and null dimensions.

Affected Areas

• rust/rust-sysdb/src/types.rs
• rust/rust-sysdb/src/spanner.rs
• rust/rust-sysdb/src/backend.rs
• rust/rust-sysdb/src/server.rs
• rust/spanner-migrations/migrations/0009-create_collections_lookup_index.spanner.sql
• rust/spanner-migrations/migrations/0010-create_collections_list_index.spanner.sql
• rust/spanner-migrations/migrations/migrations.sum

This summary was automatically generated by @propel-code-bot

Comment on lines +310 to +315
if let Some(limit) = req.limit {
filter = filter.limit(limit as u32);
}
if let Some(offset) = req.offset {
filter = filter.offset(offset as u32);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

[Logic] Potential integer underflow/wrapping risk: If the protobuf limit or offset fields are signed int32 (standard in proto3), casting a negative value to u32 will result in a very large number (e.g., -1 becomes u32::MAX).

Use try_from to safely validate that values are non-negative before casting.

Suggested change
if let Some(limit) = req.limit {
filter = filter.limit(limit as u32);
}
if let Some(offset) = req.offset {
filter = filter.offset(offset as u32);
}
// Handle limit and offset
if let Some(limit) = req.limit {
let limit = u32::try_from(limit).map_err(|_| SysDbError::InvalidArgument("Limit must be non-negative".to_string()))?;
filter = filter.limit(limit);
}
if let Some(offset) = req.offset {
let offset = u32::try_from(offset).map_err(|_| SysDbError::InvalidArgument("Offset must be non-negative".to_string()))?;
filter = filter.offset(offset);
}
Context for Agents
Potential integer underflow/wrapping risk: If the protobuf `limit` or `offset` fields are signed `int32` (standard in proto3), casting a negative value to `u32` will result in a very large number (e.g., `-1` becomes `u32::MAX`). 

Use `try_from` to safely validate that values are non-negative before casting.

```suggestion
        // Handle limit and offset
        if let Some(limit) = req.limit {
            let limit = u32::try_from(limit).map_err(|_| SysDbError::InvalidArgument("Limit must be non-negative".to_string()))?;
            filter = filter.limit(limit);
        }
        if let Some(offset) = req.offset {
            let offset = u32::try_from(offset).map_err(|_| SysDbError::InvalidArgument("Offset must be non-negative".to_string()))?;
            filter = filter.offset(offset);
        }
```

File: rust/rust-sysdb/src/types.rs
Line: 315

SELECT c.collection_id
FROM collections c
WHERE {where_clause}
ORDER BY c.created_at ASC
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

[Logic] Ordering by created_at alone is non-deterministic for collections created at the same timestamp. This can lead to unstable pagination (skipping or duplicating items across pages). Add collection_id as a tie-breaker to ensure a consistent sort order.

Suggested change
ORDER BY c.created_at ASC
ORDER BY c.created_at ASC, c.collection_id ASC

Additional affected locations:

  • rust/rust-sysdb/src/spanner.rs:713
Context for Agents
Ordering by `created_at` alone is non-deterministic for collections created at the same timestamp. This can lead to unstable pagination (skipping or duplicating items across pages). Add `collection_id` as a tie-breaker to ensure a consistent sort order.

```suggestion
                ORDER BY c.created_at ASC, c.collection_id ASC
```

**Additional affected locations:**
- rust/rust-sysdb/src/spanner.rs:713

File: rust/rust-sysdb/src/spanner.rs
Line: 684

@sanketkedia sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch from 0197b36 to ecca61f Compare January 11, 2026 21:25
@sanketkedia sanketkedia force-pushed the 01-10-_enh_get_collections_impl branch from 75c2da6 to 578c9ec Compare January 11, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants