Skip to content

Performance: Stream query results instead of materializing full lists #229

@mikemcdougall

Description

@mikemcdougall

Problem

Query endpoints materialize full result sets in memory instead of streaming, increasing memory pressure and latency for large datasets.

Evidence (current behavior)

  • OGC Features: featureStore.QueryAsync then result.Items.ToArray() in src/Honua.Server/Features/OgcFeatures/FeaturesEndpoints.cs:309 and src/Honua.Server/Features/OgcFeatures/FeaturesEndpoints.cs:310.
  • FeatureServer: QueryAsync in src/Honua.Server/Features/FeatureServer/FeatureServerHandler.cs:1434 and full formatting on result.Items in src/Honua.Server/Features/FeatureServer/FeatureServerHandler.cs:219 / src/Honua.Server/Features/FeatureServer/FeatureServerHandler.cs:227.
  • OData: QueryAsync then queryResult.Items.Select(...).ToArray() in src/Honua.Server/Features/OData/ODataEndpoints.cs:398 and src/Honua.Server/Features/OData/ODataEndpoints.cs:409.
  • Storage layer builds lists: ExecuteSelectQueryAsync accumulates List<Feature> in src/Honua.Postgres/Features/FeatureStore/Services/FeatureDataAccess.cs:83.

Direction

  • Use existing IStreamingFeatureStore (src/Honua.Core/Features/FeatureStore/Abstractions/IStreamingFeatureStore.cs:11).
  • Stream features via IAsyncEnumerable + DbDataReader and serialize incrementally (e.g., Utf8JsonWriter writing arrays).
  • Keep count as a separate query or optional when returnCountOnly is false.

Tasks

  • Add streaming response writers for FeatureServer / OGC / OData query endpoints.
  • Wire endpoints to IStreamingFeatureStore when available.
  • Ensure JSON/GML serialization supports incremental writing.
  • Add tests covering large results and returnIdsOnly/count paths.

Acceptance criteria

  • No full in‑memory materialization for large queries in core endpoints.
  • Streaming path uses existing IStreamingFeatureStore and Postgres streaming methods.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions