[Search Improvements] Phase 3: Advanced Search Features

[carlesarnal]

Parent Epic

Part of #7009 - SQL Search Functionality Improvements

Overview

Add advanced search features including new filter types, query expression parser, faceted search, and search suggestions/auto-complete.

Goals

1. Add new search filter types (owner, date ranges, hasLabels, hasReferences)
2. Implement query expression parser for advanced query syntax
3. Add faceted search support for filter refinement
4. Implement search suggestions/auto-complete
5. Add result highlighting

Tasks

1. New Search Filter Types

• Add new types to SearchFilterType enum:

  // New filter types
  owner,           // Search by artifact owner/creator
  createdAfter,    // Date range: created after
  createdBefore,   // Date range: created before
  modifiedAfter,   // Date range: modified after
  modifiedBefore,  // Date range: modified before
  hasLabels,       // Has any labels (boolean)
  hasReferences,   // Has references to other artifacts
  referencedBy,    // Is referenced by other artifacts
  references,      // References specific artifact

• Add factory methods to SearchFilter:

  SearchFilter.ofOwner(String owner)
  SearchFilter.ofCreatedAfter(Instant after)
  SearchFilter.ofCreatedBefore(Instant before)
  SearchFilter.ofModifiedAfter(Instant after)
  SearchFilter.ofModifiedBefore(Instant before)
  SearchFilter.ofHasLabels(boolean hasLabels)
  SearchFilter.ofHasReferences(boolean hasReferences)

• Implement SQL generation for each new filter type
• Add REST API parameters for new filters

2. Query Expression Parser

• Create SearchQueryParser class to parse advanced query syntax:

  // Supported syntax:
  "user service"                    // Full-text search
  name:UserService                  // Field-specific exact match
  name:User* AND type:AVRO          // Wildcards + boolean
  labels:team:platform              // Label with key:value
  created:>2024-01-01               // Date comparison
  state:ENABLED OR state:DEPRECATED // OR condition
  NOT owner:admin                   // Negation
  

• Implement tokenizer for query parsing
• Build AST (Abstract Syntax Tree) for complex queries
• Convert AST to Set
• Handle syntax errors gracefully with meaningful messages

3. Faceted Search Support

• Create FacetValue DTO:

  public class FacetValue {
      private String value;
      private Long count;
  }

• Add facets to search results:

  public class ArtifactSearchResultsDto {
      // Existing
      private List<SearchedArtifactDto> artifacts;
      private Integer count;
      // New
      private Map<String, List<FacetValue>> facets;
  }

• Implement facet aggregation queries:

  -- Artifact type facet
  SELECT type, COUNT(*) as count
  FROM artifacts a WHERE ... GROUP BY type ORDER BY count DESC;
  
  -- Label facet
  SELECT labelKey, labelValue, COUNT(*) as count
  FROM artifact_labels l JOIN artifacts a ON ...
  WHERE ... GROUP BY labelKey, labelValue ORDER BY count DESC LIMIT 20;

• Add includeFacets query parameter to REST API
• Implement facets for: artifactType, labels, state, groupId, owner

4. Search Suggestions/Auto-complete

• Create new REST endpoint:

  @GET
  @Path("/search/suggestions")
  public List<SearchSuggestion> getSuggestions(
      @QueryParam("prefix") String prefix,
      @QueryParam("field") String field,  // name, label, groupId
      @QueryParam("limit") int limit
  );

• Create SearchSuggestion DTO:

  public class SearchSuggestion {
      private String value;
      private String field;
      private Long frequency;
  }

• Implement suggestion queries:

  -- Name suggestions
  SELECT DISTINCT name, COUNT(*) as freq
  FROM artifacts WHERE name ILIKE ? || '%'
  GROUP BY name ORDER BY freq DESC, name LIMIT ?;
  
  -- Label suggestions
  SELECT DISTINCT labelKey, COUNT(*) as freq
  FROM artifact_labels WHERE labelKey ILIKE ? || '%'
  GROUP BY labelKey ORDER BY freq DESC LIMIT ?;

• Add debouncing recommendations in API docs

5. Result Highlighting

• Add matchHighlights field to search result DTOs:

  public class SearchedArtifactDto {
      // Existing fields...
      private Map<String, String> matchHighlights;
  }

• Implement PostgreSQL highlighting with ts_headline:

  ts_headline('english', name, query, 'StartSel=<mark>, StopSel=</mark>')

• Implement simple pattern-based highlighting for other databases
• Add includeHighlights query parameter

6. REST API Updates

• Update /search/artifacts endpoint with new parameters:

  parameters:
    - name: owner
    - name: createdAfter
    - name: createdBefore
    - name: modifiedAfter
    - name: modifiedBefore
    - name: includeFacets
    - name: includeHighlights

• Add new /search/suggestions endpoint
• Update OpenAPI specification
• Update API documentation

7. Testing

• Unit tests for SearchQueryParser
• Unit tests for each new filter type
• Integration tests for faceted search
• Integration tests for suggestions
• Integration tests for highlighting
• Test complex query expressions
• Test edge cases and error handling

Files to Modify

• app/src/main/java/io/apicurio/registry/storage/dto/SearchFilterType.java
• app/src/main/java/io/apicurio/registry/storage/dto/SearchFilter.java
• app/src/main/java/io/apicurio/registry/storage/dto/ArtifactSearchResultsDto.java
• app/src/main/java/io/apicurio/registry/storage/dto/SearchedArtifactDto.java
• app/src/main/java/io/apicurio/registry/storage/impl/sql/AbstractSqlRegistryStorage.java
• app/src/main/java/io/apicurio/registry/rest/v3/impl/SearchResourceImpl.java
• app/src/main/java/io/apicurio/registry/rest/v3/SearchResource.java
• common/src/main/resources/META-INF/openapi.json

New Files

• app/src/main/java/io/apicurio/registry/storage/dto/FacetValue.java
• app/src/main/java/io/apicurio/registry/storage/dto/SearchSuggestion.java
• app/src/main/java/io/apicurio/registry/storage/search/SearchQueryParser.java
• app/src/main/java/io/apicurio/registry/storage/search/QueryToken.java
• app/src/main/java/io/apicurio/registry/storage/search/QueryAST.java

Acceptance Criteria

• All new filter types work correctly
• Query expression parser handles complex queries
• Faceted search returns accurate aggregations
• Suggestions return relevant auto-complete options
• Highlighting marks matched terms in results
• New REST endpoints documented in OpenAPI spec
• Backward compatible - existing queries continue to work
• All tests pass

Configuration

apicurio.search.facets.enabled=true
apicurio.search.facets.max-values=50
apicurio.search.suggestions.enabled=true
apicurio.search.suggestions.max=10
apicurio.search.highlights.enabled=true

API Examples

Advanced Query

GET /search/artifacts?q=name:User* AND type:AVRO AND created:>2024-01-01

Faceted Search

GET /search/artifacts?labels=team:platform&includeFacets=true

Response:
{
  "artifacts": [...],
  "count": 42,
  "facets": {
    "artifactType": [
      {"value": "AVRO", "count": 25},
      {"value": "PROTOBUF", "count": 17}
    ],
    "labels": [
      {"value": "env:prod", "count": 30},
      {"value": "env:dev", "count": 12}
    ]
  }
}

Suggestions

GET /search/suggestions?prefix=user&field=name&limit=5

Response:
[
  {"value": "user-service", "field": "name", "frequency": 15},
  {"value": "user-api", "field": "name", "frequency": 8},
  {"value": "user-events", "field": "name", "frequency": 5}
]

---

Labels: enhancement, storage, search, api