Skip to content

[Streams] Add data quality badge to Streams table #234946

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 37 commits into from
Sep 18, 2025
Merged

[Streams] Add data quality badge to Streams table #234946

merged 37 commits into from
Sep 18, 2025

Conversation

@ElenaStoeva
Copy link
Contributor

@ElenaStoeva ElenaStoeva commented Sep 12, 2025

Closes #233966

Summary

This PR adds a Data quality column to the Streams table.

Screenshot 2025-09-12 at 17 06 26

@ElenaStoeva ElenaStoeva self-assigned this Sep 12, 2025
@ElenaStoeva ElenaStoeva added Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t// release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Feature:Streams This is the label for the Streams Project labels Sep 12, 2025
@ElenaStoeva ElenaStoeva marked this pull request as ready for review September 15, 2025 09:35
@ElenaStoeva ElenaStoeva requested review from a team as code owners September 15, 2025 09:35
@elasticmachine
Copy link
Contributor

elasticmachine commented Sep 15, 2025

Pinging @elastic/kibana-management (Team:Kibana Management)

@yngrdyn
Copy link
Contributor

yngrdyn commented Sep 15, 2025

/oblt-deploy

@flash1293
Copy link
Contributor

flash1293 commented Sep 16, 2025

Seems like we fetch the total count twice now - once for the histogram column and once for the quality column. I guess the reason you added it is because it's handled in a different component, but ideally we don't do multiple requests for the same information. Could we lift it up one level or merge the columns together?

Not sure whether we can even fix this, but if there are no failures for a stream, the request throws an error:

POST http://localhost:5601/internal/streams/esql 400 (Bad Request)

A more conceptual thing, but in this case there are two docs, one in the failure store and one in the index:
Screenshot 2025-09-16 at 11 30 15

Should we show a count of 2 here? The change would be that for the histogram we query FROM <stream name>,<stream name>::failure.

@flash1293
Copy link
Contributor

flash1293 commented Sep 16, 2025

Should we show a count of 2 here? The change would be that for the histogram we query FROM ,::failure.

@yngrdyn any opinion on this? We should probably keep in sync with the data set quality page

@yngrdyn
Copy link
Contributor

yngrdyn commented Sep 16, 2025

@yngrdyn any opinion on this? We should probably keep in sync with the data set quality page

yes, we should include now in the count the ::failures index. That would also help in the histogram, imagine you are ingesting documents and all of them are falling into the failure store, in the histogram you will continue seeing the ingestion trend, otherwise you will see gaps. Also as you pointed out @flash1293 this is aligned with dataset quality

@yngrdyn
Copy link
Contributor

yngrdyn commented Sep 17, 2025

One thing we don't need to resolve in this PR (could you please create a follow up issue @ElenaStoeva?)
It's confusing navigating from the main page to the dataset quality tab, in this case the page is inspecting the last 15min, and the quality is good, but if you navigate to the tab there we are using the last 24hours so you see some differences (quality is degraded and we actually see some documents)
Can we navigate to dataset quality tab with the timerange the user has selected in the streams list?

Screen.Recording.2025-09-17.at.11.07.12.mov

ElenaStoeva and others added 4 commits September 17, 2025 10:50
@ElenaStoeva @yngrdyn
Update x-pack/platform/plugins/shared/streams_app/public/components/s…
982f1ad 
…tream_list_view/data_quality_column.tsx

Co-authored-by: Yngrid Coello <[email protected]>
@ElenaStoeva @yngrdyn
Update x-pack/platform/plugins/shared/streams_app/public/components/s…
f07c843 
…tream_list_view/data_quality_column.tsx

Co-authored-by: Yngrid Coello <[email protected]>
@ElenaStoeva @yngrdyn
Update x-pack/platform/plugins/shared/streams_app/public/hooks/use_st…
ef94bb5 
…reams_histogram_fetch.ts

Co-authored-by: Yngrid Coello <[email protected]>
@ElenaStoeva @yngrdyn
Update x-pack/platform/plugins/shared/streams_app/public/components/s…
3ed5d15 
…tream_list_view/data_quality_column.tsx

Co-authored-by: Yngrid Coello <[email protected]>
@ElenaStoeva
Copy link
Contributor Author

ElenaStoeva commented Sep 17, 2025

Thanks for the review @yngrdyn! I addressed your comments and opened a follow-up issue #235337.

@ElenaStoeva ElenaStoeva force-pushed the streams/data-quality-badge branch from 4222e30 to 54d2770 Compare September 17, 2025 10:27
@yngrdyn
Copy link
Contributor

yngrdyn commented Sep 17, 2025

Should we add some tests to this? e.g. making sure we are getting the correct total number of documents or the correct quality

@ElenaStoeva
Copy link
Contributor Author

ElenaStoeva commented Sep 17, 2025

@yngrdyn how would you suggest to test the total doc count? We are using the existing logic for calculating this number in the Documents column - is this tested somewhere?

@yngrdyn
Copy link
Contributor

yngrdyn commented Sep 17, 2025

We are using the existing logic for calculating this number in the Documents column

We are including now the documents in ::failures indices, that's why I wanted to have like a guardrail there, so if we change it in the future we do it intentionally

@yngrdyn
Copy link
Contributor

yngrdyn commented Sep 17, 2025

In order to not block this, as discussed offline, please create an issue to create some tests for the streams list related to documents including the ones inside ::failures

@ElenaStoeva
Copy link
Contributor Author

ElenaStoeva commented Sep 17, 2025

Created #235387 to track the work around improving test coverage for the recently added/changed features

yngrdyn
yngrdyn approved these changes Sep 17, 2025
View reviewed changes
Copy link
Contributor

@yngrdyn yngrdyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@ElenaStoeva ElenaStoeva enabled auto-merge (squash) September 17, 2025 18:24
@elasticmachine
Copy link
Contributor

elasticmachine commented Sep 17, 2025

💔 Build Failed

Failed CI Steps

Metrics [docs]

‼️ ERROR: no builds found for mergeBase sha [340ca65]

History

cc @ElenaStoeva

@ElenaStoeva ElenaStoeva merged commit f80ee7a into elastic:main Sep 18, 2025
12 checks passed
@ElenaStoeva ElenaStoeva mentioned this pull request Sep 17, 2025
Closed
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Sep 24, 2025
@ElenaStoeva @kibanamachine
@flash1293 @yngrdyn @CAWilson94
[Streams] Add data quality badge to Streams table (elastic#234946)
88eae16 
Closes elastic#233966

## Summary

This PR adds a Data quality column to the Streams table.

<img width="1167" height="666" alt="Screenshot 2025-09-12 at 17 06 26"
src="https://github.com/user-attachments/assets/5d1102ce-0b5d-41fa-ba6d-7ba8eae1ab1e"
/>

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Joe Reuter <[email protected]>
Co-authored-by: Yngrid Coello <[email protected]>
niros1 pushed a commit that referenced this pull request Sep 30, 2025
@ElenaStoeva @kibanamachine
@flash1293 @yngrdyn @niros1
[Streams] Add data quality badge to Streams table (#234946)
072a56d 
Closes #233966

## Summary

This PR adds a Data quality column to the Streams table.

<img width="1167" height="666" alt="Screenshot 2025-09-12 at 17 06 26"
src="https://github.com/user-attachments/assets/5d1102ce-0b5d-41fa-ba6d-7ba8eae1ab1e"
/>

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Joe Reuter <[email protected]>
Co-authored-by: Yngrid Coello <[email protected]>
@ElenaStoeva ElenaStoeva mentioned this pull request Sep 30, 2025
Closed
rylnd pushed a commit to rylnd/kibana that referenced this pull request Oct 17, 2025
@ElenaStoeva @kibanamachine
@flash1293 @yngrdyn
[Streams] Add data quality badge to Streams table (elastic#234946)
cf30d5d 
Closes elastic#233966

## Summary

This PR adds a Data quality column to the Streams table.

<img width="1167" height="666" alt="Screenshot 2025-09-12 at 17 06 26"
src="https://github.com/user-attachments/assets/5d1102ce-0b5d-41fa-ba6d-7ba8eae1ab1e"
/>

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Joe Reuter <[email protected]>
Co-authored-by: Yngrid Coello <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flash1293 flash1293 flash1293 left review comments

@yngrdyn yngrdyn yngrdyn approved these changes

@jbudz jbudz jbudz approved these changes

Assignees

@ElenaStoeva ElenaStoeva

Labels

backport:skip This PR does not require backporting Feature:Streams This is the label for the Streams Project release_note:skip Skip the PR/issue when compiling release notes Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t// v9.2.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Streams] Add Data quality Column to Overview page

6 participants

@ElenaStoeva @elasticmachine @yngrdyn @flash1293 @jbudz @kibanamachine