Loading bar when querying ClickHouse

[RoryCrispin]

Some of our queries take a while to process and it would be good to give the user feedback.
This is likely a Grafana limitation - though I see that in the new UI for 9.5 the loading spinner runs across the top of each panel - so maybe there is, or will soon be, support for full progress bars.

The spinner mentioned is visible in this clip:
https://grafana.com/media/docs/grafana/screen-recording-panel-header-redesign-whats-new-9-4.mp4

[mshustov]

<Loading bar> element in the dashboard is controlled by Grafana data management logic. If it's not rendered correctly, please create a bug report

[RoryCrispin]

My question here is whether we can hook up the grafana loading bar to the ClickHouse client loading bar,

When executing a query against a large dataset in CH, it's useful to see the progress bar in CC - I wonder if we can display that in grafana too

[SpencerTorres]

Just as a note for anyone who finds this discussion, I have put together a proof of concept for streaming that includes the possibility of adding a loading bar: https://github.com/SpencerTorres/clickhouse-datasource/tree/the-great-streaming-update

The feature uses Grafana Live to run all queries and stream the results in, including errors and progress packets.

[hsuanyi]

@SpencerTorres is there a plan to merge your pr?

[SpencerTorres]

We need to find a different way to stream the data. My branch uses Grafana Live for all queries, but this means that they run through a single connection. For larger queries this will cause the connection to break, and then other queries get cancelled too.

We've had some discussions recently on how to solve this. I'm not sure if we'll go the streaming route, or if we'll use incremental querying. It depends on what users need.

I don't believe users want actual streaming, they just want to see the results of a query faster. Streaming a query would make sense if you're returning millions of rows, but Grafana queries aren't like this. Some queries simply take a long time to generate results, and streaming will not make those results appear faster. Query progress is still a useful feature, but we have nowhere to display it

[hsuanyi]

@SpencerTorres Thanks for quick response. Yea. I am convinced on the streaming query front. Thanks

I am actually more interested in pagination. Specifically like kibaba experience, users can just scroll down and kibaba just knows where to pick up.

In grafana-clickhouse, it is much harder since we usually have LIMIT in the query so the grafana time range is not the real boundary. instead, the boundary should be defined by the last log in the clickhouse result.

[SpencerTorres]

#992 has something similar to pagination built in by shifting the timestamp, but it's hard to make a generic version of this since the primary keys can be different for each table