Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for format JSONEachRow with Logs Panel #148

Open
mxlxm opened this issue Jun 24, 2022 · 6 comments
Open

add support for format JSONEachRow with Logs Panel #148

mxlxm opened this issue Jun 24, 2022 · 6 comments

Comments

@mxlxm
Copy link

mxlxm commented Jun 24, 2022

Is your feature request related to a problem? Please describe.
display the whole logs within logs panel without complex query

Describe the solution you'd like
add support for format JSONEachRow, display the whole log and allow user to specify the time column.
query would by more simple
select * from xxx format JSONEachRow

Describe alternatives you've considered
complex query like select time,toJSONString(tuple(x,y,z)) from xxx, while in this way, column name is absent.

@gingerwizard
Copy link
Collaborator

We will support JSON type properly and avoid JSONEachRow - the driver uses native format. See #129

@mxlxm
Copy link
Author

mxlxm commented Jun 27, 2022

No, it's not JSON type, consider that we've created a table for nginx access_log, each field with a separate column.
e.g.

create table nginx_access (
    `time` DateTime,
    `host` LowCardinality(String),
    `request_uri` String,
    ...
)

with FOTMAT JSONEachRow, we can see a complete log(with field name).

@gingerwizard
Copy link
Collaborator

Format controls the response format from ClickHouse. The driver uses the native format for performance reasons - Grafana uses this for communication. Introducing support in the driver for JSONEachRow just for a logs panel is going to be hard to justify - not to mention we'd need to detect the format usage and switch the encoder.

If you used the JSON type for all the fields except time, then it will just be rendered as a JSON line shortly (next release probably). Tuples will also be rendered as JSON in the next release btw.

@mxlxm
Copy link
Author

mxlxm commented Jun 28, 2022

JSON type is pretty good for semi-structured data, but with nginx access_log we definitely know how the log looks like.

On the other hand, it should give a significant increase in performance (especially in the right indexing) with separate column, moreover we can use LowCardinality and codec to reduce disk space.

I didn't try with JSON type, but format limited rows vs store whole log with JSON type, I think I would prefer the former

@gingerwizard
Copy link
Collaborator

gingerwizard commented Jun 28, 2022

Yes id imagine configuring some columns explicitly, to exploit codes etc, with a column for any remaining dynamic JSON. Either way in the next release of the plugin, tuples, maps and nested (and thus JSON) will be displayed as JSON.

I don't think this will be done in the driver however - I'll raise a ticket to support JSONEachRow in the driver but i don't see this happening soon given the likely minimal use outside of this Grafana usecase. We could also consider a mode in the plugin which does this transformation if it gets enough +1.

@TimSimmons
Copy link

Something fairly similar here with respect to the log view, some way to view just any arbitrary selected row in that format would be wonderful. It's a very confusing experience for folks if you have "logs" in a table.

You might have a log (like an nginx log) in a table with an actual schema, but you can only view columns that are string types at all, and even then only one shows up until you click a log and hit the 👁️ next to the field. Some way to be able to show all the fields in that view besides the Grafana table view would be wonderful. It's possible of course to toString each column, but if you wanted to dynamically select those columns somehow (like with a Grafana variable), you're out of luck.

@aangelisc aangelisc added the enhancement New feature or request label Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Feature Requests
Development

No branches or pull requests

5 participants