Ideas for TSDB CLI tool

[fabxc]

• A ls that gives a tabular view of blocks and info about them
• Scanning tool that checks for corruption and reports on which sections in which file are affected
• Doing compactions on blocks offline
• Getting info about values for labels, series for matchers, sizes of postings lists – anything querying related basically
• Downsampling all or some series in blocks.
• Tombstoning (and possibly immediately compacting) a set of series in blocks. That's effectively dynamic retention.

Lots of things possible. But not all of them are super useful aside from things being possible in theory.
Some are possible through Prometheus' APIs in one way or another. But it seems fine to have multiple interfaces to the same functionality.

[krasi-georgiev]

DIBS

[BasPH]

@krasi-georgiev I want to implement some of the other suggestions, are you working on it?

[krasi-georgiev]

@BasPH not at the moment so go ahead.

[krasi-georgiev]

• ls that gives a tabular view of blocks and info about them
• Scanning tool that checks for corruption and reports on which sections in which file are affected
• Doing compactions on blocks offline
• Getting info about values for labels, series for matchers, sizes of postings lists – anything querying related basically
• Downsampling all or some series in blocks.
• Tombstoning (and possibly immediately compacting) a set of series in blocks. That's effectively dynamic retention.

[krasi-georgiev]

please share use cases and I can implement the remaining ideas that would be useful.

[FUSAKLA]

I'd like to take a look at dumping of blocks to human readable data for debugging once I'll get to it.

[krasi-georgiev]

sure , just don't forget to discuss how/why would this be useful to get some feedback before starting the design.

[codesome]

@krasi-georgiev @FUSAKLA

I plan to add new features to TSDB CLI in my GSoC (Section 4.5 in my proposal). I will be discussing the design with the team soon. I am surely open for suggestions.

[krasi-georgiev]

I think we should also discuss a way to simplify the debugging as replicating bugs at the moment is quite hard.

I was thinking maybe add some debugging mode that will enable tsdb to duplicate the data in a debug folder where all labels are scrambled to mask the user data or add this to the cli tool. The main problem with the first approach is that this needs to be in the tsdb code and the main problem with the second approach is that in some cases the data would be corrupted so no way to scramble it simply because it cannot be red.

[brian-brazil]

Adding an extra code path is likely to introduce bugs and pose maintenance challenges, and doesn't help if the user hasn't enabled this in advance. If we want a user's data to aid in debugging we can work with them to find a way to get it.

[krasi-georgiev]

I haven't looked thoroughly , but I think that the duplication can happen at the end when we write to disk so should be a worry for introducing bugs.
It seems that in most issues users can replicate the bugs so enabling debug mode should help in most cases

To be honest I don't like either idea, but can't think of anything else at the moment.

@brian-brazil Do you have any other ideas?