[feature request]: Implement Flush method.

[bwplotka]

Hello TSDB folks!

We are chasing for a safe solution to quickly "terminate" Prometheus server without losing any monitoring data stored in memory (and WAL). By terminate, we mean killing whole instance, including potential persistent disk. We use thanos for uploading the blocks that are in tsdb-path blocks into object store, so we would like to dump in-mem HEAD block to the filesystem on demand and let Thanos to upload it. But there is no flush API for TSDB (thus no Flush endpoint for Prometheus). The example scenario would look like:

1. We need to scale down Prometheus servers
2. We remove all scrape targets, so nothing new is scraped (nothing new is added to TSDB)
3. We hit Flush endpoint. Head block is flushed to the filesystem and truncated in memory.
4. We wait until Thanos uploads all, including flushed head block.
5. We terminate the instance.

The obvious workaround is TSDB Snapshot method, but that is actually not "safe". TSDB blocks are immutable and overlaps are not tolerated, so:

After we do the snapshot with withHead=true to separate directory (and making Thanos upload from those), we have indeed a portion of HEAD in the object storage (let's called it A) as we wanted. However:

• we are ultimately marking this instance as dirty, because any new TSDB block from HEAD that got "written" into filesystem (because db.compact() decided so) as block B is strictly overlapping with A and thus this instance cannot be used anymore again.
• there is race condition possible that while we are doing a snapshot and uploading those, the B block can be created and also uploaded by Thanos.

All of these problems make our case really difficult to handle, and just single flush logic will help us a lot here. Do you think we can enable those in TSDB (and maybe further in Prometheus?). Would you be ok to take a PR for it?

We would propose something like Flush method that will have logic similar db.compact() method, but with force db.compactor.Write of head block:

func (db *DB) Flush() error {
	db.cmtx.Lock()
	defer db.cmtx.Unlock()

	db.mtx.RLock()
	defer db.mtx.RUnlock()

	// Wrap head into a range that bounds all reads to it.
	head := &rangeHead{
		head: db.head,
		mint: mint,
		maxt: maxt,
	}
	if _, err = db.compactor.Write(db.dir, head, mint, maxt); err != nil {
		return  errors.Wrap(err, "persist head block")
	}

	runtime.GC()
	if err := db.reload(); err != nil {
		return  errors.Wrap(err, "reload blocks")
	}
	runtime.GC()
        return nil
}

What do you think? @gouthamve @fabxc @krasi-georgiev

For context: We are experimenting with something that will auto-scale horizontally Prometheus servers in the highly dynamic environment (scrape targets changing a lot). We have implemented code that assigns targets to each Prometheus server automatically and scales up and down a number of Prometheus instances.

[bwplotka]

CC: @domgreen

[krasi-georgiev]

IIUC the problem is the pending WAL right?
why not just add this to the tsdb cli tool - read the WAL and make a block out of it.

Otherwise about the Flush method wouldn't this suffer the same problems as the snapshots and maybe cause some other race conditions.

• What if another compaction gets triggered at the same time?
• What if the WAL gets truncated at the same time as flush is called?

[krasi-georgiev]

or you meant that the Flush point will be run after Prometheus is shutdown?

[brian-brazil]

I agree, this is best handled by shutting down Prometheus and then working with the data.

[bwplotka]

I agree, this is best handled by shutting down Prometheus and then working with the data.

hmm, but with just shutting down Prometheus we don't have the data from memory in a form of a TSDB block. And when we do Snapshot right before a shutdown, there still might be the race between what snapshot gives us as "head block" and Prometheus writing some block just before shutdown potentially.

[bwplotka]

@krasi-georgiev I mean rather Flush that will write what it has and continue working.

why not just add this to the tsdb cli tool - read the WAL and make a block out of it.

I think it is totally the same as Snapshot API, right? It just gives us potential overlaps with what TSDB writes down to the main directory

[bwplotka]

What if another compaction gets triggered at the same time?

There won't be any since Flush (that truncates head) is synchronized with compaction.

What if the WAL gets truncated at the same time as flush is called?

Again, this is synced, but in the worst case flushed TSDB block is super tiny (which is not perfect, I agree)

[bwplotka]

Or ... @brian-brazil @krasi-georgiev there is some TSDB CLI tool that gives me TSDB block from WAL file somehow after Prometheus termination? 🤔

That would work as well for us I think ^

BTW, thanks for quick responses 👍

[krasi-georgiev]

I was refering to this one.
https://github.com/prometheus/tsdb/tree/master/cmd/tsdb

[bwplotka]

yea, adding some logic that understands WAL file and is able to write block from it, maybe solves this.

[domgreen]

@krasi-georgiev I take it this means we would want to have Prometheus shutdown and call WAL.Close() to flush the head. Therefore, all data in memory is now in the WAL which we can turn into a block for upload.

To then ensure that we do not upload multiple blocks would we need to clear / delete the contents of /wal after we create the block?

[krasi-georgiev]

I will wait for the Scan PR to be merged and will add this next.

[domgreen]

So for our use case we would end up doing something like the following:

- signal for termination of Thanos / Prometheus
- mark instance as dirty
- call /-/quit on Prom
- call Flush on CLI
- Upload created block
- scale down statefulset

However, if we added /admin/tsdb/flush via admin API it would allow us to flush the head/wal to a block and not have to mark the pod as dirty which would be beneficial to scaling up / down instances.

With either approach there is some work to be done the main benefit of exposing it as an API is that the instance could be reused after it has been flushed so we don't have scale down till after a time period we are confident we will not have to scale up again which would make auto scaling much cleaner.

[krasi-georgiev]

I just checked the code again and everything that is in the HEAD is added to the WAL at the same time.
https://github.com/prometheus/tsdb/blob/c848349f07c83bd38d5d19faa5ea71c7fd8923ea/head.go#L526-L530

I think at the moment the only way would be to shutdown Prometheus so it closes the WAL and than you convert it to a block , but there is also a PR that should allow doing this while the TSDB is in use.

#332
Here is the PR for a new WAL format which also add checkpointing. I haven't looked into the checkpointing part yet, but might align with what you trying to do. The idea of the PR is to allow ingesting of the recent metrics by third party systems. The design doc is in few comments below the desc of the PR