Expose metrics about the database dump

[fridex]

Is your feature request related to a problem? Please describe.

As Thoth operator and Thoth administrator, I would like to have an overview on the database dumps available on Ceph. This way, I can be sure there are done periodic dumps (and they are running).

Describe the solution you'd like

Create a metric that exposes information about database dumps available in the deployment. If there are no database dumps for a certain period of time (let's say 2 days), Thoth operator should be alerted about this fact.

• list available PG dumps on Ceph
• parse datetime when they were created (part of the filename)
• expose this information in grafana dashboard
• create an alert if there is no database backup done in the past X days (parametrizable)
• expose information about the number of dumps available
• alert if the database dumps go bellow certain number or reach certain number (parametrizable)
  • indicates that the dumps are not properly cleaned or they are dropped

[pacospace]

Thanks @fridex, what about having metrics in https://github.com/thoth-station/graph-backup-job that tell us if dump was create or not?

[fridex]

Thanks @fridex, what about having metrics in https://github.com/thoth-station/graph-backup-job that tell us if dump was create or not?

That sounds great 👍🏻

[pacospace]

Thanks @fridex, what about having metrics in https://github.com/thoth-station/graph-backup-job that tell us if dump was create or not?

That sounds great 👍🏻

Added metrics in thoth-station/graph-backup-job#215 🚀

[pacospace]

Is your feature request related to a problem? Please describe.

As Thoth operator and Thoth administrator, I would like to have an overview on the database dumps available on Ceph. This way, I can be sure there are done periodic dumps (and they are running).

Describe the solution you'd like

Create a metric that exposes information about database dumps available in the deployment. If there are no database dumps for a certain period of time (let's say 2 days), Thoth operator should be alerted about this fact.

• list available PG dumps on Ceph

@fridex do you mean exposing all pg dumps in a grafana panel, to see all dates?

• parse datetime when they were created (part of the filename)

• expose this information in grafana dashboard

• create an alert if there is no database backup done in the past X days (parametrizable)

• expose information about the number of dumps available

• alert if the database dumps go bellow certain number or reach certain number (parametrizable)

  • indicates that the dumps are not properly cleaned or they are dropped

[codificat]

/sig observability
/unassign @pacospace
/remove-lifecycle active

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[mayaCostantini]

/remove lifecycle-stale
/lifecycle frozen

[VannTen]

Thanks @fridex, what about having metrics in https://github.com/thoth-station/graph-backup-job that tell us if dump was create or not?

Is that really a safe way ? I mean, we want a metric about the state, not the action right ?
@goern still priority critical ?

[codificat]

/remove-priority critical-urgent
/priority important-longterm