Skip to content

PMM-14324 PMM HA monitoring #4683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 30 commits into
base: pmm-ha
Choose a base branch
from
Draft

PMM-14324 PMM HA monitoring #4683

wants to merge 30 commits into from
+382 −203

Conversation

@ademidoff
Copy link
Member

@ademidoff ademidoff commented Oct 18, 2025 •
edited
Loading

PMM-14324

Link to the Feature Build: SUBMODULES-4079

@ademidoff ademidoff mentioned this pull request Oct 18, 2025
Draft
@ademidoff ademidoff force-pushed the PMM-14324-pmm-ha-monitoring branch from e0bfaf9 to c52753c Compare October 19, 2025 11:09
@codecov
Copy link

codecov bot commented Oct 19, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 10.00000% with 54 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.68%. Comparing base (0ebddc2) to head (0793177).

Files with missing lines Patch % Lines
...anaged/services/victoriametrics/victoriametrics.go 23.52% 13 Missing ⚠️
managed/services/vmalert/vmalert.go 0.00% 10 Missing ⚠️
managed/services/ha/highavailability.go 0.00% 9 Missing ⚠️
managed/services/victoriametrics/prometheus.go 0.00% 5 Missing and 1 partial ⚠️
managed/cmd/pmm-managed/main.go 0.00% 4 Missing ⚠️
managed/models/database.go 20.00% 4 Missing ⚠️
managed/utils/envvars/parser.go 25.00% 3 Missing ⚠️
managed/services/alerting/service.go 0.00% 2 Missing ⚠️
managed/services/ha/services.go 0.00% 2 Missing ⚠️
managed/services/agents/state.go 0.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           pmm-ha    #4683      +/-   ##
==========================================
- Coverage   45.69%   45.68%   -0.02%     
==========================================
  Files         364      364              
  Lines       37826    37810      -16     
==========================================
- Hits        17286    17274      -12     
+ Misses      18883    18874       -9     
- Partials     1657     1662       +5
Flag Coverage Δ
managed 46.15% <10.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ademidoff ademidoff force-pushed the PMM-14324-pmm-ha-monitoring branch 4 times, most recently from 6c06bf1 to 8e9a690 Compare October 20, 2025 20:24
@ademidoff ademidoff mentioned this pull request Oct 26, 2025
Merged
@ademidoff ademidoff force-pushed the PMM-14324-pmm-ha-monitoring branch from a9ea40b to 9fe8ae5 Compare October 29, 2025 22:15
ademidoff and others added 5 commits October 30, 2025 01:17
@ademidoff
Merge branch 'pmm-ha' into PMM-14324-pmm-ha-monitoring
bc3c7ae 
commit a55a9d1
Author: Alex Demidoff <[email protected]>
Date:   Thu Oct 30 00:37:16 2025 +0300

    PMM-14305 Reapply fix ClickHouse dirty migration state

commit 0c578b5
Merge: 92a59a2 4a62c3e
Author: Alex Demidoff <[email protected]>
Date:   Thu Oct 30 00:06:06 2025 +0300

    Merge branch 'v3' into pmm-ha

commit 4a62c3e
Author: Alex Demidoff <[email protected]>
Date:   Wed Oct 29 19:01:25 2025 +0300

    PMM-7 Rpmbuild maintenance (#4703)

commit 92a59a2
Author: Jiří Čtvrtka <[email protected]>
Date:   Wed Oct 29 10:29:13 2025 +0100

    PMM-13812 HA ClickHouse improvements without driver (#4714)

    * PMM-13812 Fix fir templateFS driver.

    * PMM-13812 Include custom cluster name in check for engine.

    * PMM-13812 Lint.

    * PMM-13812 Lint.

    * PMM-13812 Remove methods not explicitly needed by driver.

    * PMM-13812 Lint.

    * PMM-13812 Lint.

    * PMM-13812 Lint.

    * PMM-13812 Checking isCluster everytime before wait on ready, refactor.

    * PMM-13812 Including DB in ReplicatedMergeTree, small refactor.

    * PMM-13812 Lint.

    * PMM-13812 Test.

    * PMM-13812 Refactor TemplateFS to support directory specification and enhance file reading.

    * PMM-13812 Refactor TemplateFS tests to simplify directory handling and improve error assertions.

    * PMM-13812 Fix tests.

    * PMM-13812 Format.

    * PMM-13812 Lint.

    * PMM-13812 Lint.

    * PMM-13812 Remove extra lines.

    ---------

    Co-authored-by: Nurlan Moldomurov <[email protected]>

commit 7e3e667
Author: Alex Demidoff <[email protected]>
Date:   Wed Oct 29 09:46:59 2025 +0300

    PMM-10461 [Fix] No absolute Nginx redirects (#4711)

commit 7a21c3e
Author: Alex Demidoff <[email protected]>
Date:   Tue Oct 28 12:30:02 2025 +0300

    PMM-14429 Provide ability to generate the encryption key (#4707)

    * PMM-14429 Provide ability to generate the encryption key

    * PMM-14429 Fix linter errors

    * PMM-14429 Add a default value for --generate-key param

    * PMM-14429 Remove the verision print, which is already handled by --help

    * PMM-14429 Initialize Encryption instance lazily

    * PMM-14429 Fix linter errors

    * PMM-14429 Remove default value for --generate-key

commit 98c594a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Oct 28 10:13:59 2025 +0200

    Bump actions/upload-artifact from 4.6.2 to 5.0.0 (#4708)

    Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.6.2 to 5.0.0.
    - [Release notes](https://github.com/actions/upload-artifact/releases)
    - [Commits](actions/upload-artifact@ea165f8...330a01c)

    ---
    updated-dependencies:
    - dependency-name: actions/upload-artifact
      dependency-version: 5.0.0
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 06b252a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Oct 28 07:57:55 2025 +0100

    Bump anchore/sbom-action from 0.20.8 to 0.20.9 (#4704)

    Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.20.8 to 0.20.9.
    - [Release notes](https://github.com/anchore/sbom-action/releases)
    - [Changelog](https://github.com/anchore/sbom-action/blob/main/RELEASE.md)
    - [Commits](anchore/sbom-action@aa0e114...8e94d75)

    ---
    updated-dependencies:
    - dependency-name: anchore/sbom-action
      dependency-version: 0.20.9
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 041cacb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Oct 27 23:32:57 2025 +0300

    Bump github/codeql-action from 4.30.9 to 4.31.0 (#4709)

    Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.30.9 to 4.31.0.
    - [Release notes](https://github.com/github/codeql-action/releases)
    - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
    - [Commits](github/codeql-action@16140ae...4e94bd1)

    ---
    updated-dependencies:
    - dependency-name: github/codeql-action
      dependency-version: 4.31.0
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 86f5dae
Author: Alex Demidoff <[email protected]>
Date:   Mon Oct 27 14:18:09 2025 +0300

    PMM-14305 Fix ClickHouse dirty migration state (#4651)

    * PMM-14305 Fix ClickHouse dirty state

    * PMM-14305 Refactor err handling

    * PMM-14305 Fix linter errors

    * PMM-14305 Migrate to stdlib errors pkg

    * PMM-14305 Remove a redundant method

    * PMM-14305 Fix an adge case when the first migration fails

    * PMM-14305 Minor update

commit f6a51f7
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Oct 27 14:17:31 2025 +0300

    Bump golang.org/x/tools from 0.36.0 to 0.38.0 (#4669)

    Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.36.0 to 0.38.0.
    - [Release notes](https://github.com/golang/tools/releases)
    - [Commits](golang/tools@v0.36.0...v0.38.0)

    ---
    updated-dependencies:
    - dependency-name: golang.org/x/tools
      dependency-version: 0.38.0
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Percona CLA Assistant. <[email protected]>
    Co-authored-by: Alex Demidoff <[email protected]>

commit 6b1aa1c
Author: Talha Bin Rizwan <[email protected]>
Date:   Mon Oct 27 13:27:34 2025 +0500

    PMM-14401 Install eksctl and helm cli tools in Jenkins agent. (#4676)

    * PMM-14401 Install eksctl and helm cli tools.

    * PMM-14401 Update eksctl and helm cli versions to latest stable.

    * PMM-14401 Update kubectl cli version to 1.33.5

    * PMM-7 Download and run SSM

    * PMM-14401 Fix the names and tags

    * PMM-14401 Grant ssm-agent higher privileges

    * PMM-14401 Set SELinux context for ssm-user

    ---------

    Co-authored-by: Alex Demidoff <[email protected]>

commit 085ceaa
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Oct 27 10:16:41 2025 +0200

    Bump golang.org/x/sys from 0.36.0 to 0.37.0 (#4668)

    Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.36.0 to 0.37.0.
    - [Commits](golang/sys@v0.36.0...v0.37.0)

    ---
    updated-dependencies:
    - dependency-name: golang.org/x/sys
      dependency-version: 0.37.0
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c5ea2b7
Merge: 9ceab4f 8de1d05
Author: Alex Demidoff <[email protected]>
Date:   Mon Oct 27 09:48:18 2025 +0300

    Merge branch 'v3' into pmm-ha

commit 8de1d05
Author: Alex Demidoff <[email protected]>
Date:   Thu Oct 23 21:21:19 2025 +0300

    PMM-7 Update dependabot config (#4696)

    * PMM-7 Update dependabot config

    * PMM-7 Bump Go version to 1.24.7

commit 42e3c8c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Oct 23 14:25:12 2025 +0300

    Bump vite in /ui in the npm_and_yarn group across 1 directory (#4694)

    Bumps the npm_and_yarn group with 1 update in the /ui directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite).

    Updates `vite` from 5.4.20 to 5.4.21
    - [Release notes](https://github.com/vitejs/vite/releases)
    - [Changelog](https://github.com/vitejs/vite/blob/v5.4.21/packages/vite/CHANGELOG.md)
    - [Commits](https://github.com/vitejs/vite/commits/v5.4.21/packages/vite)

    ---
    updated-dependencies:
    - dependency-name: vite
      dependency-version: 5.4.21
      dependency-type: direct:development
      dependency-group: npm_and_yarn
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Matej Kubinec <[email protected]>

commit 69a0193
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Oct 23 09:53:41 2025 +0300

    Bump github.com/apache/skywalking-eyes from 0.7.0 to 0.8.0 in /tools (#4685)

    Bumps [github.com/apache/skywalking-eyes](https://github.com/apache/skywalking-eyes) from 0.7.0 to 0.8.0.
    - [Release notes](https://github.com/apache/skywalking-eyes/releases)
    - [Changelog](https://github.com/apache/skywalking-eyes/blob/main/CHANGES.md)
    - [Commits](apache/skywalking-eyes@v0.7.0...v0.8.0)

    ---
    updated-dependencies:
    - dependency-name: github.com/apache/skywalking-eyes
      dependency-version: 0.8.0
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 48f6581
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Oct 23 09:27:06 2025 +0300

    Bump github/codeql-action from 4.30.8 to 4.30.9 (#4684)

    Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.30.8 to 4.30.9.
    - [Release notes](https://github.com/github/codeql-action/releases)
    - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
    - [Commits](github/codeql-action@f443b60...16140ae)

    ---
    updated-dependencies:
    - dependency-name: github/codeql-action
      dependency-version: 4.30.9
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c06a50d
Author: Alex Demidoff <[email protected]>
Date:   Thu Oct 23 01:23:31 2025 +0300

    PMM-10461 No absolute Nginx redirects (#4576)

    * PMM-10461 No absolute Nginx redirects

    * PMM-10461 Simplify swagger redirect

    * PMM-10461 Drop "permanent" from the rewrite rule for Grafana

commit f595bf5
Author: Jiří Čtvrtka <[email protected]>
Date:   Wed Oct 22 17:48:42 2025 +0200

    PMM-7 Oracle 9.5+ (#4699)

commit cee22bb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Oct 22 00:57:08 2025 +0300

    Bump golang.org/x/text from 0.29.0 to 0.30.0 (#4667)

    Bumps [golang.org/x/text](https://github.com/golang/text) from 0.29.0 to 0.30.0.
    - [Release notes](https://github.com/golang/text/releases)
    - [Commits](golang/text@v0.29.0...v0.30.0)

    ---
    updated-dependencies:
    - dependency-name: golang.org/x/text
      dependency-version: 0.30.0
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 862b363
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Oct 22 00:01:58 2025 +0300

    Bump github.com/bufbuild/buf from 1.58.0 to 1.59.0 in /tools (#4692)

    Bumps [github.com/bufbuild/buf](https://github.com/bufbuild/buf) from 1.58.0 to 1.59.0.
    - [Release notes](https://github.com/bufbuild/buf/releases)
    - [Changelog](https://github.com/bufbuild/buf/blob/main/CHANGELOG.md)
    - [Commits](bufbuild/buf@v1.58.0...v1.59.0)

    ---
    updated-dependencies:
    - dependency-name: github.com/bufbuild/buf
      dependency-version: 1.59.0
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit e3f6de2
Author: Tibor Korocz <[email protected]>
Date:   Tue Oct 21 14:17:16 2025 +0200

    PMM-14157 - Update MongoDB PBM backup alert template to use max_over_… (#4690)

    * PMM-14157 - Update MongoDB PBM backup alert template to use max_over_time for last successful backup check

    - Modified the expression to calculate the age of the last successful backup using max_over_time for better accuracy over a 30-day window.
    - Updated the summary and description annotations to include cluster and replica set information for clearer context in alerts.

    * Refine MongoDB PBM backup alert template to simplify expression by removing replica_set grouping. This change enhances clarity in the alerting logic while maintaining the focus on cluster-level backup status.

    * Refine MongoDB PBM backup alert template by removing replica_set information from summary and description annotations. This change enhances clarity and focuses on the cluster-level backup status.

commit 2b105b1
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Oct 19 19:17:04 2025 +0100

    Bump umbrelladocs/action-linkspector from 1.3.7 to 1.4.0 (#4674)

    Bumps [umbrelladocs/action-linkspector](https://github.com/umbrelladocs/action-linkspector) from 1.3.7 to 1.4.0.
    - [Release notes](https://github.com/umbrelladocs/action-linkspector/releases)
    - [Commits](UmbrellaDocs/action-linkspector@874d01c...652f85b)

    ---
    updated-dependencies:
    - dependency-name: umbrelladocs/action-linkspector
      dependency-version: 1.4.0
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit f04104a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Oct 19 11:33:27 2025 +0200

    Bump actions/setup-node from 5 to 6 (#4666)

    Bumps [actions/setup-node](https://github.com/actions/setup-node) from 5 to 6.
    - [Release notes](https://github.com/actions/setup-node/releases)
    - [Commits](actions/setup-node@v5...v6)

    ---
    updated-dependencies:
    - dependency-name: actions/setup-node
      dependency-version: '6'
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c5dd364
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sat Oct 18 07:15:07 2025 +0000

    Bump anchore/sbom-action from 0.20.6 to 0.20.8 (#4680)

    Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.20.6 to 0.20.8.
    - [Release notes](https://github.com/anchore/sbom-action/releases)
    - [Changelog](https://github.com/anchore/sbom-action/blob/main/RELEASE.md)
    - [Commits](anchore/sbom-action@f8bdd1d...aa0e114)

    ---
    updated-dependencies:
    - dependency-name: anchore/sbom-action
      dependency-version: 0.20.8
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 13c4fd3
Author: Jiří Čtvrtka <[email protected]>
Date:   Fri Oct 17 12:29:06 2025 +0200

    PMM-13103 Doc for MySQL partial certs. (#4659)
@ademidoff
Merge branch 'pmm-ha' into PMM-14324-pmm-ha-monitoring
dfe82d5
@ademidoff
Merge branch 'pmm-ha' into PMM-14324-pmm-ha-monitoring
d00c8e6
@ademidoff
Merge branch 'PMM-14324-pmm-ha-monitoring' of ssh://github.com/percon…
835cdb5 
…a/pmm into PMM-14324-pmm-ha-monitoring
@ademidoff
PMM-14387 Fix error message formatting
8ad0160
@ademidoff ademidoff force-pushed the PMM-14324-pmm-ha-monitoring branch from cf1f8fd to 8ad0160 Compare November 7, 2025 18:01
@ademidoff ademidoff force-pushed the PMM-14324-pmm-ha-monitoring branch from f9ea7fe to 5b84063 Compare November 8, 2025 09:50
@ademidoff ademidoff force-pushed the PMM-14324-pmm-ha-monitoring branch 2 times, most recently from 94871c7 to 5009cfa Compare November 10, 2025 00:14
@ademidoff ademidoff force-pushed the PMM-14324-pmm-ha-monitoring branch from 5009cfa to 1c7e707 Compare November 10, 2025 08:49
@ademidoff ademidoff force-pushed the PMM-14324-pmm-ha-monitoring branch from c759faa to 4246d79 Compare November 11, 2025 01:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ademidoff