New API Reporter for Http (ex: Grafana) (#3540)

* New API Reporter for Http (ex: Grafana)

Fixes #3531

* Config variables

* API_REPORTER_DEBUG

* delete launch

* test

* Fix ApiReporter

* fix mypy

* fix gitpod

* Formatting

* Fix

* Use search_parent_directories=True to get Repo

* Revert "Use search_parent_directories=True to get Repo"

This reverts commit 379f67f.

* Murf

* Add skip-checkout: true

* Fix actionlint test case

* GO

* test case

* revert

* format

* trvy

* alpha action

* use ghcr

* fix identifiers & git stuff

* get repo name

* Fix github branch name

* Fix gitIdentifier

* check val

* Handle sfdx-hardis config format

* [automation] Auto-update linters version, help and documentation

* [MegaLinter] Apply linters fixes

* Fix actionlint test case

* disable checkmake

* Fix actionlint

* Fix alpha

* test method for get_first_var_set

* Fix variable names

* Fix org identifier and add job url

* Start of metrics payload management

* Add metrics endpoint

* Lint fixes

* lint fix

* flake8

* fixes

* [automation] Auto-update linters version, help and documentation

* [MegaLinter] Apply linters fixes

* Remove swiftlint deprecated --path argument

* Fix docker issues

* build

* Remove deprecated linters

  - CSS_SCSSLINT: [Project discontinued and advising to use stylelint](https://github.com/sds/scss-lint#notice-consider-other-tools-before-adopting-scss-lint)
  - OPENAPI_SPECTRAL: Replaced by API_SPECTRAL (same linter but more formats handled)

* Remove sqllint & dependencies

* changelog

* Add ApiReporter & grafana config in doc

* Update doc

* Test with Alpha

* Temp link to alpha from python flavor

* Fix :)

* cspell

* format md table

* Improve logs

* fix

* logs

* factorize

* Use api_metrics_url

* metrics_payload

* metrics format

* do not send metrics as json

* more metrics keys + docs

* Yeah

* cspell

* Doc

* format

* Rollback tests

* runner

---------

Co-authored-by: nvuillam <[email protected]>

.cspell.json

@@ -32,14 +32,17 @@
         "KNXNX0kxdddddddoc",
         "Koning",
         "Ksection",
+        "NOTIF",
         "OXXXXKd",
         "Paren",
         "Prego",
         "Programmez",
         "QIDAQAB",
+        "REPOSITORYNAME",
         "RQASWB",
         "SCSSLINT",
-        "Stéphane",
+        "St\u00e9phane",
+        "SOURCEBRANCHNAME",
         "Ywarn",
         "abefhkmnptuvx",
         "admiralawkbar",
@@ -252,6 +255,7 @@
         "Csrf",
         "C\u00e9dric",
         "DARTANALYZER",
+        "DDHG",
         "DEVSKIM",
         "DIRC",
         "DJLINT",
@@ -280,10 +284,13 @@
         "Edouard",
         "Envir",
         "Etienne",
+        "FHDHGDH",
         "FILEIO",
         "FIRSTPARTY",
         "Fougerouse",
+        "Fswg",
         "GHSA",
+        "GHTRGDHD",
         "GOBIN",
         "GOLANGCI",
         "GOODCHECK",
@@ -304,9 +311,13 @@
         "Hadolint",
         "Hardcoding",
         "HashiCorp",
+        "Hdhghg",
+        "Hfgfgjfh",
         "Hosom",
         "HyphDash",
         "ISORT",
+        "Iiwib",
+        "Ijoi",
         "Intellij",
         "Isort",
         "Ivar",
@@ -356,6 +367,7 @@
         "Ncss",
         "Nextflow",
         "Nonwords",
+        "OCIs",
         "OPENAPI",
         "OPTIONNAME",
         "Omeed",
@@ -402,6 +414,7 @@
         "Protolint",
         "Pylint",
         "Pytest",
+        "QESRDTHFKGKH",
         "R2DevOps",
         "RAKU",
         "RAKULIB",
@@ -470,6 +483,7 @@
         "TFLINT",
         "THIRDPARTY",
         "TIBANNA",
+        "TODOFROMHERE",
         "TRUFFLEHOG",
         "TSQL",
         "TSQLLINT",
@@ -498,6 +512,7 @@
         "YAMLLINT",
         "YMLs",
         "YOURBRANCH",
+        "YOURORGNAME",
         "YOURUSERNAME",
         "abhith",
         "absolutized",
@@ -837,6 +852,7 @@
         "goodwithtech",
         "gosec",
         "govet",
+        "grafanacloud",
         "gridftp",
         "grimmjo",
         "groovylintrc",
@@ -1222,6 +1238,7 @@
         "pyyaml",
         "qsub",
         "qualit\u00e9",
+        "queryparam",
         "quickfixes",
         "quickstart",
         "qzkc",
@@ -1258,6 +1275,7 @@
         "returncode",
         "returnrules",
         "rexec",
+        "rgba",
         "risd",
         "rmfamily",
         "rockspec",
@@ -1310,6 +1328,7 @@
         "setslice",
         "setuptools",
         "sfdx",
+        "sfdxhardis",
         "sgerrand",
         "shellcheck",
         "shellcheckrc",
@@ -1416,6 +1435,7 @@
         "thirdparty",
         "tibanna",
         "tikzpicture",
+        "timepicker",
         "timonwong",
         "tipa",
         "tlsv",

.github/workflows/mega-linter.yml

@@ -67,7 +67,7 @@ jobs:
 
         # You can override MegaLinter flavor used to have faster performances
         # More info at https://megalinter.io/flavors/
-        uses: oxsecurity/megalinter/flavors/python@beta # ml workflow change
+        uses: oxsecurity/megalinter/flavors/python@beta
 
         id: ml
 
@@ -97,6 +97,16 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           EMAIL_REPORTER_SMTP_PASSWORD: ${{ secrets.SMTP_PASSWORD }} # ml workflow addtion
 
+          # Uncomment to use ApiReporter (Grafana)
+          API_REPORTER: true
+          API_REPORTER_URL: ${{ secrets.API_REPORTER_URL }}
+          API_REPORTER_BASIC_AUTH_USERNAME: ${{ secrets.API_REPORTER_BASIC_AUTH_USERNAME }}
+          API_REPORTER_BASIC_AUTH_PASSWORD: ${{ secrets.API_REPORTER_BASIC_AUTH_PASSWORD }}
+          API_REPORTER_METRICS_URL: ${{ secrets.API_REPORTER_METRICS_URL }}
+          API_REPORTER_METRICS_BASIC_AUTH_USERNAME: ${{ secrets.API_REPORTER_METRICS_BASIC_AUTH_USERNAME }}
+          API_REPORTER_METRICS_BASIC_AUTH_PASSWORD: ${{ secrets.API_REPORTER_METRICS_BASIC_AUTH_PASSWORD }}
+          API_REPORTER_DEBUG: false
+
           # ADD YOUR CUSTOM ENV VARIABLES HERE OR DEFINE THEM IN A FILE
           # .mega-linter.yml AT THE ROOT OF YOUR REPOSITORY
 

.vscode/launch.json

@@ -5,13 +5,10 @@
     "version": "0.2.0",
     "configurations": [
         {
-            "name": "Attach by Process ID",
-            "processId": "${command:PickProcess}",
+            "name": "Python Debugger: Attach using Process Id",
+            "type": "debugpy",
             "request": "attach",
-            "skipFiles": [
-                "<node_internals>/**"
-            ],
-            "type": "node"
+            "processId": "${command:pickProcess}"
         },
         {
             "name": "Python Debugger: Current File",

CHANGELOG.md

@@ -27,6 +27,7 @@ Note: Can be used with `oxsecurity/megalinter@beta` in your GitHub Action mega-l
 - Linters enhancements
 
 - Reporters
+  - New **ApiReporter** (can be used to build Grafana dashboards)
 
 - Fixes
   - [terrascan](https://runterrascan.io/) fixed errors and removed redundant code
@@ -44,6 +45,7 @@ Note: Can be used with `oxsecurity/megalinter@beta` in your GitHub Action mega-l
     - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data
     - Port Beta workflows to use docker/metadata-action, by @echoix
   - AutoUpdate linters: Always create a PR if the job has been started manually
+  - Add `skip_checkout: true` to default MegaLinter GitHub Action template
 
 - Linter versions upgrades
   - [protolint](https://github.com/yoheimuta/protolint) from 0.50.2 to **0.50.3** on 2024-07-07
@@ -304,8 +306,6 @@ Note: Can be used with `oxsecurity/megalinter@beta` in your GitHub Action mega-l
   - `KOTLIN_KTLINT` now supports `list_of_files` mode, and has better error counting
   - Upgrade `KOTLIN_DETEKT` and make it work with cli_lint_mode = project
 
-- Reporters
-
 - Fixes
   - Change `golangci-lint` lint mode to `project`, by @wandering-tales in <https://github.com/oxsecurity/megalinter/pull/3509>
   - Disable sql-lint as it is no longer maintained

README.md

@@ -1299,6 +1299,7 @@ MegaLinter can generate various reports that you can activate / deactivate and c
 | [Gitlab Merge Request comments](https://github.com/oxsecurity/megalinter/tree/main/docs/reporters/GitlabCommentReporter.md)        | Mega-Linter posts a comment on the MR with a summary of lint results, and links to detailed logs              | Active if in Gitlab CI       |
 | [Azure Pipelines Pull Request comments](https://github.com/oxsecurity/megalinter/tree/main/docs/reporters/AzureCommentReporter.md) | Mega-Linter posts a comment on the PR with a summary of lint results, and links to detailed logs              | Active if in Azure Pipelines |
 | [Bitbucket Pull Request comments](https://github.com/oxsecurity/megalinter/tree/main/docs/reporters/BitbucketCommentReporter.md)   | Mega-Linter posts a comment on the PR with a summary of lint results, and links to detailed logs              | Active if in Bitbucket CI    |
+| [API (Grafana)](https://github.com/oxsecurity/megalinter/tree/main/docs/reporters/ApiReporter.md)                                  | Sends logs and metrics to Grafana endpoint (Loki / Prometheus)                                                | Inactive                     |
 | [Updated sources](https://github.com/oxsecurity/megalinter/tree/main/docs/reporters/UpdatedSourcesReporter.md)                     | Zip containing **all formatted and autofixed sources** so you can extract them in your repository             | Active                       |
 | [IDE Configuration](https://github.com/oxsecurity/megalinter/tree/main/docs/reporters/ConfigReporter.md)                           | Apply MegaLinter configuration in your local IDE with linter config files and IDE extensions                  | Active                       |
 | [GitHub Status](https://github.com/oxsecurity/megalinter/tree/main/docs/reporters/GitHubStatusReporter.md)                         | One GitHub status by linter on the PR, with links to detailed logs                                            | Active if GitHub Action      |

docs/config-variables-security.md

@@ -58,6 +58,12 @@ SECURED_ENV_VARIABLES_DEFAULT contains:
 - CI_JOB_TOKEN
 - GITLAB_ACCESS_TOKEN_MEGALINTER
 - GITLAB_CUSTOM_CERTIFICATE
+- API_REPORTER_BEARER_TOKEN
+- API_REPORTER_BASIC_AUTH_USERNAME
+- API_REPORTER_BASIC_AUTH_PASSWORD
+- API_REPORTER_METRICS_BEARER_TOKEN
+- API_REPORTER_METRICS_BASIC_AUTH_USERNAME
+- API_REPORTER_METRICS_BASIC_AUTH_PASSWORD
 - WEBHOOK_REPORTER_BEARER_TOKEN
 - NODE_TOKEN
 - NPM_TOKEN