Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(alerting): Add Incident.io alerting provider #972

Merged
merged 27 commits into from
Feb 7, 2025
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
bb1ffa9
feat(alerting): added incident.io provider alerting.
ImTheCurse Jan 20, 2025
7e814c5
Tests: added incident.io provider unit tests.
ImTheCurse Jan 20, 2025
fd1b09b
Documentation: added incidentio documentation.
ImTheCurse Jan 20, 2025
caa98f0
Refactor: Changed documentation + types to an alphabetical order.
ImTheCurse Jan 21, 2025
767ad43
Merge branch 'master' into feat-incident-io
ImTheCurse Jan 21, 2025
8ae19b4
Refactor: change wrong comment.
ImTheCurse Jan 22, 2025
60975c5
Update README.md
ImTheCurse Jan 22, 2025
621fae5
Update alerting/provider/incidentio/incident_io.go
ImTheCurse Jan 22, 2025
51693ae
Update alerting/provider/incidentio/incident_io.go
ImTheCurse Jan 22, 2025
41b6723
Update alerting/provider/incidentio/incident_io.go
ImTheCurse Jan 22, 2025
d9b5129
Update alerting/provider/incidentio/incident_io.go
ImTheCurse Jan 22, 2025
888e891
Refactor: changed alertSourceID to url.
ImTheCurse Jan 25, 2025
8754bee
Refactor: changed documentation.
ImTheCurse Jan 25, 2025
9634886
Refactor: refactored tests, removed status from config.
ImTheCurse Jan 25, 2025
2c46f0a
Readme: updated docs.
ImTheCurse Jan 25, 2025
dabbec5
Refactor: removed duplication key in favor of ResolveKey.
ImTheCurse Jan 26, 2025
ba047a0
Refactor: change variable format.
ImTheCurse Jan 26, 2025
cf28ca2
Feat + Test: added support for passing metadata and source url, added…
ImTheCurse Jan 27, 2025
3284c2d
Merge branch 'master' into feat-incident-io
TwiN Feb 6, 2025
851ffa3
Refactor: chaned variable naming
ImTheCurse Feb 6, 2025
426183a
Merge branch 'master' into feat-incident-io
ImTheCurse Feb 6, 2025
70e8ad5
Update alerting/config.go
TwiN Feb 6, 2025
dc03d61
Update README.md
TwiN Feb 6, 2025
6684218
Update README.md
TwiN Feb 7, 2025
0b3bda0
Update README.md
TwiN Feb 7, 2025
a144588
Apply suggestions from code review
TwiN Feb 7, 2025
6c516f9
Refactor: sort var by abc
ImTheCurse Feb 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ Have any feedback or questions? [Create a discussion](https://github.com/TwiN/ga
- [Configuring GitLab alerts](#configuring-gitlab-alerts)
- [Configuring Google Chat alerts](#configuring-google-chat-alerts)
- [Configuring Gotify alerts](#configuring-gotify-alerts)
- [Configuring Incident.io alerts](#configuring-incidentio-alerts)
- [Configuring JetBrains Space alerts](#configuring-jetbrains-space-alerts)
- [Configuring Matrix alerts](#configuring-matrix-alerts)
- [Configuring Mattermost alerts](#configuring-mattermost-alerts)
Expand Down Expand Up @@ -579,6 +580,7 @@ endpoints:
| `alerting.gitlab` | Configuration for alerts of type `gitlab`. <br />See [Configuring GitLab alerts](#configuring-gitlab-alerts). | `{}` |
| `alerting.googlechat` | Configuration for alerts of type `googlechat`. <br />See [Configuring Google Chat alerts](#configuring-google-chat-alerts). | `{}` |
| `alerting.gotify` | Configuration for alerts of type `gotify`. <br />See [Configuring Gotify alerts](#configuring-gotify-alerts). | `{}` |
| `alerting.incident-io` | Configuration for alerts of type `incidentio`. <br />See [Configuring Incident.io alerts](#configuring-incidentio-alerts). | `{}` |
| `alerting.jetbrainsspace` | Configuration for alerts of type `jetbrainsspace`. <br />See [Configuring JetBrains Space alerts](#configuring-jetbrains-space-alerts). | `{}` |
| `alerting.matrix` | Configuration for alerts of type `matrix`. <br />See [Configuring Matrix alerts](#configuring-matrix-alerts). | `{}` |
| `alerting.mattermost` | Configuration for alerts of type `mattermost`. <br />See [Configuring Mattermost alerts](#configuring-mattermost-alerts). | `{}` |
Expand Down Expand Up @@ -904,6 +906,41 @@ Here's an example of what the notifications look like:

![Gotify notifications](.github/assets/gotify-alerts.png)

#### Configuring Incident.io alerts
| Parameter | Description | Default |
|:-----------------------------------|:-------------------------------------------------------------------------------------------|:--------------|
| `alerting.incident-io` | Configuration for alerts of type `incident-io` | `{}` |
| `alerting.incident-io.url` | url to trigger an alert event. | Required `""` |
| `alerting.incident-io.auth-token` | Token that is used for authentication. | Required `""` |
| `alerting.incident-io.overrides` | List of overrides that may be prioritized over the default configuration | `[]` |
| `alerting.incident-io.default-alert` | Default alert configuration. <br />See [Setting a default alert](#setting-a-default-alert) | N/A |
| `alerting.incident-io.overrides[].group` | Endpoint group for which the configuration will be overridden by this configuration | `""` |
| `alerting.incident-io.overrides[].*` | See `alerting.incident-io.*` parameters | `{}` |

```yaml
alerting:
incident-io:
url: "*****************"
auth-token: "********************************************"

endpoints:
- name: website
url: "https://twin.sh/health"
interval: 30s
conditions:
- "[STATUS] == 200"
- "[BODY].status == UP"
- "[RESPONSE_TIME] < 300"
alerts:
- type: incident-io
description: "healthcheck failed"
send-on-resolved: true
```
in order to get the required alert source config id and authentication token, you must configure an HTTP alert source.

> **_NOTE:_** the source config id is of the form { api.incident.io/v2/alert_events/http/{CONFIG-SOURCE-ID}}

> **_NOTE:_** the auth token is of the form { "Authorization": "Bearer {AUTH-TOKEN}" }

#### Configuring JetBrains Space alerts
| Parameter | Description | Default |
Expand Down Expand Up @@ -1258,6 +1295,7 @@ Here's an example of what the notifications look like:
![Slack notifications](.github/assets/slack-alerts.png)



#### Configuring Teams alerts *(Deprecated)*

> [!CAUTION]
Expand Down
3 changes: 3 additions & 0 deletions alerting/alert/type.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ const (
// TypeGotify is the Type for the gotify alerting provider
TypeGotify Type = "gotify"

// TypeIncidentio is the Type for the incident.io alerting provider
TypeIncidentIO Type = "incident-io"

// TypeJetBrainsSpace is the Type for the jetbrains alerting provider
TypeJetBrainsSpace Type = "jetbrainsspace"

Expand Down
4 changes: 4 additions & 0 deletions alerting/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/TwiN/gatus/v5/alerting/provider/gitlab"
"github.com/TwiN/gatus/v5/alerting/provider/googlechat"
"github.com/TwiN/gatus/v5/alerting/provider/gotify"
"github.com/TwiN/gatus/v5/alerting/provider/incidentio"
"github.com/TwiN/gatus/v5/alerting/provider/jetbrainsspace"
"github.com/TwiN/gatus/v5/alerting/provider/matrix"
"github.com/TwiN/gatus/v5/alerting/provider/mattermost"
Expand Down Expand Up @@ -61,6 +62,9 @@ type Config struct {
// Gotify is the configuration for the gotify alerting provider
Gotify *gotify.AlertProvider `yaml:"gotify,omitempty"`

// IncidentIo is the configuration for the incident.io alerting provider
IncidentIO *incidentio.AlertProvider `yaml:"incident-io,omitempty"`

// JetBrainsSpace is the configuration for the jetbrains space alerting provider
JetBrainsSpace *jetbrainsspace.AlertProvider `yaml:"jetbrainsspace,omitempty"`

Expand Down
207 changes: 207 additions & 0 deletions alerting/provider/incidentio/incident_io.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
package incidentio

import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strconv"
"strings"

"github.com/TwiN/gatus/v5/alerting/alert"
"github.com/TwiN/gatus/v5/client"
"github.com/TwiN/gatus/v5/config/endpoint"
"github.com/TwiN/logr"
"gopkg.in/yaml.v3"
)

const (
restAPIUrl = "https://api.incident.io/v2/alert_events/http/"
)

var (
ErrURLNotSet = errors.New("url not set.")
ErrDuplicateGroupOverride = errors.New("duplicate group override")
ErrAuthTokenNotSet = errors.New("auth-token not set.")
)

type Config struct {
URL string `yaml:"url,omitempty"`
AuthToken string `yaml:"auth-token,omitempty"`
SourceURL string `yaml:"source-url,omitempty"`
Metadata map[string]interface{} `yaml:"metadata,omitempty"`
}

func (cfg *Config) Validate() error {
if len(cfg.URL) == 0 {
return ErrURLNotSet
}
if len(cfg.AuthToken) == 0 {
return ErrAuthTokenNotSet
}
return nil
}

func (cfg *Config) Merge(override *Config) {
if len(override.URL) > 0 {
cfg.URL = override.URL
}
if len(override.AuthToken) > 0 {
cfg.AuthToken = override.AuthToken
}
if len(override.SourceURL) > 0 {
cfg.SourceURL = override.SourceURL
}
if len(override.Metadata) > 0 {
cfg.Metadata = override.Metadata
}
}

// AlertProvider is the configuration necessary for sending an alert using incident.io
type AlertProvider struct {
DefaultConfig Config `yaml:",inline"`

// DefaultAlert is the default alert configuration to use for endpoints with an alert of the appropriate type
DefaultAlert *alert.Alert `yaml:"default-alert,omitempty"`

// Overrides is a list of Override that may be prioritized over the default configuration
Overrides []Override `yaml:"overrides,omitempty"`
}

type Override struct {
Group string `yaml:"group"`
Config `yaml:",inline"`
}

func (provider *AlertProvider) Validate() error {
registeredGroups := make(map[string]bool)
if provider.Overrides != nil {
for _, override := range provider.Overrides {
if isAlreadyRegistered := registeredGroups[override.Group]; isAlreadyRegistered || override.Group == "" {
return ErrDuplicateGroupOverride
}
registeredGroups[override.Group] = true
}
}
return provider.DefaultConfig.Validate()
}

func (provider *AlertProvider) Send(ep *endpoint.Endpoint, alert *alert.Alert, result *endpoint.Result, resolved bool) error {
cfg, err := provider.GetConfig(ep.Group, alert)
if err != nil {
return err
}
buffer := bytes.NewBuffer(provider.buildRequestBody(cfg, ep, alert, result, resolved))
req, err := http.NewRequest(http.MethodPost, cfg.URL, buffer)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+cfg.AuthToken)
response, err := client.GetHTTPClient(nil).Do(req)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode > 399 {
body, _ := io.ReadAll(response.Body)
return fmt.Errorf("call to provider alert returned status code %d: %s", response.StatusCode, string(body))
}
incidentioResponse := Response{}
err = json.NewDecoder(response.Body).Decode(&incidentioResponse)
if err != nil {
// Silently fail. We don't want to create tons of alerts just because we failed to parse the body.
logr.Errorf("[incident-io.Send] Ran into error decoding pagerduty response: %s", err.Error())
}
alert.ResolveKey = incidentioResponse.DeduplicationKey
return err
}

type Body struct {
AlertSourceConfigID string `json:"alert_source_config_id"`
Status string `json:"status"`
Title string `json:"title"`
DeduplicationKey string `json:"deduplication_key,omitempty"`
Description string `json:"description,omitempty"`
SourceURL string `json:"source_url,omitempty"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
}

type Response struct {
DeduplicationKey string `json:"deduplication_key"`
}

func (provider *AlertProvider) buildRequestBody(cfg *Config, ep *endpoint.Endpoint, alert *alert.Alert, result *endpoint.Result, resolved bool) []byte {
var message, formattedConditionResults, status string
if resolved {
message = "An alert has been resolved after passing successfully " + strconv.Itoa(alert.SuccessThreshold) + " time(s) in a row"
status = "resolved"
} else {
message = "An alert has been triggered due to having failed " + strconv.Itoa(alert.FailureThreshold) + " time(s) in a row"
status = "firing"
}
for _, conditionResult := range result.ConditionResults {
var prefix string
if conditionResult.Success {
prefix = "🟢"
} else {
prefix = "🔴"
}
// No need for \n since incident.io trims it anyways.
formattedConditionResults += fmt.Sprintf(" %s %s ", prefix, conditionResult.Condition)
}
if len(alert.GetDescription()) > 0 {
message += " with the following description: " + alert.GetDescription()
}

message += fmt.Sprintf(" and the following conditions: %s ", formattedConditionResults)
var body []byte
alertSourceID := strings.Split(cfg.URL, restAPIUrl)[1]
body, _ = json.Marshal(Body{
AlertSourceConfigID: alertSourceID,
Title: "Gatus: " + ep.DisplayName(),
Status: status,
DeduplicationKey: alert.ResolveKey,
Description: message,
SourceURL: cfg.SourceURL,
Metadata: cfg.Metadata,
})
fmt.Printf("%v", string(body))
return body

}
func (provider *AlertProvider) GetConfig(group string, alert *alert.Alert) (*Config, error) {
cfg := provider.DefaultConfig
// Handle group overrides
if provider.Overrides != nil {
for _, override := range provider.Overrides {
if group == override.Group {
cfg.Merge(&override.Config)
break
}
}
}
// Handle alert overrides
if len(alert.ProviderOverride) != 0 {
overrideConfig := Config{}
if err := yaml.Unmarshal(alert.ProviderOverrideAsBytes(), &overrideConfig); err != nil {
return nil, err
}
cfg.Merge(&overrideConfig)
}
// Validate the configuration
err := cfg.Validate()
return &cfg, err
}

// GetDefaultAlert returns the provider's default alert configuration
func (provider *AlertProvider) GetDefaultAlert() *alert.Alert {
return provider.DefaultAlert
}

func (provider *AlertProvider) ValidateOverrides(group string, alert *alert.Alert) error {
_, err := provider.GetConfig(group, alert)
return err
}
Loading