diff --git a/packages/chargeback/_dev/build/docs/README.md b/packages/chargeback/_dev/build/docs/README.md index b2b8ab95268..a21f691b820 100755 --- a/packages/chargeback/_dev/build/docs/README.md +++ b/packages/chargeback/_dev/build/docs/README.md @@ -95,31 +95,17 @@ The integration creates the following transforms to aggregate cost and usage dat These transforms produce lookup indices that are queried by the dashboard using ES|QL LOOKUP JOINs to correlate billing costs with actual usage patterns. -### Starting the Transforms +### Transform Auto-Start -After installing the integration, you need to manually start the four usage-related transforms: +All Chargeback transforms start automatically when the integration is installed. No manual intervention is required to start the transforms. -1. Navigate to **Stack Management → Transforms** -2. Filter for `chargeback` to see all Chargeback transforms -3. Start the following transforms: - - `cluster_deployment_contribution` - - `cluster_datastream_contribution` - - `cluster_tier_contribution` - - `cluster_tier_and_ds_contribution` +**Performance Note:** On clusters with months of historical monitoring data for multiple deployments, the initial transform execution may process a large volume of data. This can cause temporary performance impact during the first run. The transforms will then run incrementally on their configured schedules (15-60 minute intervals), processing only new data with minimal overhead. -The `billing_cluster_cost` transform starts automatically and does not require manual intervention. +You can verify the transforms are running by navigating to **Stack Management → Transforms** and filtering for `chargeback`. ### Transform Health Monitoring -To set up alerts that notify you when transforms are not working: - -1. Navigate to **Stack Management → Transforms** -2. Filter for `chargeback` to see all Chargeback transforms -3. Select a transform and click the **Actions** menu -4. Select **Create alert rule** -5. Configure the alert rule to notify when the transform health status changes - -This will create a transform health rule that monitors the selected transform and sends notifications when issues are detected. +The integration includes a **Transform Health Monitoring** alert rule template that can be installed from the integration page. This rule monitors all Chargeback transforms and alerts when they encounter issues or failures, providing proactive notification of any problems with data processing. ## Dashboard @@ -148,80 +134,15 @@ The `billing_cluster_cost` transform automatically extracts these tags from the **Note:** Each deployment should have only one `chargeback_group` tag. Having multiple tags can cause issues and lead to unpredictable cost allocation. -## Observability Rules - -The following are sample observability rules that can help ensure data validity by notifying you when events occur that could compromise the accuracy of your chargeback data: - -### Rule 1: New Chargeback Group Detected - -Detects when a new `chargeback_group` tag is added to a deployment, allowing teams to be notified when new cost allocation groups are created. - -**To create this alert**, navigate to **Dev Tools** in Kibana and run: -```json -POST kbn:/api/alerting/rule/chargeback_new_group_detected -{ - "name": "[Chargeback] New chargeback group detected", - "tags": ["Chargeback"], - "consumer": "alerts", - "rule_type_id": ".es-query", - "schedule": { - "interval": "1h" - }, - "params": { - "size": 100, - "esqlQuery": { - "esql": "FROM billing_cluster_cost_lookup | STATS count = COUNT(*) BY deployment_group | SORT deployment_group | KEEP deployment_group" - }, - "threshold": [0], - "timeField": "@timestamp", - "searchType": "esqlQuery", - "timeWindowSize": 3, - "timeWindowUnit": "d", - "thresholdComparator": ">", - "excludeHitsFromPreviousRun": true - }, - "actions": [] -} -``` +## Observability Alerting -### Rule 2: Deployment with Chargeback Group Missing Usage Data +This integration includes 3 pre-configured alert rule templates that can be installed directly from the integration page in Kibana: -Detects when a deployment has a chargeback group assigned but is not sending usage/consumption data. This indicates a potential configuration issue or data collection problem. +1. **Transform Health Monitoring** - Monitors the health of all Chargeback transforms and alerts when they encounter issues or failures +2. **New Chargeback Group Detected** - Notifies when a new `chargeback_group` tag is added to a deployment +3. **Deployment with Chargeback Group Missing Usage Data** - Detects when a deployment has a chargeback group assigned but is not sending usage/consumption data -**To create this alert**, navigate to **Dev Tools** in Kibana and run: -```json -POST kbn:/api/alerting/rule/chargeback_deployment_missing_usage_data -{ - "name": "[Chargeback] Deployment with chargeback group missing usage data", - "tags": ["Chargeback"], - "consumer": "alerts", - "rule_type_id": ".es-query", - "schedule": { - "interval": "1h" - }, - "params": { - "size": 100, - "esqlQuery": { - "esql": """FROM billing_cluster_cost_lookup -| WHERE deployment_group != "" -| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key -| WHERE cluster_name IS NULL -| INLINE STATS count = COUNT(*) BY deployment_id, deployment_name, deployment_group -| EVAL result = CONCAT("Deployment `", deployment_name,"` (`", deployment_id,"`) in deployment group `", deployment_group, "` did not have usage data since ", left(composite_key,10),".") -| STATS result = VALUES(result) -| MV_EXPAND result""" - }, - "threshold": [0], - "timeField": "@timestamp", - "searchType": "esqlQuery", - "timeWindowSize": 3, - "timeWindowUnit": "d", - "thresholdComparator": ">", - "excludeHitsFromPreviousRun": true - }, - "actions": [] -} -``` +**Important:** For alert rules 2 and 3, ensure that the Chargeback transforms are running before setting them up. These alerting rules query the lookup indices created by the transforms (`billing_cluster_cost_lookup`, `cluster_deployment_contribution_lookup`, etc.). If the transforms are not started, the alerts will not function correctly. ### Alert actions diff --git a/packages/chargeback/changelog.yml b/packages/chargeback/changelog.yml index bb12d8af9df..c747da7d351 100644 --- a/packages/chargeback/changelog.yml +++ b/packages/chargeback/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: 0.2.8 + changes: + - description: "Add Kibana alerting rule templates for transform health monitoring, detecting new chargeback groups, and identifying deployments with missing usage data. Templates can be installed directly from the package. All transforms now auto-start on installation." + type: enhancement + link: https://github.com/elastic/integrations/pull/16229 - version: 0.2.7 changes: - description: "Add observability alerting rule templates and documentation for monitoring new chargeback groups and missing usage data. Update Elasticsearch version requirement to 9.2.0+ for smart lookup join support." diff --git a/packages/chargeback/docs/README.md b/packages/chargeback/docs/README.md index f66fb208b4e..a3ab4167eaa 100644 --- a/packages/chargeback/docs/README.md +++ b/packages/chargeback/docs/README.md @@ -95,31 +95,17 @@ The integration creates the following transforms to aggregate cost and usage dat These transforms produce lookup indices that are queried by the dashboard using ES|QL LOOKUP JOINs to correlate billing costs with actual usage patterns. -### Starting the Transforms +### Transform Auto-Start -After installing the integration, you need to manually start the four usage-related transforms: +All Chargeback transforms start automatically when the integration is installed. No manual intervention is required to start the transforms. -1. Navigate to **Stack Management → Transforms** -2. Filter for `chargeback` to see all Chargeback transforms -3. Start the following transforms: - - `cluster_deployment_contribution` - - `cluster_datastream_contribution` - - `cluster_tier_contribution` - - `cluster_tier_and_ds_contribution` +**Performance Note:** On clusters with months of historical monitoring data for multiple deployments, the initial transform execution may process a large volume of data. This can cause temporary performance impact during the first run. The transforms will then run incrementally on their configured schedules (15-60 minute intervals), processing only new data with minimal overhead. -The `billing_cluster_cost` transform starts automatically and does not require manual intervention. +You can verify the transforms are running by navigating to **Stack Management → Transforms** and filtering for `chargeback`. ### Transform Health Monitoring -To set up alerts that notify you when transforms are not working: - -1. Navigate to **Stack Management → Transforms** -2. Filter for `chargeback` to see all Chargeback transforms -3. Select a transform and click the **Actions** menu -4. Select **Create alert rule** -5. Configure the alert rule to notify when the transform health status changes - -This will create a transform health rule that monitors the selected transform and sends notifications when issues are detected. +The integration includes a **Transform Health Monitoring** alert rule template that can be installed from the integration page. This rule monitors all Chargeback transforms and alerts when they encounter issues or failures, providing proactive notification of any problems with data processing. ## Dashboard @@ -148,80 +134,15 @@ The `billing_cluster_cost` transform automatically extracts these tags from the **Note:** Each deployment should have only one `chargeback_group` tag. Having multiple tags can cause issues and lead to unpredictable cost allocation. -## Observability Rules - -The following are sample observability rules that can help ensure data validity by notifying you when events occur that could compromise the accuracy of your chargeback data: - -### Rule 1: New Chargeback Group Detected - -Detects when a new `chargeback_group` tag is added to a deployment, allowing teams to be notified when new cost allocation groups are created. - -**To create this alert**, navigate to **Dev Tools** in Kibana and run: -```json -POST kbn:/api/alerting/rule/chargeback_new_group_detected -{ - "name": "[Chargeback] New chargeback group detected", - "tags": ["Chargeback"], - "consumer": "alerts", - "rule_type_id": ".es-query", - "schedule": { - "interval": "1h" - }, - "params": { - "size": 100, - "esqlQuery": { - "esql": "FROM billing_cluster_cost_lookup | STATS count = COUNT(*) BY deployment_group | SORT deployment_group | KEEP deployment_group" - }, - "threshold": [0], - "timeField": "@timestamp", - "searchType": "esqlQuery", - "timeWindowSize": 3, - "timeWindowUnit": "d", - "thresholdComparator": ">", - "excludeHitsFromPreviousRun": true - }, - "actions": [] -} -``` +## Observability Alerting -### Rule 2: Deployment with Chargeback Group Missing Usage Data +This integration includes 3 pre-configured alert rule templates that can be installed directly from the integration page in Kibana: -Detects when a deployment has a chargeback group assigned but is not sending usage/consumption data. This indicates a potential configuration issue or data collection problem. +1. **Transform Health Monitoring** - Monitors the health of all Chargeback transforms and alerts when they encounter issues or failures +2. **New Chargeback Group Detected** - Notifies when a new `chargeback_group` tag is added to a deployment +3. **Deployment with Chargeback Group Missing Usage Data** - Detects when a deployment has a chargeback group assigned but is not sending usage/consumption data -**To create this alert**, navigate to **Dev Tools** in Kibana and run: -```json -POST kbn:/api/alerting/rule/chargeback_deployment_missing_usage_data -{ - "name": "[Chargeback] Deployment with chargeback group missing usage data", - "tags": ["Chargeback"], - "consumer": "alerts", - "rule_type_id": ".es-query", - "schedule": { - "interval": "1h" - }, - "params": { - "size": 100, - "esqlQuery": { - "esql": """FROM billing_cluster_cost_lookup -| WHERE deployment_group != "" -| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key -| WHERE cluster_name IS NULL -| INLINE STATS count = COUNT(*) BY deployment_id, deployment_name, deployment_group -| EVAL result = CONCAT("Deployment `", deployment_name,"` (`", deployment_id,"`) in deployment group `", deployment_group, "` did not have usage data since ", left(composite_key,10),".") -| STATS result = VALUES(result) -| MV_EXPAND result""" - }, - "threshold": [0], - "timeField": "@timestamp", - "searchType": "esqlQuery", - "timeWindowSize": 3, - "timeWindowUnit": "d", - "thresholdComparator": ">", - "excludeHitsFromPreviousRun": true - }, - "actions": [] -} -``` +**Important:** For alert rules 2 and 3, ensure that the Chargeback transforms are running before setting them up. These alerting rules query the lookup indices created by the transforms (`billing_cluster_cost_lookup`, `cluster_deployment_contribution_lookup`, etc.). If the transforms are not started, the alerts will not function correctly. ### Alert actions diff --git a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml index 3bc79b887ba..c1ad6780fe1 100644 --- a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml +++ b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml @@ -22,7 +22,7 @@ source: emit(''); dest: index: billing_cluster_cost_lookup - pipeline: 0.2.7-billing + pipeline: 0.2.8-billing frequency: 60m sync: time: diff --git a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/manifest.yml b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/manifest.yml index 88be0537be7..ccaa87d98f1 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/manifest.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/manifest.yml @@ -1,8 +1,8 @@ -start: false +start: true destination_index_template: settings: index: - mode: "lookup" - codec: best_compression + mode: "lookup" + codec: best_compression mappings: dynamic: false \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml index 2c8d3270dd0..2fb6d5ba3cf 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml @@ -4,7 +4,7 @@ source: - monitoring-indices # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. dest: index: cluster_datastream_contribution_lookup - pipeline: 0.2.7-usage + pipeline: 0.2.8-usage frequency: 60m sync: time: diff --git a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/manifest.yml b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/manifest.yml index 889c9565cac..ea03fc70dcb 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/manifest.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/manifest.yml @@ -1,4 +1,4 @@ -start: false +start: true destination_index_template: settings: index: diff --git a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml index 3dd5cf97cb1..939e478739f 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml @@ -4,7 +4,7 @@ source: - monitoring-indices # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. dest: index: cluster_deployment_contribution_lookup - pipeline: 0.2.7-usage + pipeline: 0.2.8-usage frequency: 60m sync: time: diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/manifest.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/manifest.yml index 702df10634b..ccaa87d98f1 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/manifest.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/manifest.yml @@ -1,4 +1,4 @@ -start: false +start: true destination_index_template: settings: index: diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml index 77881d59e97..371b5c16a11 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml @@ -4,7 +4,7 @@ source: - monitoring-indices # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. dest: index: cluster_tier_and_datastream_contribution_lookup - pipeline: 0.2.7-usage + pipeline: 0.2.8-usage frequency: 60m sync: time: diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/manifest.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/manifest.yml index bb260785b20..bf148d40424 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/manifest.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/manifest.yml @@ -1,4 +1,4 @@ -start: false +start: true destination_index_template: settings: index: diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml index 61d102dd35b..739d072c985 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml @@ -4,7 +4,7 @@ source: - monitoring-indices # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. dest: index: cluster_tier_contribution_lookup - pipeline: 0.2.7-usage + pipeline: 0.2.8-usage frequency: 60m sync: time: diff --git a/packages/chargeback/kibana/alerting_rule_template/chargeback-deployment-missing-usage-data.json b/packages/chargeback/kibana/alerting_rule_template/chargeback-deployment-missing-usage-data.json new file mode 100644 index 00000000000..f3ed9a70802 --- /dev/null +++ b/packages/chargeback/kibana/alerting_rule_template/chargeback-deployment-missing-usage-data.json @@ -0,0 +1,25 @@ +{ + "id": "chargeback-deployment-missing-usage-data", + "type": "alerting_rule_template", + "attributes": { + "name": "[Chargeback] Deployment with chargeback group missing usage data", + "tags": ["Chargeback"], + "ruleTypeId": ".es-query", + "schedule": { + "interval": "1h" + }, + "params": { + "searchType": "esqlQuery", + "timeWindowSize": 3, + "timeWindowUnit": "d", + "esqlQuery": { + "esql": "FROM billing_cluster_cost_lookup\n| WHERE deployment_group != \"\"\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| WHERE cluster_name IS NULL\n| INLINE STATS count = COUNT(*) BY deployment_id, deployment_name, deployment_group\n| EVAL result = CONCAT(\"Deployment `\", deployment_name,\"` (`\", deployment_id,\"`) in deployment group `\", deployment_group, \"` did not have usage data since \", left(composite_key,10),\".\")\n| STATS result = VALUES(result)\n| MV_EXPAND result" + }, + "size": 100, + "threshold": [0], + "thresholdComparator": ">", + "timeField": "@timestamp", + "excludeHitsFromPreviousRun": true + } + } +} diff --git a/packages/chargeback/kibana/alerting_rule_template/chargeback-new-group-detected.json b/packages/chargeback/kibana/alerting_rule_template/chargeback-new-group-detected.json new file mode 100644 index 00000000000..cd79993ecfd --- /dev/null +++ b/packages/chargeback/kibana/alerting_rule_template/chargeback-new-group-detected.json @@ -0,0 +1,25 @@ +{ + "id": "chargeback-new-group-detected", + "type": "alerting_rule_template", + "attributes": { + "name": "[Chargeback] New chargeback group detected", + "tags": ["Chargeback"], + "ruleTypeId": ".es-query", + "schedule": { + "interval": "1h" + }, + "params": { + "searchType": "esqlQuery", + "timeWindowSize": 3, + "timeWindowUnit": "d", + "esqlQuery": { + "esql": "FROM billing_cluster_cost_lookup | STATS count = COUNT(*) BY deployment_group | SORT deployment_group | KEEP deployment_group" + }, + "size": 100, + "threshold": [0], + "thresholdComparator": ">", + "timeField": "@timestamp", + "excludeHitsFromPreviousRun": true + } + } +} diff --git a/packages/chargeback/kibana/alerting_rule_template/chargeback-transform-health.json b/packages/chargeback/kibana/alerting_rule_template/chargeback-transform-health.json new file mode 100644 index 00000000000..1e19bba294a --- /dev/null +++ b/packages/chargeback/kibana/alerting_rule_template/chargeback-transform-health.json @@ -0,0 +1,26 @@ +{ + "id": "chargeback-transform-health", + "type": "alerting_rule_template", + "attributes": { + "name": "[Chargeback] Transform health monitoring", + "tags": ["Chargeback", "Transforms"], + "ruleTypeId": "transform_health", + "schedule": { + "interval": "1m" + }, + "params": { + "includeTransforms": [ + "logs-chargeback.billing_cluster_cost-*", + "logs-chargeback.cluster_deployment_contribution-*", + "logs-chargeback.cluster_datastream_contribution-*", + "logs-chargeback.cluster_tier_contribution-*", + "logs-chargeback.cluster_tier_and_ds_contribution-*" + ], + "excludeTransforms": null, + "testsConfig": null + }, + "alertDelay": { + "active": 1 + } + } +} diff --git a/packages/chargeback/manifest.yml b/packages/chargeback/manifest.yml index ff7270d44fc..c7b070d19c1 100644 --- a/packages/chargeback/manifest.yml +++ b/packages/chargeback/manifest.yml @@ -1,7 +1,7 @@ format_version: 3.4.0 name: chargeback title: "Chargeback" -version: 0.2.7 +version: 0.2.8 description: "This package calculates chargeback based on billing and consumption data" type: integration categories: