Skip to content

Commit

Permalink
[receiver/huaweicloudces]: Add huaweicloudces receiver
Browse files Browse the repository at this point in the history
  • Loading branch information
@narcis96
narcis96 committed Nov 7, 2024
1 parent 5032867 commit 9dae0d1
Show file tree
Hide file tree
Showing 11 changed files with 770 additions and 0 deletions.
1 change: 1 addition & 0 deletions receiver/huaweicloudcesreceiver/Makefile
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include ../../Makefile.Common
267 changes: 267 additions & 0 deletions receiver/huaweicloudcesreceiver/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,267 @@
# Huawei Cloud CES Receiver

<!-- status autogenerated section -->
| Status | |
| ------------- |-----------|
| Stability | [development]: metrics |
| Distributions | [] |
| Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fhuaweicloudces%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fhuaweicloudces) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fhuaweicloudces%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fhuaweicloudces) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@heitorganzeli](https://www.github.com/heitorganzeli), [@narcis96](https://www.github.com/narcis96) |

[development]: https://github.com/open-telemetry/opentelemetry-collector#development
<!-- end autogenerated section -->

This receiver contains the implementation of the Huawei Cloud [Cloud Eye Service](https://www.huaweicloud.com/intl/en-us/product/ces.html) (CES) receiver for the OpenTelemetry Collector. The receiver collects metrics from Huawei Cloud's CES service and sends them to the OpenTelemetry Collector for processing and exporting.

## Configuration

The following settings are required:

- `region_id`: The ID of the Huawei Cloud region from which metrics are collected. For example, `eu-west-101`. The full list of the available regions can be found [here](https://pkg.go.dev/github.com/huaweicloud/[email protected]/services/ces/v1/region).

- `project_id`: The ID of the project in Huawei Cloud. This is used to identify which project's metrics are to be collected. See [Obtaining a Project ID](https://support.huaweicloud.com/intl/en-us/devg-apisign/api-sign-provide-proid.html).

- `period`: The aggregation granularity of metrics retrieved from CES in seconds. For details about the aggregation, see [What Is Rollup?](https://support.huaweicloud.com/intl/en-us/ces_faq/ces_faq_0009.html). Possible values are 1, 300, 1200, 3600, 14400, and 86400.
- 1: Cloud Eye performs no aggregation and displays raw data.
- 300: Cloud Eye aggregates data every 5 minutes.
- 1200: Cloud Eye aggregates data every 20 minutes.
- 3600: Cloud Eye aggregates data every hour.
- 14400: Cloud Eye aggregates data every 4 hours.
- 86400: Cloud Eye aggregates data every 24 hours.
- `filter`: The filter field determines the aggregation method used for the metrics. This aggregation is applied to the data points within the specified period. Valid values for filter include:

- `average`: Calculates the average value over the period.
- `min`: Retrieves the minimum value within the period.
- `max`: Retrieves the maximum value within the period.
- `sum`: Computes the sum of all data points within the period.
- `variance`: Calculates the variance (square of the standard deviation) for the data points.

- `no_verify_ssl`: A boolean flag indicating whether SSL verification should be disabled. Set to True to disable SSL verification.

- `access_key`: The access key needed for CES authentification. Check `Huawei Cloud SDK Authentication Setup` section for more details.

- `secret_key`: The secret key needed for CES authentification. Check `Huawei Cloud SDK Authentication Setup` section for more details.

The following settings are optional:

- `initial_delay`: The delay before the first collection of metrics begins. This is a duration field, such as 5s for 5 seconds.

- `collection_interval` (default = `60s`): This is the interval at which this receiver collects metrics. This value must be a string readable by Golang's [time.ParseDuration](https://pkg.go.dev/time#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. We recommend a polling interval of at least one minute.

- `retry_on_failure`: The following configurations can be used to control the retry policy of the CES client. The default values are suitable for most deployment scenarios.
- `enabled` (default true)
- `initial_interval` (default 100ms)
- `max_interval` (default 1s)
- `max_elapsed_time` (default 15s)
- `randomization_factor` (default 0.5)
- `multiplier` (default 1.5)

### Example Configuration

```yaml
receivers:
huaweicloudcesreceiver:
collection_interval: 3h
initial_delay: 5s
region_id: eu-west-101
access_key: ${env:HUAWEICLOUD_SDK_AK}
secret_key: ${env:HUAWEICLOUD_SDK_SK}
project_id: "project_1"
period: 300
filter: average
no_verify_ssl: True
```
The full list of settings exposed for this receiver are documented [here](./config.go).
### Huawei Cloud SDK Authentication Setup
To ensure secure authentication, the Access Key (AK) and Secret Key (SK) used by the Huawei Cloud SDK must be stored in environment variables. See [Obtaining an AK/SK](https://support.huaweicloud.com/intl/en-us/devg-apisign/api-sign-provide-aksk.html).
Before running the application, you need to set the environment variables `HUAWEICLOUD_SDK_AK` and `HUAWEICLOUD_SDK_SK` in your local environment. Here’s how you can do it:

1. Open your terminal.
2. Set the environment variables by executing the following commands:

```sh
export HUAWEICLOUD_SDK_AK=your-access-key
export HUAWEICLOUD_SDK_SK=your-secret-key
```

3. Verify that the variables are set correctly:

```sh
echo $HUAWEICLOUD_SDK_AK
echo $HUAWEICLOUD_SDK_SK
```

## Error handling
If you encounter any errors, please refer to:
- [Huawei Cloud Error Codes](https://support.huaweicloud.com/intl/en-us/devg-apisign/api-sign-errorcode.html)
- [CES Error Codes](https://support.huaweicloud.com/intl/en-us/api-ces/ErrorCode.html)
- [Quota management](https://support.huaweicloud.com/intl/en-us/usermanual-ces/en-us_topic_0154940152.html)


## Converting CES metric representation to Open Telementery metric representation


| Source Field | Target Field | Description |
|--------------------------|----------------------------------------------|-------------------------------------------------------------------------------------------------------|
| **metric_name** | `scoped_metric.metric.name` | The name of the metric. |
| **unit** | `scoped_metric.metric.unit` | The unit of the metric, e.g., `%`, `bit/s`. |
| **datapoints.value** | `scoped_metric.metric.gauge.data_points.value` | The value of the metric at the specified timestamp. |
| **datapoints.timestamp** | `scoped_metric.metric.gauge.data_points.timestamp` | The timestamp of the metric in nanoseconds. Converted from milliseconds by multiplying by 1,000,000. |
| **dimensions.name** | `key of scoped_metric.metric.metadata` | The dimension name stored as an attribute. |
| **dimensions.value** | `value of scoped_metric.metric.metadata` | The dimension value stored as an attribute. |
| **Receiver Config** | `resource.attributes["project.id"]` | The project ID used in the configuration file of the receiver. |
| **Receiver Config** | `resource.attributes["region.id"]` | The region ID used in the configuration file of the receiver. |
| **namespace** | `resource.attributes["service.namespace"]` | The namespace of the metric, e.g., `SYS.VPC`, stored as an attribute with the key `service.namespace`. |
| *N/A* | `resource.attributes["cloud.provider"]` | Set to `"huawei_cloud"` as the cloud provider. |
| *N/A* | `scoped_metric.scope.name` | Set to `"huawei_cloud_ces"` as the scope name. |
| *N/A* | `scoped_metric.scope.version` | Set to `"v1"` as the scope version. |
|

### Notes

- The `timestamp` field in the source is converted from milliseconds to nanoseconds in the target field.
- Some fields are added in the target format with constant values to provide additional context and metadata.

### Example:

```json
[
{
"unit": "%",
"datapoints": [
{ "average": 10, "timestamp": 1722580500000 },
{ "average": 20, "timestamp": 1722580800000 }
],
"namespace": "SYS.ECS",
"metric_name": "cpu_util",
"dimensions": [
{
"name": "instance_id",
"value": "faea5b75-e390-4e2b-8733-9226a9026070"
}
]
},
{
"unit": "%",
"datapoints": [
{ "average": 30, "timestamp": 1722580500000 },
{ "average": 40, "timestamp": 1722580800000 }
],
"namespace": "SYS.ECS",
"metric_name": "mem_util",
"dimensions": [
{
"name": "instance_id",
"value": "abcea5b75-e390-4e2b-8733-9226a9026070"
}
]
},
{
"unit": "bit/s",
"datapoints": [
{ "average": 1024, "timestamp": 1722580500000},
{ "average": 2048, "timestamp": 1722580800000}
],
"namespace": "SYS.VPC",
"metric_name": "upstream_bandwidth_usage",
"dimensions": [
{
"name": "publicip_id",
"value": "test-baae-4dd9-ad3f-1234"
}
]
}
]
```

converts to

```json
{
"resourceMetrics": [
{
"resource": {
"attributes": {
"cloud.provider": "huawei_cloud",
"project.id": "project_1",
"region.id": "eu-west-101",
"system.namespace": "SYS.ECS",
}
},
"scopeMetrics": [
{
"scope": { "name": "huawei_cloud_ces", "version": "v1" },
"metrics": [
{
"name": "cpu_util",
"unit": "%",
"gauge": {
"dataPoints": [
{ "timeUnixNano": "1722580500000000000", "asDouble": 10 },
{ "timeUnixNano": "1722580800000000000", "asDouble": 20 }
]
},
"metadata": {
"instance_id": "faea5b75-e390-4e2b-8733-9226a9026070",
}
}
]
},
{
"scope": { "name": "huawei_cloud_ces", "version": "v1" },
"metrics": [
{
"name": "mem_util",
"unit": "%",
"gauge": {
"dataPoints": [
{ "timeUnixNano": "1722580500000000000", "asDouble": 30 },
{ "timeUnixNano": "1722580800000000000", "asDouble": 40 }
]
},
"metadata": {
"instance_id": "abcea5b75-e390-4e2b-8733-9226a9026070",
}
}
]
}
]
},
{
"resource": {
"attributes": {
"cloud.provider": "huawei_cloud",
"project.id": "project_1",
"region.id": "eu-west-101",
"system.namespace": "SYS.VPC",
}
},
"scopeMetrics": [
{
"scope": { "name": "huawei_cloud_ces", "version": "v1" },
"metrics": [
{
"name": "upstream_bandwidth_usage",
"unit": "bits/s",
"gauge": {
"dataPoints": [
{ "timeUnixNano": "1722580500000000000", "asDouble": 1024 },
{ "timeUnixNano": "1722580800000000000", "asDouble": 2048 }
]
},
"metadata": {
"publicip_id": "test-baae-4dd9-ad3f-1234",
}
}
]
}
]
}
]
}
```
75 changes: 75 additions & 0 deletions receiver/huaweicloudcesreceiver/config.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package huaweicloudcesreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/huaweicloudcesreceiver"

import (
"errors"

"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/config/configopaque"
"go.opentelemetry.io/collector/config/configretry"
"go.opentelemetry.io/collector/receiver/scraperhelper"
)

var (
// Predefined error responses for configuration validation failures
errInvalidCollectionInterval = errors.New(`invalid period; must be less than "collection_interval"`)
errMissingProjectID = errors.New(`"project_id" is not specified in config`)
errMissingRegionID = errors.New(`"region_id" is not specified in config`)
errInvalidProxy = errors.New(`"proxy_address" must be specified if "proxy_user" or "proxy_password" is set"`)
)

// Config represent a configuration for the CloudWatch logs exporter.
type Config struct {
scraperhelper.ControllerConfig `mapstructure:",squash"`
confighttp.ClientConfig `mapstructure:",squash"`
// Nodes defines the nodes to scrape.
// See https://www.elastic.co/guide/en/elasticsearch/reference/7.9/cluster.html#cluster-nodes for which selectors may be used here.
// If Nodes is empty, no nodes will be scraped.
Nodes []string `mapstructure:"nodes"`
// SkipClusterMetrics indicates whether cluster level metrics from /_cluster/* endpoints should be scraped or not.
SkipClusterMetrics bool `mapstructure:"skip_cluster_metrics"`
// Indices defines the indices to scrape.
// See https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#index-stats-api-path-params
// for which names are viable.
// If Indices is empty, no indices will be scraped.
Indices []string `mapstructure:"indices"`
// Set of attributes used to configure huawei's CES SDK connection
huaweiSessionConfig `mapstructure:",squash"`

// ProjectID is a string to reference project where metrics should be associated with.
// If ProjectID is not filled in, the SDK will automatically call the IAM service to query the project id corresponding to the region.
ProjectID string `mapstructure:"project_id"`

// RegionID is the ID of the CES region.
RegionID string `mapstructure:"region_id"`

// How retrieved data from Cloud Eye is aggregated.
// Possible values are 1, 300, 1200, 3600, 14400, and 86400.
// 1: Cloud Eye performs no aggregation and displays raw data.
// 300: Cloud Eye aggregates data every 5 minutes.
// 1200: Cloud Eye aggregates data every 20 minutes.
// 3600: Cloud Eye aggregates data every hour.
// 14400: Cloud Eye aggregates data every 4 hours.
// 86400: Cloud Eye aggregates data every 24 hours.
// For details about the aggregation, see https://support.huaweicloud.com/intl/en-us/ces_faq/ces_faq_0009.html
Period int32 `mapstructure:"period"`

// Data aggregation method. The supported values ​​are max, min, average, sum, variance.
Filter string `mapstructure:"filter"`

BackOffConfig configretry.BackOffConfig `mapstructure:"retry_on_failure"`
}

type huaweiSessionConfig struct {
AccessKey configopaque.String `mapstructure:"access_key"`

SecretKey configopaque.String `mapstructure:"secret_key"`
// Number of seconds before timing out a request.
NoVerifySSL bool `mapstructure:"no_verify_ssl"`
// Upload segments to AWS X-Ray through a proxy.
ProxyAddress string `mapstructure:"proxy_address"`
ProxyUser string `mapstructure:"proxy_user"`
ProxyPassword string `mapstructure:"proxy_password"`
}
48 changes: 48 additions & 0 deletions receiver/huaweicloudcesreceiver/factory.go
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package huaweicloudcesreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/huaweicloudcesreceiver"

import (
"context"
"time"

"github.com/cenkalti/backoff/v4"
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/configretry"
"go.opentelemetry.io/collector/consumer"
"go.opentelemetry.io/collector/receiver"

"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/huaweicloudcesreceiver/internal/metadata"
)

func NewFactory() receiver.Factory {
return receiver.NewFactory(
metadata.Type,
createDefaultConfig,
receiver.WithMetrics(createMetricsReceiver, metadata.MetricsStability))
}

func createDefaultConfig() component.Config {
return &Config{
BackOffConfig: configretry.BackOffConfig{
Enabled: true,
InitialInterval: 100 * time.Millisecond,
MaxInterval: time.Second,
MaxElapsedTime: 15 * time.Second,
RandomizationFactor: backoff.DefaultRandomizationFactor,
Multiplier: backoff.DefaultMultiplier,
},
huaweiSessionConfig: huaweiSessionConfig{
NoVerifySSL: false,
},
}
}

func createMetricsReceiver(
_ context.Context,
params receiver.Settings,
cfg component.Config,
next consumer.Metrics) (receiver.Metrics, error) {
return nil, nil
}
Loading

0 comments on commit 9dae0d1

Please sign in to comment.