Add database_observability.mysql component (#2053)

* [draft] Add dbo11y component

WIP

* basic impl of Run and Update

* export targets from component

* scaffolding for collectors

* default args

* refactor ConnectionInfo

* docs

* rename

* use instanceKey

* db engine in component name

* import logic for schema_table collector

* Add logic for QuerySample collector

* Add test for QuerySample collector

* add test for SchemaTable collector

* add test for ConnectionInfo collector

* Fix logging msgs

Co-authored-by: MattNolf <[email protected]>

* add index.md

* review feedback: add goleak verification to tests

* add example to docs

* Call Stop when exiting Run loop

* Add more logging before exiting loop

* rename Run into Start

* add todo about possible refactoring

* fix tests after rename

* Update internal/component/database_observability/mysql/collector/connection_info.go

Co-authored-by: William Dumont <[email protected]>

* fixes

* rename component

* rename to collect_interval

* no need to save tmp new args

* Update docs/sources/reference/components/database_observability/database_observability.mysql.md

Co-authored-by: Clayton Cornell <[email protected]>

* extend docs structure

* Update docs/sources/reference/components/database_observability/database_observability.mysql.md

Co-authored-by: Clayton Cornell <[email protected]>

* fix table

* review docs again

* review shutdown

---------

Co-authored-by: MattNolf <[email protected]>
Co-authored-by: William Dumont <[email protected]>
Co-authored-by: Clayton Cornell <[email protected]>

Author: 4 people

Diff for: docs/sources/reference/compatibility/_index.md

@@ -43,6 +43,10 @@ The following components, grouped by namespace, _export_ Targets.
 - [beyla.ebpf](../components/beyla/beyla.ebpf)
 {{< /collapse >}}
 
+{{< collapse title="database_observability" >}}
+- [database_observability.mysql](../components/database_observability/database_observability.mysql)
+{{< /collapse >}}
+
 {{< collapse title="discovery" >}}
 - [discovery.azure](../components/discovery/discovery.azure)
 - [discovery.consul](../components/discovery/discovery.consul)
@@ -236,6 +240,10 @@ The following components, grouped by namespace, _consume_ Loki `LogsReceiver`.
 
 <!-- START GENERATED SECTION: CONSUMERS OF Loki `LogsReceiver` -->
 
+{{< collapse title="database_observability" >}}
+- [database_observability.mysql](../components/database_observability/database_observability.mysql)
+{{< /collapse >}}
+
 {{< collapse title="faro" >}}
 - [faro.receiver](../components/faro/faro.receiver)
 {{< /collapse >}}

Diff for: docs/sources/reference/components/database_observability/_index.md

@@ -0,0 +1,12 @@
+---
+canonical: https://grafana.com/docs/alloy/latest/reference/components/database_observability/
+description: Learn about the database_observability components in Grafana Alloy
+title: database_observability
+weight: 100
+---
+
+# database_observability
+
+This section contains reference documentation for the `database_observability` components.
+
+{{< section >}}

Diff for: docs/sources/reference/components/database_observability/database_observability.mysql.md

@@ -0,0 +1,88 @@
+---
+canonical: https://grafana.com/docs/alloy/latest/reference/components/database_observability.mysql/
+description: Learn about database_observability.mysql
+title: database_observability.mysql
+---
+
+<span class="badge docs-labels__stage docs-labels__item">Experimental</span>
+
+# database_observability.mysql
+
+{{< docs/shared lookup="stability/experimental.md" source="alloy" version="<ALLOY_VERSION>" >}}
+
+## Usage
+
+```alloy
+database_observability.mysql "LABEL" {
+  data_source_name = DATA_SOURCE_NAME
+  forward_to       = [LOKI_RECEIVERS]
+}
+```
+
+## Arguments
+
+The following arguments are supported:
+
+| Name                 | Type           | Description                                                                                                         | Default | Required |
+| -------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------- | ------- | -------- |
+| `data_source_name` | `secret`             | [Data Source Name](https://github.com/go-sql-driver/mysql#dsn-data-source-name) for the MySQL server to connect to.                   |         | yes      |
+| `forward_to`       | `list(LogsReceiver)` | Where to forward log entries after processing.     |         | yes      |
+| `collect_interval` | `duration`           | How frequently to collect query samples from database | `"10s"` | no       |
+
+## Blocks
+
+The `database_observability.mysql` component does not support any blocks, and is configured fully through arguments.
+
+## Example
+
+```alloy
+database_observability.mysql "orders_db" {
+  data_source_name = "user:pass@mysql:3306/"
+  forward_to = [loki.write.logs_service.receiver]
+}
+
+prometheus.scrape "orders_db" {
+  targets = database_observability.mysql.orders_db.targets
+  honor_labels = true // required to keep job and instance labels
+  forward_to = [prometheus.remote_write.metrics_service.receiver]
+}
+
+prometheus.remote_write "metrics_service" {
+  endpoint {
+    url = sys.env("GCLOUD_HOSTED_METRICS_URL")
+    basic_auth {
+      username = sys.env("GCLOUD_HOSTED_METRICS_ID")
+      password = sys.env("GCLOUD_RW_API_KEY")
+    }
+  }
+}
+
+loki.write "logs_service" {
+  endpoint {
+    url = sys.env("GCLOUD_HOSTED_LOGS_URL")
+    basic_auth {
+      username = sys.env("GCLOUD_HOSTED_LOGS_ID")
+      password = sys.env("GCLOUD_RW_API_KEY")
+    }
+  }
+}
+```
+
+<!-- START GENERATED COMPATIBLE COMPONENTS -->
+
+## Compatible components
+
+`database_observability.mysql` can accept arguments from the following components:
+
+- Components that export [Loki `LogsReceiver`](../../../compatibility/#loki-logsreceiver-exporters)
+
+`database_observability.mysql` has exports that can be consumed by the following components:
+
+- Components that consume [Targets](../../../compatibility/#targets-consumers)
+
+{{< admonition type="note" >}}
+Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.
+Refer to the linked documentation for more details.
+{{< /admonition >}}
+
+<!-- END GENERATED COMPATIBLE COMPONENTS -->

Diff for: go.mod

@@ -8,6 +8,7 @@ require (
 	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.13.0
 	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0
 	github.com/Azure/go-autorest/autorest v0.11.29
+	github.com/DATA-DOG/go-sqlmock v1.5.2
 	github.com/IBM/sarama v1.43.3
 	github.com/KimMachineGun/automemlimit v0.6.0
 	github.com/Lusitaniae/apache_exporter v0.11.1-0.20220518131644-f9522724dab4
@@ -768,7 +769,7 @@ require (
 	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
 	github.com/xhit/go-str2duration/v2 v2.1.0 // indirect
 	github.com/xo/dburl v0.20.0 // indirect
-	github.com/xwb1989/sqlparser v0.0.0-20180606152119-120387863bf2 // indirect
+	github.com/xwb1989/sqlparser v0.0.0-20180606152119-120387863bf2
 	github.com/yl2chen/cidranger v1.0.2 // indirect
 	github.com/youmark/pkcs8 v0.0.0-20240424034433-3c2c7870ae76 // indirect
 	github.com/yusufpapurcu/wmi v1.2.4 // indirect

Diff for: go.sum

@@ -1645,6 +1645,7 @@ github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvW
 github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/kisielk/sqlstruct v0.0.0-20201105191214-5f3e10d3ab46/go.mod h1:yyMNCyc/Ib3bDTKd379tNMpB/7/H5TjM2Y9QJ5THLbE=
 github.com/klauspost/asmfmt v1.3.2 h1:4Ri7ox3EwapiOjCki+hw14RyKk201CN4rzyCJRFLpK4=
 github.com/klauspost/asmfmt v1.3.2/go.mod h1:AG8TuvYojzulgDAMCnYn50l/5QV3Bs/tp6j0HLHbNSE=
 github.com/klauspost/compress v1.11.0/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=

Diff for: internal/component/all/all.go

@@ -3,6 +3,7 @@ package all
 
 import (
 	_ "github.com/grafana/alloy/internal/component/beyla/ebpf"                               // Import beyla.ebpf
+	_ "github.com/grafana/alloy/internal/component/database_observability/mysql"             // Import database_observability.mysql
 	_ "github.com/grafana/alloy/internal/component/discovery/aws"                            // Import discovery.aws.ec2 and discovery.aws.lightsail
 	_ "github.com/grafana/alloy/internal/component/discovery/azure"                          // Import discovery.azure
 	_ "github.com/grafana/alloy/internal/component/discovery/consul"                         // Import discovery.consul

Diff for: internal/component/database_observability/mysql/collector/connection_info.go

@@ -0,0 +1,71 @@
+package collector
+
+import (
+	"context"
+	"net"
+	"regexp"
+	"strings"
+
+	"github.com/go-sql-driver/mysql"
+	"github.com/prometheus/client_golang/prometheus"
+)
+
+var rdsRegex = regexp.MustCompile(`(?P<identifier>[^\.]+)\.([^\.]+)\.(?P<region>[^\.]+)\.rds\.amazonaws\.com`)
+
+type ConnectionInfoArguments struct {
+	DSN      string
+	Registry *prometheus.Registry
+}
+
+type ConnectionInfo struct {
+	DSN        string
+	Registry   *prometheus.Registry
+	InfoMetric *prometheus.GaugeVec
+}
+
+func NewConnectionInfo(args ConnectionInfoArguments) (*ConnectionInfo, error) {
+	infoMetric := prometheus.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "connection_info",
+		Help: "Information about the connection",
+	}, []string{"provider_name", "region", "db_instance_identifier"})
+
+	args.Registry.MustRegister(infoMetric)
+
+	return &ConnectionInfo{
+		DSN:        args.DSN,
+		Registry:   args.Registry,
+		InfoMetric: infoMetric,
+	}, nil
+}
+
+func (c *ConnectionInfo) Start(ctx context.Context) error {
+	cfg, err := mysql.ParseDSN(c.DSN)
+	if err != nil {
+		return err
+	}
+
+	var (
+		providerName         = "unknown"
+		providerRegion       = "unknown"
+		dbInstanceIdentifier = "unknown"
+	)
+
+	host, _, err := net.SplitHostPort(cfg.Addr)
+	if err == nil && host != "" {
+		if strings.HasSuffix(host, "rds.amazonaws.com") {
+			providerName = "aws"
+			matches := rdsRegex.FindStringSubmatch(host)
+			if len(matches) > 3 {
+				dbInstanceIdentifier = matches[1]
+				providerRegion = matches[3]
+			}
+		}
+	}
+
+	c.InfoMetric.WithLabelValues(providerName, providerRegion, dbInstanceIdentifier).Set(1)
+	return nil
+}
+
+func (c *ConnectionInfo) Stop() {
+	c.Registry.Unregister(c.InfoMetric)
+}

Diff for: internal/component/database_observability/mysql/collector/connection_info_test.go

@@ -0,0 +1,57 @@
+package collector
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"testing"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/testutil"
+	"github.com/stretchr/testify/require"
+	"go.uber.org/goleak"
+)
+
+func TestConnectionInfo(t *testing.T) {
+	defer goleak.VerifyNone(t)
+
+	const baseExpectedMetrics = `
+	# HELP connection_info Information about the connection
+	# TYPE connection_info gauge
+	connection_info{db_instance_identifier="%s",provider_name="%s",region="%s"} 1
+`
+
+	testCases := []struct {
+		name            string
+		dsn             string
+		expectedMetrics string
+	}{
+		{
+			name:            "generic dsn",
+			dsn:             "user:pass@tcp(localhost:3306)/db",
+			expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "unknown", "unknown", "unknown"),
+		},
+		{
+			name:            "AWS/RDS dsn",
+			dsn:             "user:pass@tcp(products-db.abc123xyz.us-east-1.rds.amazonaws.com:3306)/db",
+			expectedMetrics: fmt.Sprintf(baseExpectedMetrics, "products-db", "aws", "us-east-1"),
+		},
+	}
+
+	for _, tc := range testCases {
+		reg := prometheus.NewRegistry()
+
+		collector, err := NewConnectionInfo(ConnectionInfoArguments{
+			DSN:      tc.dsn,
+			Registry: reg,
+		})
+		require.NoError(t, err)
+		require.NotNil(t, collector)
+
+		err = collector.Start(context.Background())
+		require.NoError(t, err)
+
+		err = testutil.GatherAndCompare(reg, strings.NewReader(tc.expectedMetrics))
+		require.NoError(t, err)
+	}
+}