Add central management (#310)

* Add main page

* Add SDK edits

* Other edits

* Add file to toc

* Update central-configuration.md

* Update docs/reference/central-configuration.md

Co-authored-by: jackshirazi <[email protected]>

* Update EDOT Python docs for central configuration

* Update docs

* Add supported settings

* Edits

* Add feedback

* Add config tables

* Update central-configuration.md

* Add sentence

* Absolute links

* Add applies to for PHP

* Add applies to tags

* Remove iOS

* iOS comment

* Fix word

* Update docs/reference/central-configuration.md

Co-authored-by: Aleksandra Spilkowska <[email protected]>

* Update docs/reference/central-configuration.md

Co-authored-by: Aleksandra Spilkowska <[email protected]>

* Update docs/reference/central-configuration.md

Co-authored-by: Aleksandra Spilkowska <[email protected]>

* Update docs/reference/edot-collector/config/default-config-standalone.md

Co-authored-by: Aleksandra Spilkowska <[email protected]>

* Update docs/reference/edot-sdks/ios/configuration.md

Co-authored-by: Aleksandra Spilkowska <[email protected]>

---------

Co-authored-by: jackshirazi <[email protected]>
Co-authored-by: Riccardo Magliocchetti <[email protected]>
Co-authored-by: Aleksandra Spilkowska <[email protected]>

Author: 4 people

docs/reference/central-configuration.md

@@ -0,0 +1,125 @@
+---
+navigation_title: Central configuration
+description: Reference documentation for the central configuration of EDOT SDKs.
+applies_to:
+  deployment:
+      ess: preview 9.1
+  stack: preview 9.1
+  serverless: unavailable
+products:
+  - id: observability
+  - id: kibana
+  - id: edot-collector
+---
+
+# Central configuration for EDOT SDKs
+
+Manage {{edot}} (EDOT) SDKs through the APM Agent Central Configuration feature in the Applications UI. Changes are automatically propagated to the deployed [EDOT SDKs](/reference/edot-sdks/index.md). Refer to [APM Agent Central Configuration](docs-content://solutions/observability/apm/apm-agent-central-configuration.md) for more information.
+
+This feature implements the Open Agent Management Protocol (OpAMP). Refer to [Open Agent Management Protocol
+](https://opentelemetry.io/docs/specs/opamp/) for more information.
+
+## Prerequisites
+
+To use APM Agent Central Configuration for EDOT SDKs, you need:
+
+* An Elastic self-managed or {{ecloud}} deployment, version 9.1 or higher.
+* A standalone [EDOT Collector](/reference/edot-collector/index.md), in either Agent or Collector mode.
+* [EDOT SDKs](/reference/edot-sdks/index.md) instrumenting your application.
+
+The following versions of EDOT and {{stack}} support central configuration:
+
+| Component | Minimum version |
+|-----------|----------------|
+| Kibana | 9.1 or higher |
+| EDOT Collector | 8.19, 9.1 or higher |
+| EDOT Android | 1.2.0 or higher |
+| EDOT Java | 1.5.0 or higher |
+| EDOT Node.js | 1.2.0 or higher |
+| EDOT PHP | 1.1.1 or higher |
+| EDOT Python | 1.4.0 or higher |
+
+::::{note}
+Serverless deployments are not currently supported.
+::::
+
+## Activate central configuration
+
+To activate APM Agent Central Configuration for EDOT SDKs, follow these steps.
+
+:::::{stepper}
+
+::::{step} Edit the EDOT Collector configuration
+
+Edit the EDOT Collector configuration file to use the `apmconfig` extension. You need a valid {{es}} API key to authenticate to the {{es}} endpoint. 
+
+:::{include} _snippets/retrieve-credentials.md
+:::
+
+The example configuration is:
+
+```yaml
+extensions:
+  bearertokenauth:
+    scheme: "APIKey"
+    token: "<ENCODED_ELASTICSEARCH_APIKEY>"
+
+  apmconfig:
+    opamp:
+      protocols:
+        http:
+          # Default is localhost:4320
+          # endpoint: "<CUSTOM_OPAMP_ENDPOINT>"
+    source:
+      elasticsearch:
+        endpoint: "<ELASTICSEARCH_ENDPOINT>"
+        auth:
+          authenticator: bearertokenauth
+```
+
+Restart the Elastic Agent to also restart the Collector and apply the changes. Refer to [EDOT Collector configuration](/reference/edot-collector/config/default-config-standalone.md#central-configuration) for more information.
+
+::::
+
+::::{step} Set the environment variable for the SDKs
+
+Activate the central configuration feature in the SDKs by setting the `ELASTIC_OTEL_OPAMP_ENDPOINT` environment variable to the URL endpoint of the `apmconfig` extension that you configured in the previous step. For example:
+
+```sh
+export ELASTIC_OTEL_OPAMP_ENDPOINT="http://localhost:4320"
+```
+
+Restart the instrumented application to apply the changes.
+
+:::{note}
+Central configuration uses the `service.name` and `deployment.environment.name` OpenTelemetry resource attributes to target specific instances with a configuration. If no environment is specified, the central configuration feature will match `All` as the environment.
+:::
+
+::::
+
+::::{step} Check that the EDOT SDK appears in central configuration
+
+Wait some time for the EDOT SDK to appear in {{kib}} under Agent Configuration.
+
+1. Go to **{{kib}}** → **Observability** → **Applications** and select a service.
+2. Select **Settings** and go to **Agent Configuration**.
+
+:::{note}
+Your application must produce and send telemetry data for the EDOT SDK to appear in Agent Configuration.
+:::
+
+::::
+
+:::::{stepper}
+
+## Supported settings
+
+For a list of settings that you can configure through APM Agent Central Configuration, refer to the configuration reference of each EDOT SDK:
+
+- [EDOT Android](/reference/edot-sdks/android/configuration.md)
+- [EDOT Java](/reference/edot-sdks/java/configuration.md#central-configuration)
+- [EDOT Node.js](/reference/edot-sdks/nodejs/configuration.md#central-configuration)
+- [EDOT PHP](/reference/edot-sdks/php/configuration.md#central-configuration)
+- [EDOT Python](/reference/edot-sdks/python/configuration.md#central-configuration)
+
+EDOT iOS currently supports APM Agent Central Configuration through APM Server. Refer to [EDOT iOS configuration](/reference/edot-sdks/ios/configuration.md) for more details.

docs/reference/compatibility/collectors.md

@@ -61,7 +61,7 @@ For information on the compatibility of each Collector component, refer to the [
 
 Non-EDOT distributions of the OTel Collector, such as custom Collector builds, upstream Collector distributions, and so on aren't officially supported through Elastic but are technically compatible ([Compatible]) if they contain the [required OTel Collector components](/reference/edot-collector/custom-collector.md) and are configured like the EDOT Collector.
 
-You can retrieve required components and configuration options from the [sample configuration files](https://github.com/elastic/elastic-agent/tree/v<COLLECTOR_VERSION>/internal/pkg/otel/samples/linux) for the EDOT Collector.
+You can retrieve required components and configuration options from the [example configuration files](https://github.com/elastic/elastic-agent/tree/v<COLLECTOR_VERSION>/internal/pkg/otel/samples/linux) for the EDOT Collector.
 
 For a comparison between EDOT and the upstream OTel, refer to [EDOT compared to upstream OTel](edot-vs-upstream.md).
 

docs/reference/compatibility/edot-vs-upstream.md

@@ -25,6 +25,7 @@ Here are some key differences and considerations when using EDOT compared to ups
 | Integration | Seamless integration with Elastic Stack. | Requires additional configuration for Elastic. |
 | Components | Curated list of components for Elastic Observability. | Generic components that may not support all Elastic features. |
 | Deployment | Easier to deploy with Elastic Stack. | Requires manual setup and configuration. |
+| Central management | Central management of OTel SDKs and Collectors. | No central management. |
 | Compatibility | Fully compatible with Elastic Stack components. | Compatible but may require additional configuration. |
 | Self-managed/ECH | Required for full functionality. | Compatible but without guaranteed support. |
 | Updates | Future updates aligned with Elastic Stack releases. | Updates depend on upstream OpenTelemetry release cycle. |

docs/reference/compatibility/features.md

@@ -38,7 +38,7 @@ The following table shows Elastic features and their level of support and compat
 | Usage of [Time Series Data Streams]                         | [Compatible]     | [Supported]      |
 | **Central Management**                                      | [Incompatible]   | [Not supported]  |
 | Central management of OTel collectors                       | [Incompatible]   | [Not supported]  |
-| Central management of OTel SDKs                             | [Incompatible]   | [Not supported]  |
+| Central management of OTel SDKs                             | [Compatible]     | [Supported]      |
 
 
 ^1^ Refer to [limitations on host metrics](limitations.md#infrastructure-and-host-metrics)

docs/reference/edot-collector/config/default-config-standalone.md

@@ -101,9 +101,9 @@ With the {{motlp}}, there is no need to configure any Elastic-specific component
 
 In Gateway mode, the Collector ingests data from other Collectors running in Agent mode and forwards it to Elastic.
 
-## Sample configuration
+## Example configuration
 
-The following sample configuration files are available for the Gateway mode:
+The following example configuration files are available for the Gateway mode:
 
 | Version | Configuration  |
 |---------|----------------|
@@ -207,6 +207,115 @@ The service section defines separate pipelines for different telemetry types:
 
 Each pipeline connects specific receivers, processors, and exporters to handle different data types appropriately.
 
+
+## Central configuration
+
+The EDOT Collector can be configured to use [APM Agent Central Configuration](docs-content://solutions/observability/apm/apm-agent-central-configuration.md). Refer to [Central configuration docs](/reference/central-configuration.md) for more details.
+
+### Activate the apmconfig extension
+
+To activate the central configuration feature, uncomment the [`apmconfig`](https://github.com/elastic/opentelemetry-collector-components/blob/main/extension/apmconfigextension/README.md) and [`bearertokenauth`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/bearertokenauthextension/README.md) extensions and provide the necessary configuration, or add the configuration manually. For example:
+
+```yaml
+extensions:
+  bearertokenauth:
+    scheme: "APIKey"
+    token: "<ENCODED_ELASTICSEARCH_APIKEY>"
+
+  apmconfig:
+    source:
+     elasticsearch:
+       endpoint: "<ELASTICSEARCH_ENDPOINT>"
+       auth:
+         authenticator: bearertokenauth
+    opamp:
+      protocols:
+        http:
+          # Default is localhost:4320
+          # endpoint: "<CUSTOM_OPAMP_ENDPOINT> 
+```
+
+:::{note}
+Create an API Key following [these instructions](docs-content://deploy-manage/api-keys/elasticsearch-api-keys.md).
+:::
+
+### TLS configuration
+
+You can turn on TLS or mutual TLS to encrypt data in transit between EDOT SDKs and the extension. Refer to [OpenTelemetry TLS server configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#server-configuration) for more details.
+
+For example:
+
+```yaml
+extensions:
+  apmconfig:
+    opamp:
+      protocols:
+        http:
+          endpoint: ":4320"
+          tls:
+            cert_file: server.crt
+            key_file: server.key
+   ...
+```
+
+### Authentication settings
+
+In addition to TLS, you can configure authentication to ensure that only authorized agents can communicate with the extension and retrieve their corresponding remote configurations.
+
+The `apmconfig` extension supports any configauth authenticator. Use the `apikeyauth` extension to authenticate with Elasticsearch API keys:
+
+```yaml
+extensions:
+  apikeyauth:
+    endpoint: "<YOUR_ELASTICSEARCH_ENDPOINT>"
+    application_privileges:
+      - application: "apm"
+        privileges:
+          - "config_agent:read"
+        resources:
+          - "-"
+  apmconfig:
+    opamp:
+      protocols:
+        http:
+          auth:
+            authenticator: apikeyauth
+   ...
+```
+
+The server expects incoming HTTP requests to include an API key with sufficient privileges, using the following header format: `Authorization: ApiKey <base64(id:api_key)>`.
+
+You can create an API key with the minimum required application permissions through {{kib}} by going to **Observability** → **Applications** → **Settings** → **Agent Keys**, or by using the Elasticsearch Security API:
+
+```sh
+POST /_security/api_key
+{
+  "name": "apmconfig-opamp-test-sdk",
+  "metadata": {
+    "application": "apm"
+  },
+  "role_descriptors": {
+    "apm": {
+      "cluster": [],
+      "indices": [],
+      "applications": [
+        {
+          "application": "apm",
+          "privileges": [
+            "config_agent:read"
+          ],
+          "resources": [
+            "*"
+          ]
+        }
+      ],
+      "run_as": [],
+      "metadata": {}
+    }
+  }
+}
+```
+
 [`attributes`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor
 [`filelog`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver
 [`hostmetrics`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver

docs/reference/edot-sdks/features.yml

@@ -352,17 +352,17 @@ features:
       status: not-available
       min_version: 
     Java:
-      status: not-available
-      min_version: 
+      status: tech-preview
+      min_version: 1.5.0
     Node.js:
-      status: not-available
-      min_version: 
+      status: tech-preview
+      min_version: 1.2.0
     PHP:
-      status: not-available
-      min_version: 
+      status: tech-preview
+      min_version: 1.1.0
     Python:
-      status: not-available
-      min_version: 
+      status: tech-preview
+      min_version: 1.4.0
     Android:
       status: not-available
       min_version: 

docs/reference/edot-sdks/ios/configuration.md

@@ -37,7 +37,7 @@ You can configure the `AgentConfigBuilder` with the following functions.
 * **Type:** URL
 * **Default:** nil
 
-The URL host endpoint that handles both OTLP data export as well as Elastic Central Config.
+The URL host endpoint that handles both OTLP data export as well as Elastic Central Configuration.
 This configuration option is deprecated. Use `withExportUrl` instead.
 
 ### `withExportUrl` [withExportUrl]
@@ -198,7 +198,7 @@ Deployment environment is set to `default`. This can be overridden using the `OT
 
 ### Dynamic configuration ![dynamic config](images/dynamic-config.svg "") [dynamic-configuration]
 
-Dynamic configurations are available through the kibana UI and are read by the SDK remotely to apply configuration on all active agents deployed in the field. More info on dynamic configurations can be found in  [agent configurations](docs-content://solutions/observability/apps/apm-agent-central-configuration.md).
+Dynamic configurations are available through the {{kib}} UI and are read by the SDK remotely to apply configuration on all active agents deployed in the field. More info on dynamic configurations can be found in  [agent configurations](docs-content://solutions/observability/apps/apm-agent-central-configuration.md).
 
 #### Recording [recording]
 

docs/reference/edot-sdks/java/configuration.md

@@ -87,6 +87,31 @@ The following settings are available:
 | `ELASTIC_OTEL_JAVA_INSTRUMENTATION_GENAI_EMIT_EVENTS` | value of `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | If set to `true`, the agent will generate log events for OpenAI requests and responses. Potentially sensitive content will only be included if `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` is `true`                                                                    |
 | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`  | `false`                                                       | If set to `true`, enables the capturing of OpenAI request and response content in the log events outputted by the agent.                                                                                                                                                       ↪ |
 
+## Central configuration
+
+```{applies_to}
+serverless: unavailable
+stack: preview 9.1 
+product:
+  edot_java: preview 1.5.0
+```
+
+APM Agent Central Configuration lets you configure EDOT Java instances remotely, see [Central configuration docs](/reference/central-configuration.md) for more details.
+
+### Central configuration settings
+
+You can modify the following settings for EDOT Java through APM Agent Central Configuration:
+
+| Setting | Central configuration name | Type |
+|---------|--------------------------|------|
+| Logging level | `logging_level` | Dynamic |
+| Turn off instrumentations | `deactivate_instrumentations` | Dynamic |
+| Turn off all instrumentations | `deactivate_all_instrumentations` | Dynamic |
+| Send traces | `send_traces` | Dynamic |
+| Send metrics | `send_metrics` | Dynamic |
+| Send logs | `send_logs` | Dynamic |
+
+Dynamic settings can be changed without having to restart the application.
 
 ## Configuration methods
 

docs/reference/edot-sdks/java/features.md

@@ -56,6 +56,12 @@ Set `OTEL_INSTRUMENTATION_RUNTIME_TELEMETRY_EMIT_EXPERIMENTAL_TELEMETRY` to `fal
 
 {{es}} and {{kib}} work best with metrics provided in delta-temporality. Therefore, the EDOT Java changes the default value of `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` to `DELTA`. You can override this default if needed, though some provided {{kib}} dashboards will not work correctly if you do it.
 
+## Central configuration
+
+You can manage EDOT Java configurations through the [APM Agent Central Configuration feature](docs-content://solutions/observability/apm/apm-agent-central-configuration.md) in the Applications UI.
+
+Refer to [Central configuration](/reference/central-configuration.md) for more information.
+
 ## Elastic Universal Profiling integration
 
 [Universal Profiling](https://www.elastic.co/observability/universal-profiling) integration provides the ability to correlate traces with profiling data from the Elastic universal profiler. This feature is turned on by default on supported systems, and turned off otherwise.

docs/reference/edot-sdks/java/index.md

@@ -29,6 +29,7 @@ In addition to all the features of OpenTelemetry Java, with EDOT Java you have a
 * Optional features that can enhance OpenTelemetry data that is being sent to Elastic.
 * Elastic-specific processors that ensure optimal compatibility when exporting OpenTelemetry signal data to an Elastic backend like an Elastic Observability deployment.
 * Preconfigured collection of tracing and metrics signals, applying some opinionated defaults, such as which sources are collected by default.
+* Compatibility with APM Agent Central Configuration to modify the settings of the EDOT Java agent without having to restart the application.
 
 Follow the step-by-step instructions in [Setup](/reference/edot-sdks/java/setup/index.md) to get started.