-
Notifications
You must be signed in to change notification settings - Fork 163
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## Description of the changes Some small improvements for v2 docs ## Checklist - [X] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [X] I have signed all commits --------- Signed-off-by: Jonah Kowall <[email protected]>
- Loading branch information
Showing
9 changed files
with
116 additions
and
91 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| --- | ||
| title: Sampling | ||
| hasparent: true | ||
| --- | ||
|
|
||
| Sampling is essential when handling large volumes of traces. Ideally, discarding less interesting data and keeping the data needed to diagnose issues. Other capabilities like calculating metrics from traces can help reduce the volume of spans collected and stored. | ||
|
|
||
| Jaeger supports multiple types of sampling strategies which leverage OpenTelemetry. The `jaeger` binary supports specific configurations. | ||
|
|
||
| ## Head Based Sampling | ||
| Head based sampling is when the sampling decision is made when the first spans are seen by `jaeger`. The support of Head based sampling comes from OpenTelemetry. The topic is covered in the [OpenTelemetry documentation on head based sampling](https://opentelemetry.io/docs/concepts/sampling/#head-sampling). | ||
|
|
||
| Example configuration for [various head based sampling strategies](https://github.com/jaegertracing/jaeger/blob/main/cmd/jaeger/sampling-strategies.json). | ||
|
|
||
| ## Tail Based Sampling | ||
| Tail based sampling allows for the sampling decisions to be made after the trace is complete and all spans have been collected. Ideally this provides more granular control over which traces and kept and which are discarded. The downside to this is that more memory and processing power will be used by `jaeger` binaries. | ||
|
|
||
| This [configuration allows for the sampling of everything observed](https://github.com/jaegertracing/jaeger/blob/main/cmd/jaeger/config-tail-sampling-always-sample.yaml) by `jaeger` with debug logging to diagnose problems. | ||
|
|
||
| The second configuration of `jaeger` [samples for a specific service name](https://github.com/jaegertracing/jaeger/blob/main/cmd/jaeger/config-tail-sampling-service-name-policy.yaml). | ||
|
|
||
| ## Remote Sampling | ||
|
|
||
| If your client SDKs are configured to use remote sampling configuration (see [Remote Sampling API][remote-sampling-api]) then sampling rates can be centrally controlled. In this setup a sampling strategy configuration is served to the client SDK that describes endpoints and their sampling probabilities. This configuration can be generated by `jaeger` in two different ways: [periodically loaded from a file](#file-based-sampling-configuration) or [dynamically calculated based on traffic](#adaptive-sampling). The method of generation is controlled by the environment variable `SAMPLING_CONFIG_TYPE` which can be set to either `file` (default) or `adaptive`. | ||
|
|
||
| [remote-sampling-api]: ../apis#remote-sampling-configuration-stable | ||
|
|
||
| ### File-based Sampling Configuration | ||
|
|
||
| `jaeger` can be instantiated with the `--sampling.strategies-file` option that points to a file containing sampling strategies to be served to Jaeger clients. The option's value can contain a path to a JSON file, which will be automatically reloaded if its contents change, or an HTTP URL from where the file will be periodically retrieved, with reload frequency controlled by the `--sampling.strategies-reload-interval` option. | ||
|
|
||
| If no configuration is provided, **jaeger-collector**s will return the default probabilistic sampling policy with probability 0.001 (0.1%) for all services. | ||
|
|
||
| Example `strategies.json`: | ||
| ```json | ||
| { | ||
| "service_strategies": [ | ||
| { | ||
| "service": "foo", | ||
| "type": "probabilistic", | ||
| "param": 0.8, | ||
| "operation_strategies": [ | ||
| { | ||
| "operation": "op1", | ||
| "type": "probabilistic", | ||
| "param": 0.2 | ||
| }, | ||
| { | ||
| "operation": "op2", | ||
| "type": "probabilistic", | ||
| "param": 0.4 | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "service": "bar", | ||
| "type": "ratelimiting", | ||
| "param": 5 | ||
| } | ||
| ], | ||
| "default_strategy": { | ||
| "type": "probabilistic", | ||
| "param": 0.5, | ||
| "operation_strategies": [ | ||
| { | ||
| "operation": "/health", | ||
| "type": "probabilistic", | ||
| "param": 0.0 | ||
| }, | ||
| { | ||
| "operation": "/metrics", | ||
| "type": "probabilistic", | ||
| "param": 0.0 | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| `service_strategies` element defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`. | ||
|
|
||
| In the above example: | ||
|
|
||
| * All operations of service `foo` are sampled with probability 0.8 except for operations `op1` and `op2` which are probabilistically sampled with probabilities 0.2 and 0.4 respectively. | ||
| * All operations for service `bar` are rate-limited at 5 traces per second. | ||
| * Any other service will be sampled with probability 0.5 defined by the `default_strategy`. | ||
| * The `default_strategy` also includes shared per-operation strategies. In this example we disable tracing on `/health` and `/metrics` endpoints for all services by using probability 0. These per-operation strategies will apply to any new service not listed in the config, as well as to the `foo` and `bar` services unless they define their own strategies for these two operations. | ||
|
|
||
| ### Adaptive Sampling | ||
|
|
||
| Since Jaeger v1.27. | ||
|
|
||
| Adaptive sampling works in **jaeger-collector** by observing the spans received from services and recalculating sampling probabilities for each service/endpoint combination to ensure that the volume of collected traces matches `--sampling.target-samples-per-second`. When a new service or endpoint is detected, it is initially sampled with `--sampling.initial-sampling-probability` until enough data is collected to calculate the rate appropriate for the traffic going through the endpoint. | ||
|
|
||
| Adaptive sampling requires a storage backend to store the observed traffic data and computed probabilities. At the moment `memory` (for all-in-one deployment), `cassandra`, `badger`, `elasticsearch` and `opensearch` are supported as sampling storage backends. | ||
|
|
||
| By default adaptive sampling will attempt to use the backend specified by `SPAN_STORAGE_TYPE` to store data. However, a second type of backend can also be specified by using `SAMPLING_STORAGE_TYPE`. For instance, `SPAN_STORAGE_TYPE=elasticsearch SAMPLING_STORAGE_TYPE=cassandra ./jaeger-collector` will run **jaeger-collector** in a mode where it attempts to store its span data in the configured elasticsearch cluster and its adaptive sampling data in the configured cassandra cluster. Note that this feature can not be used to store span and adaptive sampling data in two different backends of the same type. | ||
|
|
||
| Read [this blog post](https://medium.com/jaegertracing/adaptive-sampling-in-jaeger-50f336f4334) for more details on adaptive sampling engine. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters