Remap shared services for edx-edxapp in DD

[robrap]

The problem is that default DD configuration for mysql, django (cache), defaultdb, requests, and potentially other libraries has each of these going to a shared service of the same name that is used across all of our IDA services.

Problems with this default approach:

• It is difficult to determine what span data belongs to which IDA/service.
• There is some metric data that may be impossible to separate between IDAs/services.

Acceptance Criteria

• Decide on and build a Proof of Concept for a service naming scheme that associates other spans (e.g. mysql, django (cache), defaultdb, mysql, requests, kafka, etc.) with the edxapp service.
  • Ensure spans do not have multiple service tags (may require SRE support)
  • Ensure we've addressed the complete list of dependent services.
  • Ensure the primary operation of the edxapp services remain as they are (e.g. django.requests, etc.).
• Document (possibly in a wiki ADR) why we made the choices we made, and communicate to other service owners. Ideally they can participate in ADR review, so we get a consistent solution across all of our services.
• implement the decided upon service naming scheme for edxapp
  • Approach: Inferred Services
    • Feature switch: feat: Support for Datadog Inferred Services at the library level configuration#122
    • stage, edge, prod
    • Just make it default to true and remove the per-env settings
    • devstack (timmc/datadog-local-testing is updated)
• Offer to other teams
  • Convert ADR into DD docs, explaining how to apply the POC approach to other services
    • https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1008500894/Datadog+configuration+and+setup+details#Global-default-service-name
  • Make available in Helm Charts
    • Configure DD agent in k8s
      • stage: https://github.com/edx/edx-internal/pull/12175
      • edge: https://github.com/edx/edx-internal/pull/12178
      • prod: https://github.com/edx/edx-internal/pull/12179
    • Add config option in django-ida helm chart, defaulting false for now: https://github.com/edx/helm-charts/pull/170
    • Available now with django-ida helm chart 0.10.0
  • Write up a recommendation for other teams
  • Test in k8s
  • Announce broadly as a recommendation to other teams
• File ticket for cleanup: Cleanup after global service names rollout #973

Notes:

• DD support ticket https://help.datadoghq.com/hc/en-us/requests/1781912 asks a variety of questions about this topic, specifically targeting django (cache) and requests services. Questions and answers should ultimately be copied out of this ticket once it is complete.
• DD support ticket https://help.datadoghq.com/hc/en-us/requests/1762193 asks similar questions about mysql, but has some additional information about metrics and multiple service tags (which should be avoided). Questions and answers should ultimately be copied out of this ticket once it is complete.
• DD support ticket https://help.datadoghq.com/hc/en-us/requests/1873842 summarizes some of the above as well as other options we've considered, and asks for guidance.
• There is an open (support) question about whether using DD_SERVICE_MAPPING is a simpler method of remapping, rather than using a variety of different settings.
  • However, remapping django to cache might be risky, because django could be used for something else in the future, so in this particular case we might want to use DD_DJANGO_CACHE_SERVICE_NAME instead.
• For each service (e.g. mysql, django, etc.) we decide to remap, we need to choose between the IDA service service:edx-edxapp-lms (spans would still have a different operation_name), or service:edx-edxapp-lms-cache (a new DD service catalog service).
  • If everything went to the same service, we may need to adjust the primary operation_name for edx-edxapp-lms if it no longer defaults to operation_name:django.requests. See these DD docs for configuring the primary operation.
• If we go with separate DD service names:
  • We would probably use the following naming convention:
    • service:edx-edxapp-lms
    • service:edx-edxapp-lms-cache (was django)
    • service:edx-edxapp-lms-defaultdb
    • Etc.
  • Unfortunately, I chose to go with service:edx-edxapp-lms-workers rather than service:edx-edxapp-workers-lms.
    • We might need to change this, because presumably we'd want service:edx-edxapp-workers-lms-cache, etc., and if we used a search like service:edx-edxapp-lms* to pick up all services related to service:edx-edxapp-lms, we would not want that to also pick up the worker service and all its sub-services.
    • Should we do this as a separate, pre-emptive ticket? We could use expand/contract to ensure monitors (listed in DD) and dashboards, etc. are updated before the name is changed. Communications will be required.
• ADR should state that we are rejecting the status quo of a shared DD service across all of our IDAs.
• What is the full list of services to address? DD has a dependency graph somewhere, and some other possible services may includes elasticsearch, redis, read_replicadb. Please review in DD to get the full list of affect dependencies. Note that some general shared services (e.g. aws.s3) probably should remain shared, but this could be discussed.

ADRish page: https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1265598591/Datadog+Service+mapping

[robrap]

Answers from DD Support ticket:

Coming back from engineering to let you know answers to your queries:

a. Are there any downsides of using DD_SERVICE_MAPPING, over the individual settings like DD_DJANGO_CACHE_SERVICE_NAME? Which is preferred? Which takes precedence?

DD_SERVICE_MAPPING allows you to remap service names in a more generalized way. It is useful when you have multiple integrations or need a consistent naming scheme across various services.
Individual settings like DD_DJANGO_CACHE_SERVICE_NAME are more specific and used for fine-grained control over the naming of particular integrations.
Precedence: Typically, DD_SERVICE_MAPPING would take precedence over individual settings because it applies a broader remapping rule. However, specific integration settings can override this if explicitly defined.

Downsides:

Using DD_SERVICE_MAPPING can be less granular, potentially leading to confusion if not documented properly.
Individual settings provide more control but can be cumbersome to manage for large-scale applications with multiple integrations.

Preferred Option:

The preferred option depends on the use case. For broad consistency, DD_SERVICE_MAPPING is preferred. For fine-grained control, individual settings are preferred.

b. If we were to set DD_TRACE_SAMPLING_RULES with service-specific rules, would we use the original service name (e.g. django), or the mapped service name? Is the answer different for each option from question 1a?

When setting DD_TRACE_SAMPLING_RULES, use the mapped service name if you are using DD_SERVICE_MAPPING. This ensures that the rules apply correctly to the spans after they have been remapped.
The same applies to individual settings. If you are changing the service name using specific integration settings, use the new service name in the sampling rules.

Regarding service Mapping and operation names:

If you map services to a single service (e.g., edx-edxapp-lms), you can differentiate by operation_name.
However, if there is a risk that cache calls might outnumber Django requests, it might change the primary operation name in the Datadog UI.

In this case I would recommend:

To maintain differentiation and control, consider mapping services to distinct names (e.g., edx-edxapp-lms-cache for cache operations).
This approach allows for clearer visibility and avoids the potential issue of the UI selecting the most popular operation name.
You can also select primary operation manually in the UI

[robrap]

The following ticket is related to this ticket, because it would drop some of the services that need to be remapped (e.g. read_replicadb, defaultdb, django (cache)).

• Disable django db and cache instrumentation for edxapp #761

[robrap]

@timmc-edx: Should this wait on the trace concatenation fix so we don't bring in a lot of new spans? Not sure how this will affect things.

[timmc-edx]

Hmm... I don't know that the two will interact that much.

[timmc-edx]

I'll be using https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1265598591/Datadog+Service+mapping to build out something ADR-like and track testing in edx-platform.