How do I gracefully shutdown the SDK?

[arielvalentin]

This discussion came out of a pull request for Rake instrumentation that registered an at_exit hook to shutdown the SDK however:

I hesitate to register at_exit hooks in instrumentations and I think it would be better suited for the SDK package. There are other cases where the tracer provider should also be shutdown, e.g. Resque Workers and Unicorn that fork child process.

cc: @open-telemetry/ruby-maintainers

Originally posted by @arielvalentin in open-telemetry/opentelemetry-ruby-contrib#80 (comment)

What is the best way for users to gracefully shutdown the SDK?

Should the SDK register at_exit hooks automatically?

Should this be left to end users? How do minimize friction for new users who want to see traces for one-off scripts?

[plantfansam]

We discussed this in SIG on August 2. IIRC — we did not decide that this functionality should definitely exist in the SDK, but rather that if we want to provide this functionality, it would be in the SDK. Am I remembering that right @mwear @robertlaurin @arielvalentin @ericmustin?

[arielvalentin]

You are correct, however our docs lack that guidance/knowledge.

[fbogsany]

The SDK provides the functions to call for graceful shutdown, but it cannot and should not anticipate every environment it might run in. Graceful shutdown is an application and/or application framework concern. IIRC we have a complicated set of Unicorn lifecycle hooks at Shopify, for example, and using at_exit can result in unpalatable situations like dropped spans or lengthy shutdown delays.

It is reasonable for Sidekiq instrumentation, for example, to optionally register job worker shutdown hooks that call the SDK's shutdown method. Whether this should default to "on" or "off" is probably framework-specific. Organizations that build their own SDK wrappers (e.g. Shopify and GitHub) have a better idea of the range of environments they need to support and can choose defaults that work for them.

[fbogsany]

What is the best way for users to gracefully shutdown the SDK?

It depends a lot on their environment and application framework. E.g. a Lambda style environment will probably want to OpenTelemetry.tracer_provider.force_flush(timeout: TIMEOUT) after each request. We (Shopify) have some complex lifecycle hooks for Unicorn, and we register one to OpenTelemetry.tracer_provider.force_flush(timeout: TIMEOUT) in the worker pre-termination hook. IIRC Sidekiq also has some shutdown hooks for job workers. We also call at_exit { OpenTelemetry.tracer_provider.shutdown(timeout: 60) } in our SDK wrapper gem, during initialization. For environments where we register lifecycle hooks, we expect the BSP to have no buffered spans by the time we call shutdown in the at_exit hook.

Without the lifecycle hooks, relying on at_exit alone can lead to unpleasant things during an incident. For example, the BSP buffer may be full, an earlier at_exit hook (or something else) may have closed all open connections (including the exporter's), and the BSP + exporter struggle mightily for several minutes to establish a connection to the collector and empty the buffer, which drastically impedes progress of an emergency deploy due to the k8s rollout strategy. I'm not saying we have direct experience with this, but it's probably happened to "someone"...

Should the SDK register at_exit hooks automatically?

No. Not least because the order of initialization will be somewhat unpredictable. at_exit hooks stack, and the order of their registration can matter a lot. Users should manage that themselves.

Should this be left to end users? How do minimize friction for new users who want to see traces for one-off scripts?

For one-off scripts, at_exit { OpenTelemetry.tracer_provider.shutdown(timeout: 30) } is a pretty simple one-liner that we could document (for some value of 30). If we ever get around to building that shared SDK wrapper gem we've talked about a few times, we could add a helper there: OTel::Wrapper.please_flush_and_wash_your_hands or something similarly evocative.

[stevenharman]

IIRC Sidekiq also has some shutdown hooks for job workers.

It does. We (Dropbox) leverage them something like this:

Sidekiq.configure_server do |config|
  config.on(:shutdown) do
    OpenTelemetry.tracer_provider.shutdown(timeout: 20) # keep the timeout shorter than Sidekiq's graceful shutdown timeout
  end
end

Should that happen auto-magically just by installing the instrumentation? Probably not. But it could/should be documented so folks know to do it.