Baggage must be explicitly injected with the returned context?

[chiarag]

Hello all, I am working on a project where we would like to use the baggage propagation across different distributed service calls (to pass non-sensitive information and also "fire-walled" in private VPC).

What I am finding is leaving me scratching my head a bit, but it seems that the baggage is not auto-injected/instrumented.

I have my env set up to use baggage: OTEL_PROPAGATORS=tracecontext,baggage and it does stop in the text_map_propagator of the Baggage::Propagation module when I set breakpoints there. So it seems like the configuration is set up correctly. Yet the baggage is not there in Current.context

Example - I took the Dice example, added baggage, and a call to another service called dice-multiply (just trying to set up a POC of sorts):

require 'net/http'
require 'opentelemetry-api'

class DiceController < ApplicationController
  def roll
    roll = rand(1..6)

    OpenTelemetry::Baggage.build { |b| b.set_value('lost_bag', 'where is it?') }
    logger.info "Other raw entries #{OpenTelemetry::Baggage.raw_entries}" # this is empty

    url = URI.parse("http://localhost:4010/multiply/#{roll}")
    http = Net::HTTP.new(url.host, url.port)
    request = Net::HTTP::Post.new(url.request_uri)

    # I put a break point in the text_map_propagator class and it stops in there, but no baggage found
    response = http.request(request)
    data = JSON.parse(response.body)

    render json: data
  end
end

In that example, the call to dice-multiply has the HTTP_TRACEPARENT but no baggage. HTTP_BAGGAGE header is missing. When I put a breakpoint in the propagator (looks like I am using v1.4) line 34 - baggage is nil and it returns early without adding anything to the request.

This code, however, works:

require 'net/http'
require 'opentelemetry-api'

class DiceController < ApplicationController
  def roll
    roll = rand(1..6)

    context_with_baggage = OpenTelemetry::Baggage.build { |b| b.set_value('lost_bag', 'where is it?') }

    url = URI.parse("http://localhost:4010/multiply/#{roll}")
    http = Net::HTTP.new(url.host, url.port)
    request = Net::HTTP::Post.new(url.request_uri)

    # Inject baggage and trace context into the request headers - this works!
    # It *HAS* to have the context_with_baggage that was returned when we built the hash above
    OpenTelemetry.propagation.inject(request, context: context_with_baggage)

    # without passing in the context above and explicitly injecting, no baggage is found with Context.current
    response = http.request(request)
    data = JSON.parse(response.body)

    render json: data
  end
`end`

Breakpoint stop shows it adds the baggage, and then in the dice-multiply service, the header HTTP_BAGGAGE is there and I can get the raw entries with the library OpenTelemetry::Baggage.raw_entries - the lost_bag is there 🧳 🎉 🤣

Is this working as expected? In order to use the baggage propagation, do we have to explicitly inject which basically means we need our own HTTP wrapper/middleware to ensure it actually gets injected and sent over the wire?

I was hoping that it would just propagate the baggage without this explicit injection requirement...

Any help you can provide is greatly appreciated!!!

[chiarag]

Note that if I add this, then it does get injected without being explicit:

context_with_baggage = OpenTelemetry::Baggage.build { |b| b.set_value('lost_bag', 'where is it?') }
OpenTelemetry::Context.attach(context_with_baggage)

Is that basically the way to make the baggage propagate properly?

[kaylareopelle]

Yes, this is the correct way to attach baggage to the current context!

[kaylareopelle]

Hi @chiarag, thanks for reaching out! I'm still pretty new to Baggage, but I'll do my best to figure it out with you. I'm also going to bring this discussion up during the Ruby SIG meeting this Tuesday, November 19, at 9:00AM PT. You're welcome to join if you're available! You can subscribe to the SIG's calendar here to get the Zoom link.

The API documentation for Baggage#builder leads me to believe the current Context is the one that should be updated, but it calls context.set_value which returns a new Context and does not update the current Context. I'm not sure if that's the intended behavior.

Unfortunately, it doesn't look like Baggage#builder is part of the spec, but the spec leads me to believe it should update the current context:

If the language has support for implicitly propagated Context (see here), the API SHOULD also provide the following functionality:

Get the currently active Baggage from the implicit context. This is equivalent to getting the implicit context, then extracting the Baggage from the context.
Set the currently active Baggage to the implicit context. This is equivalent to getting the implicit context, then inserting the Baggage to the context.

I'm assuming that "implicit" context in the excerpt above means current context.

All of this is to say that I'm not sure what the proper way is to propagate baggage. To me, it seems like your first example should be sufficient, but I haven't seen this used in the wild on real web apps to know how most users do it.

I'd like to add an example to capture best practices for Baggage based on what we learn during this discussion.

[kaylareopelle]

Hi @chiarag, we chatted about this in the SIG today. If you'd like to watch the full conversation, you can do so here. We start to talk about this topic at 26:09.

You are correct - context with baggage needs to be explicitly attached to the OpenTelemetry::Context to propagate it to the current context. There's also an option to use Context.with_current to implicitly attach the context. You can see some examples of this in the baggage tests.

We should do a better job of documenting this. I'm going to open issues to add this to the official OpenTelemetry docs and to the RubyDoc documentation for baggage.

Let us know if you need anything else!

[kaylareopelle]

#1762 and #1763 have been opened to address the documentation gaps.

[chiarag]

Thank you so much for following up! It was cool to hear/watch the discussion too.