Add tracing logs for Nexus HTTP request retries

[pdoerner]

What changed?

Added additional HTTP tracing logs for the first 3 Nexus request retries.
Both minimum and maximum attempts to add additional tracing info to are configurable.

Why?

To aid in debugging.

How did you test it?

Potential risks

Documentation

Is hotfix candidate?

[bergundy]

This should be disabled by default. Generally, I don't think the attempt number matters here, it's more about what to trace IMHO. E.g. whether TLSHandshakeStart should be traced or not. Ideally the config would be for a set of methods, but It'd be hard to express in dynamic config as a set without adding a lot of overhead. We could probably provide a config per method we support intercepting though.

[pdoerner]

I think attempt count is useful for limiting the overhead and the overall log spam. Without a notion of attempt count do you imagine we would log trace info for every retry?

Instead of adding a dynamic config per-hook, we could just use fx to provide the httptrace.ClientTrace. Then anyone who wants to change the behavior could do so with fx.replace

[bergundy]

Discussed offline, agree that adding the counts is nice, also requested that we add the dynamic config and make it global so it can be efficiently cached (a limitation of the current DC implementation).

[bergundy]

Approved with a comment.

[bergundy]

Make this cached please (you'll want to wrap it with the following):

temporal/common/dynamicconfig/cachedvalue.go

Line 43 in 74e715a

func NewGlobalCachedTypedValue[T, I any](c *Collection, setting GlobalTypedSetting[I], convert func(I) (T, error)) *GlobalCachedTypedValue[T] {

[pdoerner]

Synced with @dnr offline and changed the conversion functions to the best we can do at the moment. We will want to revisit this setting after merging #7052