Is there a way to sample overrides Servicebus requests?

[Charlie441324219]

Is your feature request related to a problem? Please describe.
After migrating from v2 to v3, we received a huge amount of requests from Servicebus. I've tried to do sample override for Servicebus requests but I wasn't able to do so. I can't find any documentation example allowing me to do so.

Describe the solution you would like
Users would be able to do sample overrides for Servicebus requests.

Describe alternatives you have considered
NA

Additional context

I'm using AppInsights Java 3.4.17. servicebus 7.10.1

[heyams]

@Charlie441324219 thanks for reaching out.
you should be able to sample out servicebus requests.

can you pls share your json config regarding sampling overrides? this is how you can find out servicebus request's attributes.

[Charlie441324219]

Hi @heyams , thank you for your reply, this is my config to sample out servicebus request

        {
          "telemetryType": "request",
          "attributes": [
            {
              "key": "http.url",
              "value": ".*servicebus.windows.net.*",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        }

[heyams]

@Charlie441324219 can you share your full applicaitoninsights.json config and your applicationinsights.log at debug logging level? we ga sampling overrides since 3.5.0 GA. our azure doc got updated. I want to make sure you use the correct config for 3.4.17. please try out 3.5.1 GA.

[Charlie441324219]

Sure @heyams , this is my full config

{
  "sampling": {
    "percentage": 100
  },
  "jmxMetrics": [
    {
      "name": "Heap Memory Usage-used",
      "objectName": "java.lang:type=Memory",
      "attribute": "HeapMemoryUsage.used"
    }
  ],
  "instrumentation": {
    "logging": {
      "level": "ALL"
    }
  },
  "preview": {
    "captureLoggingLevelAsCustomDimension": true,
    "sampling": {
      "overrides": [
        {
          "telemetryType": "dependency",
          "percentage": 0
        },
        {
          "telemetryType": "request",
          "attributes": [
            {
              "key": "http.url",
              "value": ".*servicebus.windows.net.*",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        }
      ]
    }
  },
  "selfDiagnostics": {
    "destination": "console",
    "level": "DEBUG"
  }
}

[heyams]

@Charlie441324219 applicationinsights.log please, you may share a link with me somewhere? your config looks good to me though.

[Charlie441324219]

Hi @heyams , I am afraid I cannot share that due to the company policy. could you please provide more details about how to setup the key and value based on applicationinsights.log ?

[heyams]

[Charlie441324219]

from exporting span I can see

attributes=AttributesMap{data={peer.address=stage-membership-credit.servicebus.windows.net, thread.name=boundedElastic-19, message_bus.destination=credit-sync/subscriptions/quickScreenAttachments, applicationinsights.internal.item_count=1, thread.id=130, x-opt-enqueued-time=1710354044, az.namespace=Microsoft.ServiceBus}, capacity=128, totalAddedValues=7}, totalAttributeCount=7, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=OK, description=}, hasEnded=true}

so I tried with

        {
          "telemetryType": "request",
          "attributes": [
            {
              "key": "peer.address",
              "value": ".*servicebus.windows.net",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        }

but still not work

[heyams]

i don't find attributes though. it's hard for me to help you further if you don't share your log with me. you can redact any sensitive information in the log and then share.

[Charlie441324219]

Thank you @heyams , I will raise a ticket to MSFT via company.

[heyams]

@Charlie441324219 hey Charlie, there was an upstream fix for azure-core 1.36.
your issue seems related to #3147.

can you confirm if you're using azure-core 1.36+?

[lmolkova]

@Charlie441324219
I believe you're running very old version of Azure ServiceBus where we used different set of attributes and configuration options.
With newer version of servicebus and the agent you'd be able to you'd be able to override sampling settings. It also contains a LOT of perf and reliability improvements.

If you update ServiceBus version, you would be to disable distributed tracing on the client instance using the following code.

TracingOptions tracingOptions = new TracingOptions().setEnabled(false);
ClientOptions clientOptions = new AmqpClientOptions().setTracingOptions(tracingOptions);
return new ServiceBusClientBuilder()
    .clientOptions(clientOptions )

Without updating, I think the best option would be to disable all Azure SDK instrumentation including ServiceBus - you can do it via the config or env var - https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#suppress-specific-autocollected-telemetry.

[Charlie441324219]

Hi @lmolkova ,

Thank you for your info, after disabling all Azure SDK instrumentation, I cannot see servicebus request anymore. But is there any other impact for disabling Azure SDK instrumentation ? Is that possible that I lost some important log ?

[lmolkova]

Hi @lmolkova ,

Thank you for your info, after disabling all Azure SDK instrumentation, I cannot see servicebus request anymore. But is there any other impact for disabling Azure SDK instrumentation ? Is that possible that I lost some important log ?

The Application Insights config (or environment variable) and the flag on tracing options only affect distributed tracing and not logs.
By disabling Azure SDK instrumentation in Application Insights config you disable all requests and dependencies from all Azure SDKs. By setting the flag on Service Bus client options, you disable one specific instance of this client.

You'll still get logs based on your logging provider configuration.

[Charlie441324219]

Hi @lmolkova ,
Thank you for your info, after disabling all Azure SDK instrumentation, I cannot see servicebus request anymore. But is there any other impact for disabling Azure SDK instrumentation ? Is that possible that I lost some important log ?

The Application Insights config (or environment variable) and the flag on tracing options only affect distributed tracing and not logs. By disabling Azure SDK instrumentation in Application Insights config you disable all requests and dependencies from all Azure SDKs. By setting the flag on Service Bus client options, you disable one specific instance of this client.

You'll still get logs based on your logging provider configuration.

What about exceptions from Azure SDK ? If we lost exception logs, we will have big trouble.

[lmolkova]

What about exceptions from Azure SDK ? If we lost exception logs, we will have big trouble.

Exceptions come from logs (and then are mapped to exceptions or traces with error severity if there was no exception associated with an error) and not as requests or dependencies, so this setting should not affect them.

[Charlie441324219]

Thank you @lmolkova , I will go ahead with disabling all Azure SDK instrumentation

[heyams]

@Charlie441324219 thanks! pls let me know if i can close this issue now.