Service tag wrong randomly after restarts

[bsteve99]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Seemingly randomly my service tag comes out as _RESERVED_service: blah after restarts. But restart again and it might be correct as service: blah ( when sending logging requests to datadog ). Maybe a race condition..???

The condition will not fix itself until I continue to restart vector to get it to work correctly.

I've tried versions 0.39.0 to 0.44.0 with the same issue occurring.

I've done "tap's" and do not see any difference in the json log output (prior to sending to DD); when it works correctly vs when it does not work correctly.

Configuration

Version

0.44.0

Debug Output

Example Data

No response

Additional Context

This is running on Linux.

References

No response

[pront]

Hi @bsteve99, can you share your config (redact sensitive data first) and log samples so I can better understand the differences you attempted to describe above?

If I understand correctly, you are using a datadog_logs sink and it sounds related to this #22157 (comment).

[bsteve99]

I actually may have stumbled on a fix, with many restarts and no issues.

I was using datadog_log sink and re-wrote it in http sink; and its looking good.

old sink here:
_srv_dd_log_out:
    type: datadog_logs
    inputs:
      - _strip_verbose_log_lines
    compression: gzip
    default_api_key: "${DD_API_KEY:?DD_API_KEY must be specified}"
    tls:
      enabled: true
    batch:
      max_events: ${DATADOG_LOG_BATCHSIZE:-200}
      timeout_secs: 3
    buffer:
      type: memory
      max_events: 20
      when_full: block
    request:
      concurrency: adaptive

new sink here.
  _srv_dd_log_out:
    type: http
    inputs:
      - _strip_verbose_log_lines
    compression: gzip
    uri: https://http-intake.logs.datadoghq.com/api/v2/logs
    method: post
    tls:
      verify_certificate: false
      verify_hostname: true
    batch:
      max_events: ${DATADOG_LOG_BATCHSIZE:-200}
      timeout_secs: 3
    encoding:
      codec: json
    buffer:
      type: memory
      max_events: 20
      when_full: block
    request:
      headers:
        DD-API-KEY: "${DD_API_KEY:?DD_API_KEY must be specified}"
        Content-Type: application/json
        Accept: application/json
      concurrency: adaptive

[pront]

Sounds great. But also, I am curious why datadog_logs didn't work out of the box.

[bsteve99]

Yes, agree. But random _RESERVED_service should be enough info to see why this might happen randomly, as I said seems like a race condition that might be worth looking into.

The other detail I have is once this _RESERVED_service tag starts coming out, it never stops until we can restart vector. So some state is getting set; and never re-evaluating. Restart vector is the only solution; and once its fixed, it never messes it up ( until a restart may or may not cause it to be good or bad )