Skip to content

flaky test: instrumentation-openai test tool calls (captureMessageContent=true) #698

@trentm

Description

@trentm

This test is frequently failing in the same way. Typically a re-run clears it, but it is failing frequently enough to be a pain.

E.g. https://github.com/elastic/elastic-otel-node/actions/runs/14199827632/job/39783819050?pr=693

# tool calls (captureMessageContent=true)
ok 145 exited successfully: err=null
ok 146 spans[0].name
ok 147 spans[0].kind
ok 148 spans[0].attributes['gen_ai.operation.name']
ok 149 spans[0].attributes['gen_ai.request.model']
ok 150 spans[0].attributes['gen_ai.system']
ok 151 spans[0].attributes['gen_ai.response.finish_reasons'].length
not ok 152 spans[0].attributes['gen_ai.response.finish_reasons'][0]
  ---
    operator: equal
    expected: 'tool_calls'
    actual:   'stop'
    at: assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
    stack: |-
      Error: spans[0].attributes['gen_ai.response.finish_reasons'][0]
          at Test.assert [as _assert] (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:492:48)
          at Test.strictEqual (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:670:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:93:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
          at Object.checkTelemetry (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/fixtures.test.js:754:9)
          at done (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:593:28)
          at ChildProcess.exithandler (node:child_process:414:7)
          at ChildProcess.emit (node:events:517:28)
  ...
ok 153 spans[0].attributes['gen_ai.response.id']
ok 154 spans[0].attributes['gen_ai.response.model']
ok 155 spans[0].attributes['gen_ai.usage.input_tokens']
ok 156 spans[0].attributes['gen_ai.usage.output_tokens']
ok 157 spans[0].scope.name
ok 158 log events.length
ok 159 log events[0].attributes['gen_ai.system']
ok 160 log events[0].attributes['event.name']
ok 161 log events[0].body.role
ok 162 log events[0].body.content
ok 163 log events[0].traceId
ok 164 log events[0].spanId
ok 165 log events[1].attributes['gen_ai.system']
ok 166 log events[1].attributes['event.name']
ok 167 log events[1].body.role
ok 168 log events[1].body.content
ok 169 log events[1].traceId
ok 170 log events[1].spanId
ok 171 log events[2].attributes['gen_ai.system']
ok 172 log events[2].attributes['event.name']
ok 173 log events[2].body.content
ok 174 log events[2].body.tool_calls
ok 175 log events[2].traceId
ok 176 log events[2].spanId
ok 177 log events[3].attributes['gen_ai.system']
ok 178 log events[3].attributes['event.name']
ok 179 log events[3].body.role
ok 180 log events[3].body.content
ok 181 log events[3].traceId
ok 182 log events[3].spanId
ok 183 log events[4].attributes['gen_ai.system']
ok 184 log events[4].attributes['event.name']
not ok 185 log events[4].body.finish_reason
  ---
    operator: equal
    expected: 'tool_calls'
    actual:   'stop'
    at: assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
    stack: |-
      Error: log events[4].body.finish_reason
          at Test.assert [as _assert] (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:492:48)
          at Test.strictEqual (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:670:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:93:7)
          at Object.checkTelemetry (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/fixtures.test.js:786:9)
          at done (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:593:28)
          at ChildProcess.exithandler (node:child_process:414:7)
          at ChildProcess.emit (node:events:517:28)
  ...
ok 186 log events[4].body.index
not ok 187 log events[4].body.message.tool_calls
  ---
    operator: equal
    expected: |-
      [ { id: /.+/, type: 'function', function: { name: 'get_delivery_date', arguments: /{"order_id":".*?"}/ } } ]
    actual: |-
      undefined
    at: assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
    stack: |-
      Error: log events[4].body.message.tool_calls
          at Test.assert [as _assert] (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:492:48)
          at Test.strictEqual (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:670:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
          at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:93:7)
          at Object.checkTelemetry (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/fixtures.test.js:786:9)
          at done (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:593:28)
          at ChildProcess.exithandler (node:child_process:414:7)
  ...
ok 188 log events[4].traceId
ok 189 log events[4].spanId

Metadata

Metadata

Assignees

No one assigned

    Labels

    flaky-testUnstable or unreliable test cases.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions