-
Notifications
You must be signed in to change notification settings - Fork 9
Open
Labels
flaky-testUnstable or unreliable test cases.Unstable or unreliable test cases.
Description
This test is frequently failing in the same way. Typically a re-run clears it, but it is failing frequently enough to be a pain.
E.g. https://github.com/elastic/elastic-otel-node/actions/runs/14199827632/job/39783819050?pr=693
# tool calls (captureMessageContent=true)
ok 145 exited successfully: err=null
ok 146 spans[0].name
ok 147 spans[0].kind
ok 148 spans[0].attributes['gen_ai.operation.name']
ok 149 spans[0].attributes['gen_ai.request.model']
ok 150 spans[0].attributes['gen_ai.system']
ok 151 spans[0].attributes['gen_ai.response.finish_reasons'].length
not ok 152 spans[0].attributes['gen_ai.response.finish_reasons'][0]
---
operator: equal
expected: 'tool_calls'
actual: 'stop'
at: assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
stack: |-
Error: spans[0].attributes['gen_ai.response.finish_reasons'][0]
at Test.assert [as _assert] (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:492:48)
at Test.strictEqual (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:670:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:93:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
at Object.checkTelemetry (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/fixtures.test.js:754:9)
at done (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:593:28)
at ChildProcess.exithandler (node:child_process:414:7)
at ChildProcess.emit (node:events:517:28)
...
ok 153 spans[0].attributes['gen_ai.response.id']
ok 154 spans[0].attributes['gen_ai.response.model']
ok 155 spans[0].attributes['gen_ai.usage.input_tokens']
ok 156 spans[0].attributes['gen_ai.usage.output_tokens']
ok 157 spans[0].scope.name
ok 158 log events.length
ok 159 log events[0].attributes['gen_ai.system']
ok 160 log events[0].attributes['event.name']
ok 161 log events[0].body.role
ok 162 log events[0].body.content
ok 163 log events[0].traceId
ok 164 log events[0].spanId
ok 165 log events[1].attributes['gen_ai.system']
ok 166 log events[1].attributes['event.name']
ok 167 log events[1].body.role
ok 168 log events[1].body.content
ok 169 log events[1].traceId
ok 170 log events[1].spanId
ok 171 log events[2].attributes['gen_ai.system']
ok 172 log events[2].attributes['event.name']
ok 173 log events[2].body.content
ok 174 log events[2].body.tool_calls
ok 175 log events[2].traceId
ok 176 log events[2].spanId
ok 177 log events[3].attributes['gen_ai.system']
ok 178 log events[3].attributes['event.name']
ok 179 log events[3].body.role
ok 180 log events[3].body.content
ok 181 log events[3].traceId
ok 182 log events[3].spanId
ok 183 log events[4].attributes['gen_ai.system']
ok 184 log events[4].attributes['event.name']
not ok 185 log events[4].body.finish_reason
---
operator: equal
expected: 'tool_calls'
actual: 'stop'
at: assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
stack: |-
Error: log events[4].body.finish_reason
at Test.assert [as _assert] (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:492:48)
at Test.strictEqual (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:670:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:93:7)
at Object.checkTelemetry (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/fixtures.test.js:786:9)
at done (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:593:28)
at ChildProcess.exithandler (node:child_process:414:7)
at ChildProcess.emit (node:events:517:28)
...
ok 186 log events[4].body.index
not ok 187 log events[4].body.message.tool_calls
---
operator: equal
expected: |-
[ { id: /.+/, type: 'function', function: { name: 'get_delivery_date', arguments: /{"order_id":".*?"}/ } } ]
actual: |-
undefined
at: assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
stack: |-
Error: log events[4].body.message.tool_calls
at Test.assert [as _assert] (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:492:48)
at Test.strictEqual (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/node_modules/tape/lib/test.js:670:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:108:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:105:7)
at assertDeepMatch (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:93:7)
at Object.checkTelemetry (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/fixtures.test.js:786:9)
at done (/home/runner/work/elastic-otel-node/elastic-otel-node/packages/instrumentation-openai/test/testutils.js:593:28)
at ChildProcess.exithandler (node:child_process:414:7)
...
ok 188 log events[4].traceId
ok 189 log events[4].spanId
Metadata
Metadata
Assignees
Labels
flaky-testUnstable or unreliable test cases.Unstable or unreliable test cases.