Artillery exits with code 21

[jplieninger]

Hi all,

I have a performance test running and at the end of the test after it prints the summary report it exits with error code 21. Is there any way to get more insights into why it exits with this code?

This results in a "failing test" in the dashboard, but the test run was successful.

Link to the dashboard

Config:

config:
  target: "dummy"
  http:
    timeout: 30
    maxSockets: 5
    extendedMetrics: true
  phases:
    - duration: 60 # 1m
      arrivalRate: 3
      name: "Warm-up phase"

    - duration: 120 # 2m
      arrivalRate: 3
      name: "Baseline load"

    - duration: 120 # 2m
      arrivalRate: 9
      name: "Sudden traffic spike (3x)"

    - duration: 60 # 1m
      arrivalRate: 9
      rampTo: 3
      name: "Recovery period"

    - duration: 300 # 5m
      arrivalRate: 3
      rampTo: 15
      name: "Gradual ramp to high load (5x)"

    - duration: 300 # 5m
      arrivalRate: 15
      name: "Sustained high load (5x)"

    - duration: 300 # 5m
      arrivalRate: 15
      rampTo: 30
      name: "Gradual ramp to extreme load (10x)"

    - duration: 300 # 5m
      arrivalRate: 30
      name: "Sustained extreme load (10x)"

    - duration: 120 # 2m
      arrivalRate: 30
      rampTo: 3
      name: "Cool down phase"

  processor: "../src/processor.js"
  plugins:
    metrics-by-endpoint: {
      useOnlyRequestNames: true
    }
    expect: {}
    apdex:
      threshold: 200

before:
  flow:
    - function: "setupAccounts"

scenarios:
  - name: "ShiftStatus - All Skills"
    weight: 60
    flow:
      - loop:
          - function: "getAllSkillsShifts"
          - get:
              name: "GetAllSkillsShifts"
              url: "{{ url }}"
              headers:
                Authorization: "{{ bearerToken }}"
              expect:
                - statusCode: 200
        count: 20

  - name: "ShiftStatus - Skill Shift Status"
    weight: 40
    flow:
      - loop:
          - function: "getSingleSkillShifts"
          - get:
              name: "GetSingleSkillsShifts"
              url: "{{ url }}"
              headers:
                Authorization: "{{ bearerToken }}"
              expect:
                - statusCode: 200
        count: 20

Example of the output

All VUs finished. Total time: 7 minutes, 4 seconds

--------------------------------
Summary report @ 16:09:00(+0100)
--------------------------------

apdex.frustrated: .............................................................. 41
apdex.satisfied: ............................................................... 216390
apdex.tolerated: ............................................................... 2823
errors.ECONNRESET: ............................................................. 20
http.codes.200: ................................................................ 219253
http.codes.500: ................................................................ 1
http.dns:
  min: ......................................................................... 0
  max: ......................................................................... 148
  mean: ........................................................................ 0.1
  median: ...................................................................... 0
  p95: ......................................................................... 1
  p99: ......................................................................... 3
http.downloaded_bytes: ......................................................... 15294577
http.request_rate: ............................................................. 176/sec
http.requests: ................................................................. 219274
http.response_time:
  min: ......................................................................... 19
  max: ......................................................................... 1264
  mean: ........................................................................ 127.6
  median: ...................................................................... 122.7
  p95: ......................................................................... 133
  p99: ......................................................................... 262.5
http.response_time.2xx:
  min: ......................................................................... 19
  max: ......................................................................... 1264
  mean: ........................................................................ 127.6
  median: ...................................................................... 122.7
  p95: ......................................................................... 133
  p99: ......................................................................... 262.5
http.response_time.5xx:
  min: ......................................................................... 140
  max: ......................................................................... 140
  mean: ........................................................................ 140
  median: ...................................................................... 141.2
  p95: ......................................................................... 141.2
  p99: ......................................................................... 141.2
http.responses: ................................................................ 219254
http.tcp:
  min: ......................................................................... 0
  max: ......................................................................... 186
  mean: ........................................................................ 0.7
  median: ...................................................................... 0
  p95: ......................................................................... 10.1
  p99: ......................................................................... 15
http.tls:
  min: ......................................................................... 0
  max: ......................................................................... 194
  mean: ........................................................................ 1.4
  median: ...................................................................... 0
  p95: ......................................................................... 22
  p99: ......................................................................... 29.1
http.total:
  min: ......................................................................... 116
  max: ......................................................................... 1264
  mean: ........................................................................ 130.1
  median: ...................................................................... 125.2
  p95: ......................................................................... 165.7
  p99: ......................................................................... 267.8
plugins.expect.failed: ......................................................... 1
plugins.expect.failed.statusCode: .............................................. 1
plugins.expect.ok: ............................................................. 219253
plugins.expect.ok.statusCode: .................................................. 219253
plugins.metrics-by-endpoint.GetAllSkillsShifts.codes.200: ...................... 131673
plugins.metrics-by-endpoint.GetAllSkillsShifts.codes.500: ...................... 1
plugins.metrics-by-endpoint.GetAllSkillsShifts.errors.ECONNRESET: .............. 11
plugins.metrics-by-endpoint.GetSingleSkillsShifts.codes.200: ................... 87580
plugins.metrics-by-endpoint.GetSingleSkillsShifts.errors.ECONNRESET: ........... 9
plugins.metrics-by-endpoint.response_time.GetAllSkillsShifts:
  min: ......................................................................... 19
  max: ......................................................................... 1097
  mean: ........................................................................ 127.4
  median: ...................................................................... 122.7
  p95: ......................................................................... 133
  p99: ......................................................................... 267.8
plugins.metrics-by-endpoint.response_time.GetSingleSkillsShifts:
  min: ......................................................................... 42
  max: ......................................................................... 1264
  mean: ........................................................................ 127.8
  median: ...................................................................... 125.2
  p95: ......................................................................... 133
  p99: ......................................................................... 257.3
vusers.completed: .............................................................. 10960
vusers.created: ................................................................ 10980
vusers.created_by_name.ShiftStatus - All Skills: ............................... 6594
vusers.created_by_name.ShiftStatus - Skill Shift Status: ....................... 4386
vusers.failed: ................................................................. 20
vusers.session_length:
  min: ......................................................................... 2454.2
  max: ......................................................................... 7904.4
  mean: ........................................................................ 2611.4
  median: ...................................................................... 2566.3
  p95: ......................................................................... 2893.5
  p99: ......................................................................... 3828.5
Log file: test-run-report_shiftstatus_performance.json

Process finished with exit code 21

[hassy]

@jplieninger our docs should've been better (will update) - that exit code is set by the expect plugin when there are failed expectations (here). You can see them in the logs in the test run you shared.

(We'll need to display those failures in the report by default as well, it's pretty confusing)