Option to toggle the scenario run

[ppcano]

Feature Description

When using multiple scenarios, you might want to have a quick way to run the test with or without a particular scenario.

The suggested solution is to use environment variables to edit the scenarios object.

A dummy example:

const scenarios = {
  scenario_1: {
    executor: "shared-iterations",
    iterations: 1,
    vus: 1,
  },
  scenario_2: {
    executor: "shared-iterations",
    iterations: 1,
    vus: 1,
  },
  scenario_3: {
    executor: "shared-iterations",
    iterations: 1,
    vus: 1,
  },
};

if (__ENV.SKIP_SCENARIO1) {
  delete scenarios["scenario_1"];
}
if (__ENV.SKIP_SCENARIO2) {
  delete scenarios["scenario_2"];
}
if (__ENV.SKIP_SCENARIO3) {
  delete scenarios["scenario_3"];
}

export const options = {
  scenarios: scenarios,
};

Suggested Solution

Changed the initial proposal based on @pablochacin's suggestion

Provide an scenario common option to toggle the scenario execution.

export const options = {
  scenarios: {
    scenario_1: {
      executor: "shared-iterations",
      iterations: 1,
      vus: 1,
      skip: __ENV.SKIP_SCENARIO1,
    },
    scenario_2: {
      executor: "shared-iterations",
      iterations: 1,
      vus: 1,
      skip: __ENV.SKIP_SCENARIO2,
    },
    scenario_3: {
      executor: "shared-iterations",
      iterations: 1,
      vus: 1,
      skip: __ENV.SKIP_SCENARIO3,
    },
  },
};

And/or using the CLI:

k6 run --skip scenario_1,scenario_2 test.js

Already existing or connected issues / PRs (optional)

https://community.k6.io/t/how-to-execute-single-scenario-out-of-multiple-scenarios-in-a-script/1573

[pablochacin]

I find the possibility of executing specific scenarios a basic functionality of any testing platform.

I don't have an strong opining here if the interface should allow selecting which to execute or which to exclude or both.

I personally find this proposal of using an attribute in the scenario in conjunction with environment variables inconvenient because requires modifying the scenarios.

I would prefer to add CLI parameters to k6 to enable this feature

k6 --skip scenario_1,scenario_2

This same possibility could be added as an configuration option:

options = {
     skip: ["scenario_1"] 

In practice, this feature could be implemented by setting the noRun attribute to the scenarios listed in the CLI parameters, but this implementation detail is hidden from the user.

[na--]

Hmm, if we were to document an "officially suggested" solution to this problem with the currently existing k6 versions, I wouldn't recommend the one from the OP, something like this looks much nicer:

function filterScenarios(scenarios) {
  // ... filter the scenarios however the user wants - by __ENV blocklist,
  // by __ENV allowlist, by noRun property, whatever...
}

export const options = {
  scenarios: filterScenarios({
    scenario_1: {
      // ...
    },
    // ...
  })
};

This looks quite clean and can be as simple or as flexible as is required by the user and their use case... And given how easy it is to implement a filterScenarios function, especially with something like an unofficial noRun scenario property, I am very much against adding more official properties to the scenarios JS configuration... I am not even convinced we need to do something at all, but if we determine that, then a a CLI flag seems like the way to go, instead of things that can be easily done with simple JS code. Assuming we can do it without making #883 much worse, of course.

However, both of the proposals share another problem - they use negative configuration. noRun and skip specify which scenarios must NOT be executed, which is trickier to reason about. In general, I've found it's much better for UX if such options are positive. So I suggest to flip it around and have a way to tell k6 "run these specific scenarios" with a CLI flag. This is also how I've seen it done in other testing tools, e.g. go test -run '^TestMatchRegex$' ./... This type of filter is also easier to validate and provide a meaningful error to the user when they have a wrong config, e.g. that no scenarios matched the filter.

Besides, we don't actually have any way to configure scenarios from the CLI flags or environment variables. We left that for the future when we were originally implementing them, since it's quite tricky to cram such a complex configuration in a single string value... Maybe we can have both under the same flag 🤔 Something like this:

k6 run --scenario "sc1" --scenario "sc2" script.js # run only sc1 and sc2
k6 run --scenario "scp*"  # run only the previously configured scenarios that start with scp
k6 run --scenario "executor=per-vu-iterations,iterations=100,vus=5" # actually configure the scenario directly here

This deserves quite a lot of thought, since I can see a bunch of problems with the proposal even as I make it. But, if we figure it out, it will bring much more value than just adding a blacklist to configure which scenarios should not be executed.

[mstoykov]

I don't think more configuration options is the way. In my experience providing support, people don't know and can't find some of the current ones. So having more of them when we don't need them seems like making the problem worse, not easier.

So making a blog post with how to do it with JS seems like a much better way forward.

I have also mostly seen the case where people want to run one specific scenario, usually with 1 iteration (w/e that means for each executor). Usually in the context of trying to work on a specific scenario. This seems like a lot more complex (and impossible for some executors) thing to script in js and might be worth an option 🤷 . The current workaround is to extract its default function and run it with -i 1 -u 1 most of the cases.

We left that for the future when we were originally implementing them, since it's quite tricky to cram such a complex configuration in a single string value

I was left with the impression we left it because we don't want to implement it as it seems not very useful. It is very unlikely anyone will try to write more then 1 scenario in the CLI. And doing it in env variables is "supported" even now if you just write it as the json and then parse it in the script.

But it does bring questions such as how do we merge scenario options ... which arguably already confuses people a lot see #2529.

Again IMO adding more configuration options is not making things easier as it means we need to explain how they all work together. Having them in JS code removes (to some extend) that part by doing it in js code the user can see and in a place in the configuration merging that is known (the execution of the init context).

[ppcano]

In my experience providing support, people don't know and can't find some of the current ones. So having more of them when we don't need them seems like making the problem worse.

@mstoykov Even though I understand your view, I am not convinced with this argument. I think useful APIs with proper docs are easier to discover and improve the day-to-day experience.

I find the possibility of executing specific scenarios a basic functionality of any testing platform.

I am with @pablochacin here.

• Mocha skip
• Jest skip
• Jasmine x means exclude
• JUnit disabled
• Maven skip

I have updated the proposal to use skip and the CLI flag.

[na--]

Test skipping is something that can also be done with Go tests, but its purpose has generally been to temporarily disable some broken tests until you have time to fix or rewrite them. From what I can see by skimming the links @ppcano gave, this is the case for most other testing frameworks, and it makes a lot of sense in that context.

However, that means we need to go back in our discussion and, before discussing how the solution should look like, first agree on what the problem we are trying to solve actually is 😅 Because the use case for these test skipping functions doesn't seem to be the same one we have in k6. This sentence from the OP doesn't describe the problem very well:

When using multiple scenarios, you might want to have a quick way to run the test with or without a particular scenario.

In my experience, there have often been at least 3 somewhat connected use cases that have caused users to want to do something like this:

1. During development of a complicated test with multiple scenarios, the developer wants to run only a specific scenario in isolation, so it's easier to develop and debug it. Usually, as @mstoykov says, with 1 VU and 1 iteration.
2. When an organization has a complex test script with multiple scenarios, sometimes it's impractical to run the whole test every time. It might be too long or too heavy. So the users might want to run the big tests once a day or once a week, but to run some parts (scenarios?) of it on every commit. Or maybe every team might want to run the part that tests their service on every commit, in isolation, but test all services together only occasionally.
3. The other side of 2, which is probably more common. An organization might have started by having a bunch of separate tests, e.g. every team developed one or more k6 tests for the services they are responsible for. The organization might want to be able to then combine all of the tests programmatically, to run all or a sub-set of them together, in a single test, to verify the whole system or a bigger part of it.

Here are some forum topics I could find that corroborate this, though I am probably missing a bunch:

• https://community.k6.io/t/how-to-execute-single-scenario-out-of-multiple-scenarios-in-a-script/1573
• https://community.k6.io/t/multiple-scenario-with-single-file/2846
• https://community.k6.io/t/can-we-import-js-files-in-another-k6-js-file-with-scenarios-in-it/1918
• https://community.k6.io/t/running-10-test-cases-out-of-100/448
• https://community.k6.io/t/run-all-tests-scripts-in-parallel/1149

In short, it doesn't seem like skipping tests is what our users actually want. They kind of want the opposite, to be able to run only a specific sub-set of a larger test. Or, often, a sub-set of a larger test suite, so it's arguable if scenarios are even the right abstraction layer we need to concern ourselves with, "test suites" (#1342) might be a prerequisite for this feature... 🤔 And all of these frameworks that you linked to also have an abstraction that is a better match for the use cases described above, usually called something like only, tagging and filtering, conditional execution, etc:

• Mocha only()
• Jest only()
• Jasmine f means "focused"
• JUnit apparently has tagging and filtering and conditional execution
• Maven unsurprisingly also has conditional execution
• and, as I already mentioned, Go has regexp test matching (search for -run) and you can also execute only the tests in a specific package, not the whole project

Though, again, even these are not a great fit for the current k6 capabilities, with single "tests" that may have multiple scenarios... 😞 At least until we implement test suites (#1342), k6 won't have the same semantics as other tools that use skip or exclude, or execute only a single test. A k6 "test" is usually not just a single scenario, it is the whole combination between setup, teardown, scenarios and options (especially thresholds)...

To conclude, I am still very much against adding a skip property to scenarios. I don't have a good alternative solution, but I think we don't even agree on what the problem we are trying to solve is, so we need to start from that, before we even discuss solutions. And my gut feeling is that, whatever we decide, it may actually require test suites to be implemented first.

[wisecoders]

Workaround:

This is not an efficient but works

I have created two files and conditionally executing the test.

[bystedtpontus]

I did this the other way around from OP as I always just want to run one scenario at a time:

const SCENARIO = __ENV.scenario || 'stress';
const scenarios = {
    constant: { ... },
    stress: { ... }
};

export let options = {
    ... ,
    scenarios: {}
};

// Use the chosen scenario, defaulting to scenario 'stress'
options.scenarios[SCENARIO] = scenarios[SCENARIO];

[fdenUM]

Is any new about this?

We have an API Owner that has multiple methods in his API.

Each method has a different URL, and its load scenario is very different one from another. One can be called 60 times in a minute, and the other one can be called 1000 times in a minute.

We have a k6 script which includes one function (and one scenario) per each method of the API.

The API Owner is doing his development, and he only wants to test the method that is developing, not every method every time.

Without the ability of choose which scenario we want to execute, we have to apply the workaround described above to select which scenario we want to execute. The alternative is having one script for each method, but we would have to create 2.000 scripts aprox...

With JMeter it was very easy, just disabling the Thread Group. I think it is a basic functionality too.

[jochan-clgx]

Inputs:

I want to use k6 to do regression and performance testing in DRY. First, let's see if the endpoint is functional, then to do performance testing. Developer will start writing individual regression test cases and scenarios. I used the config to describe all different scenarios

scenarios.json

{
  "userAgent": "grafana-k6",
  "scenarios": {
    "infoOk": {
      "exec": "infoOk",
    },

    "healthOk": {
      "exec": "healthOk",
    },

    "scanOk": {
      "exec": "scanOk"
    }
  }
}

But if I want to borrow scenario "scanOk" and want to run performance test with different parameters like vus, iterations ... etc. I can easily pass in as well.

For example of running regression testing
k6 run --config scenarios.json script.js

Or for running performance testing in one of the scenario
k6 run --config scenarios.json --scanOk --vus n --duration xm --iterations 20

I am hoping not to repeat the same scenario to do regression and performance; difference is the options only.
Today my workaround is to have another config.json and performance.js script