JSON and JUnit formatters are empty when the pipeline is terminated due to timeout

[vlevkoniuk]

👓 What did you see?

Hi,

We have a lot of tests built with cucumber.js and we are using Azure Devops to run the tests in a pipeline based on the schedules

and sometimes we have the pipeline timed out (we have 3 hrs pipeline timeout) due to long-running cases and retries, so if one component 1 is not able to produce results another component will fail as expected, and as a result the output XML and json are empty.

The result of such a behavior is

1. pipeline results have no Tests tab with the test summary because it is looking for junit format xml
2. not able to generate cucumber report because it is relying on json output
3. it is hard to quick-analyze what part of the system is defective without analyzing the logs

The files are created just after starting tests but it seems that it is populated only after all the tests are completed (including failed with retries)

Can we produce data that is populated to the XML and json and populate those files on-the-fly just to be able to access test results summaries in any case?

✅ What did you expect to see?

When the test pipeline is terminated or locally you terminate cucumber tests execution those file should reflect the test results before termination

📦 Which tool/library version are you using?

node 22
cucumber 11.2.0 (older have the same problems, checked starting from 8.9.*)

🔬 How could we reproduce it?

1. run test suite that consists from several scenarios
2. at the middle of execution terminate the execution (to simulate the behavior of CI/CD systems that reached timeout)
3. XML and json files are empty

📚 Any additional context?

cucumber.cjs

const common = {
    loader: ['ts-node-maintained/esm'],
    format: [
        '@cucumber/pretty-formatter',
    ],
    formatOptions: {
        snippetInterface: "async-await",
    },
    import: ['src/**/*.ts'],
    tags: "not @skip"
};

const ci = {
    ...common,
    format: [
        ...common.format,
        "allure-cucumberjs/reporter:allure",
        'json:./reports/cucumber.json',
        'html:./reports/cucumber-embedded.html',
        'junit:./reports/cucumber.xml',
        ['json', './reports/cucumber-new.json'],
    ],
    formatOptions: {
        ...common.formatOptions,
        resultsDir: "allure-results"
    },
    retry: 3
};

const local = {
    ...ci,
    retry: 0
};

module.exports = {
    default: common,
    ci: ci,
    local: local
};

test run by:
npm run test:ci

package,json

{
      ...
      scripts: {
          "build": "npm run lint && tsc --build",
          "lint": "eslint src/",
          "test:ci": "npm run test -- --profile ci",
      }
}

[davidjgoss]

Thanks for raising @vlevkoniuk.

You're right that both formatters don't write anything to the file stream until the test run finishes. There is, I think, a sort of unwritten principal that they don't write unless it's going to result in syntactically valid output. This is difficult with both JSON and XML since they both have a structure that requires elements to be closed. That's one of the bigger reasons our newer standard message protocol uses newline-delimited JSON so its output can be progressive and streamable.

With that said, I'll address each one specifically:

• The JSON formatter is in maintenance mode now, so direct enhancements are out of the question, but we do have the standalone JSON formatter which can be used to take the message formatter output (which is progressive) and convert that to the JSON format. This is probably worth a try for your use case, as I imagine you're already doing some post-Cucumber file orchestration to turn that JSON into a report via another tool.
• The JUnit XML formatter is driven by the standard messages and while we could look at making it more progressive, it would be very limiting since many useful attributes are on the root <testsuite> element and can't be known until the end. There's also the matter of whether CI tools like Azure Devops etc would tolerate slightly malformed XML when scanning for test results - something to investigate.

So progressive output from formatters is desirable but has practical difficulties. What we could and should look at though is making Cucumber exit gracefully when possible, so e.g. when it gets a SIGINT then halt test execution, end the test run as unsuccessful and give the formatters a chance to wrap up. For cucumber-js this covered in cucumber/cucumber-js#2257, and I've made a more explicit reference to the formatters use case there.

(cc @mpkorstanje @luke-hill from the other day)