Skip to content

[Bug]: The exit code for the bal test command when there are failures is zero which is wrong #44410

@TharmiganK

Description

@TharmiganK

Description

$Subject

In older versions, we are correctly getting a non-zero exit code. This might be regression issue

Steps to Reproduce

Run the following test:

import ballerina/test;

@test:Config {}
function test() returns error? {
    test:assertEquals("a", "b");
}
Compiling source
        test.bal

Running Tests

        test.bal

        test: has failed.


                [fail] test:

                    error {ballerina/test:0}TestError ("Assertion Failed!
                         
                        expected: 'b'
                        actual  : 'a'
                         
                        Diff    :

                        --- actual
                        +++ expected 
                         
                         @@ -1,1 +1,1 @@ 
                         
                         -a
                        +b
                        ")
                                callableName: createBallerinaError moduleName: ballerina.test.0 fileName: assert.bal lineNumber: 41
                                callableName: assertEquals moduleName: ballerina.test.0 fileName: assert.bal lineNumber: 109
                                callableName: test  fileName: test.bal lineNumber: 5
                                callableName: test$lambda0$  fileName: test.bal lineNumber: 8
                                callableName: call moduleName: ballerina.lang.function.0 fileName: function.bal lineNumber: 37
                                callableName: executeTestFunction moduleName: ballerina.test.0 fileName: serialExecuter.bal lineNumber: 128
                                callableName: executeNonDataDrivenTest moduleName: ballerina.test.0 fileName: serialExecuter.bal lineNumber: 74
                                callableName: executeTest moduleName: ballerina.test.0 fileName: serialExecuter.bal lineNumber: 30
                                callableName: executeTests moduleName: ballerina.test.0 fileName: execute.bal lineNumber: 81
                                callableName: startSuite moduleName: ballerina.test.0 fileName: execute.bal lineNumber: 53
                                callableName: __execute__  fileName: test.bal lineNumber: 12



                0 passing
                1 failing
                0 skipped

                Test execution time : 0.027s

Check the exit code:

$ echo $?                     
0

Affected Version(s)

Ballerina SwanLake Update 13 - 2201.13.0 and 2201.13.1

OS, DB, other environment details and versions

No response

Related area

-> Test Framework

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions