Skip to content

Latest commit

 

History

History
472 lines (350 loc) · 18.3 KB

README.md

File metadata and controls

472 lines (350 loc) · 18.3 KB
Raw

Running DMD's test suite

This is the documentation and guide for DMD's test suite. See src/README.md for additional checks performed alongside these dedicated tests.

Maybe you are looking for the contributing guide too?

Quick guide

Run all tests

./run.d

Note:

  • run.d will automatically use all available threads. Use e.g. -j4 if you need a lower parallelism
  • all commands below assume that you are in the test directory

Run only a specific subset

./run.d fail
./run.d compilable

As linking is slow the runnable tests take a bit longer to run:

./run.d runnable

Run only an individual test

./run.d fail_compilation/diag10089.d

Multiple arguments are supported too. You can use ./run.d to quickly run a custom subset of tests. For example, all diagnostic tests in fail_compilation:

./run.d fail_compilation/diag*.d

Automatically update the TEST_OUTPUT segments

Often, when you add a new error message, a few tests need to be updated as their TEST_OUTPUT has changed. This is tedious work and AUTO_UPDATE can be to automate it:

AUTO_UPDATE=1 ./run.d fail

Updating the TEST_OUTPUT can also be done for a custom subset of tests:

./run.d fail_compilation/diag*.d AUTO_UPDATE=1

Note:

  • you might need to run this command twice if you add a new error message(s) as then the line numbers of the following error messages will change
  • AUTO_UPDATE doesn't work with tests that have no, empty or multiple TEST_OUTPUT segments
  • AUTO_UPDATE can be set as an environment variable or as Makefile-like argument assignment

Running the Unit Tests

The unit tests will automatically run when all tests are run using ./run.d. To only run the unit tests the ./run.d unit_tests command can be used. For a more finer grain control over the unit tests the ./run.d -u command can be used:

To run all unit tests:

./run.d -u

To only run the unit tests in one or more specific files:

./run.d -u unit/deinitialization.d

To only run a subset of the unit tests in a single file:

./run.d -u unit/deinitialization.d --filter Expression

In the above example, the --filter flag will filter to only run the tests with a UDA matching the given value, in this case Expression.

@("Target.deinitialize")
unittest {}

@("Expression.deinitialize")
unittest {}

Of the above unit tests, only the second one will be run, since --filter Expression was specified.

The --filter flag works when no files are specified as well.

Types of Tests

There are two types of tests in the DMD test suite:

  • End-to-end test. These are tests that invokes the compiler as an external process in some kind of way. Then it asserts either the exit code or the output of the compiler. These tests are located in

  • Unit tests. These tests are more of a unit test, integration or functional style tests. These tests are using the compiler as a library. They are more flexible because they can assert state internal to the compiler which the end-to-end tests would never have access to. The unit test runner will compile all files in the unit directory into a single executable and run the tests. This should make it quick to run the tests since only a single process need to be started.

run.d targets

default | all:      run all unit tests that haven't been run yet

run_tests:          run all tests
run_runnable_tests:         run just the runnable tests
run_compilable_tests:       run just the compilable tests
run_fail_compilation_tests: run just the fail compilation tests
unit_test:                  run all unit tests (those in the "unit" directory)

quick:              run all tests with no default permuted args
                    (individual test specified options still honored)

clean:              remove all temporary or result files from previous runs

test_results/compilable/json.d.out      runs an individual test
                                        (run log of the test is stored)

Test Configuration

All tests defined within .d source files may use various settings to configure how they are to be run, i.e.

compilable/hellotest.d:

/*
REQUIRED_ARGS: -version=Foo
TEST_OUTPUT:
---
Hello, World!
---
*/
void main(string[] args)
{
    version(Foo)
    {
        pragma(msg, "Hello World");
    }
}

Test parameters can be restricted to certain targets by adding a brace-enclosed condition after the name, i.e. REQUIRED_ARGS(<condition>): .... The <condition> consists of the target operating system followed by an optional model suffix, e.g. linux, win32, freebsd64.

Valid platforms:

  • win
  • linux
  • osx
  • freebsd
  • dragonflybsd
  • netbsd

Valid models:

  • 32
  • 64

Note that test parameters MUST be followed by a colon (intermediate whitespace is allowed). The test runner will issue an error for a missing colon (e.g. REQUIRED_ARGS foo) to avoid ambiguities. Test directives embedded within other words (e.g. OPTLINK) will be ignored.

The following is a list of all available settings:

ARG_SETS:            sets off extra arguments to invoke $(DMD) with (seperated by ';').
                     default: (none)

COMPILE_SEPARATELY:  if present, forces each .d file to compile separately and linked
                     together in an extra setup. May specify additional parameters which
                     are passed to $(DMD) when linking the generated object files.
                     default: (none, aka compile/link all in one step)

COMPILED_IMPORTS:    list of modules files that are imported by the main source file that
                     should be included in compilation; this differs from the EXTRA_SOURCES
                     variable in that these files could be compiled by either explicitly
                     passing them to the compiler or by using the "-i" option. Using this
                     option will cause the test to be compiled twice, once using "-i" and
                     once by explicitly passing the modules to the compiler.
                     default: (none)

CXXFLAGS:            list of extra arguments passed to $(CC) when compiling C++ sources
                     defined in EXTRA_CPP_SOURCES.
                     default: (none)

DFLAGS:              Overrides the DFLAGS environment variable if specified in the test.
                     No values are permitted; an error will be emitted if the value is not
                     empty.

DISABLED:            selectively disable the test on specific platforms (if empty, the test is
                     considered to be enabled on all platform). Target platforms are specified
                     using nearly the same syntax as conditions of optional parameters, except for
                     `win` instead of `windows`.
                     Potential filters are `win32`, `linux`, ...
                     default: (none, enabled)

EXECUTE_ARGS:        parameters to add to the execution of the test
                     default: (none)

EXTRA_CPP_SOURCES:   list of extra C++ files to build and link along with the test
                     default: (none).

EXTRA_FILES:         list of extra files and sources used by the test, either during
                     compilation or execution of the test. It is currently ignored by the test
                     runner, but serves as documentation of the test itself.
                     default: (none)

EXTRA_OBJC_SOURCES:  list of extra Objective-C files to build and link along with the test
                     default: (none). Test files with this variable will be ignored unless
                     the D_OBJC environment variable is set to "1"

EXTRA_SOURCES:       list of extra files to build and link along with the test
                     default: (none)

GDB_MATCH:           a regular expression describing the expected output of GDB_SCRIPT. The test
                     will fail if it does not match the actual output.
                     default: (none)

GDB_SCRIPT:          if present, starts a `gdb` session for the compiled executable to run the commands
                     specified in the corresponding section. GDB_MATCH may be used to used to verfiy
                     expected output using a regex.
                     note: restricted to `runnable` tests, the executable will not be run outside of the
                           gdb session.
                     default: (none)

LINK:                enables linking (used for the compilable and fail_compilable tests).
                     default: (none)

OUTPUT_FILES:       files generated during the compilation (separated by ';').
                    The content of each file is appended to the output of the
                    compilation (in the order of this list) according to the HAR
                    format (https://code.dlang.org/packages/har).
                    Example:
                    ------------------------------------------
                    <Compilation Output>
                    === <FILENAME_1>
                    <CONTENT_1>
                    === <FILENAME_2>
                    <CONTENT_2>
                    [...]
                    ------------------------------------------
                    The merged output will then be prepared and compared to the
                    expected TEST_OUTPUT as defined below.
                    default: (none)

PERMUTE_ARGS:        the set of arguments to permute in multiple $(DMD) invocations.
                     An empty set means only one permutation with no arguments.
                     default: the make variable ARGS (see below)

POST_SCRIPT:         name of script to execute after test run
                     note: arguments to the script may be included after the name.
                           additionally, the name of the file that contains the output
                           of the compile/link/run steps is added as the last parameter.
                     default: (none)

REQUIRED_ARGS:       arguments to add to the $(DMD) command line
                     default: (none)
                     note: the make variable REQUIRED_ARGS is also added to the $(DMD)
                           command line (see below)

RUN_OUTPUT:         output expected from running the compiled executable which must match
                    the actual output. The comparison adheres to the rules defined for
                    TEST_OUTPUT and allow e.g. using special sequences as defined below.

TEST_OUTPUT:         the output is expected from the compilation (if the
                     output of the compilation doesn't match, the test
                     fails). You can use the this format for multi-line
                     output:
                     TEST_OUTPUT:
                     ---
                     Some
                     Output
                     ---
                     note: if not given, it is assumed that the compilation will be silent.
                     default: (none)

TEST_OUTPUT_FILE:   file containing the expected output as defined for TEST_OUTPUT.
                    note: Further TEST_OUTPUT sections in the test are ignored.
                    default: (none)

TRANSFORM_OUTPUT:   steps to apply to the output of the compilation before it
                    is compared to the expected TEST_OUTPUT. A step may take
                    arguments akin to a function call, e.g. `step(arg)` and arguments
                    may be quoted using "".

                    Supported transformations:
                    - sanitize_json:    Remove compiler specific information from output
                                        of -Xi (see test/tools/sanitize_json.d)
                                        arguments: none

                    - sanitize_timetrace: Parse output of -ftime-trace profiling data,
                                        extract event name / detail strings, and sort them.
                                        Raw profiler output can't be tested against because it
                                        contains non-deterministic timing data, or implementation
                                        details such as semantic analysis order and memory usage,
                                        so use this when writing a test for -ftime-trace.

                    - remove_lines:     Remove lines matching a given regex
                                        arguments: the regex
                                        note: patterns containing ')' must be quoted

UNICODE_NAMES:      file containing symbols with unicode characters in their name, which might
                    not be supported on some specific platforms. It is currently ignored by the
                    test runner, but serves as documentation of the test itself.
                    default: (none)

Environment variables

run.d uses environment variables to store test settings and as a way to pass these settings to the test wrapper tool d_do_test.d.

Note: These variables are also available inside any Bash test.

ARGS:          set to execute all combinations of
AUTO_UPDATE:   set to 1 to auto-update mismatching test output
CC:            C++ compiler to use, ex: cl, g++
DMD:           compiler to use, ex: ../src/dmd (required)
MODEL:         32 or 64 (required)
OS:            windows, linux, freebsd, osx, netbsd, dragonflybsd
REQUIRED_ARGS: arguments always passed to the compiler
RESULTS_DIR:   base directory for test results

Windows vs non-windows portability env vars:

DSEP:          \\ or /
EXE:          .exe or <null> (required)
OBJ:          .obj or .o (required)
SEP:           \ or / (required)

Bash Tests

Along with the environment variables provided by run.d (see above), an additional set of environment variables are made available to Bash tests. These variables are defined in tools/exported_vars.sh:

EXTRA_FILES        directory for extra files of this test type, e.g. runnable/extra-files

LIBEXT             platform-specific extension for library files, e.g. .a or .lib

RESULTS_TEST_DIR   the results directory for tests of this type, e.g. test_results/runnable

OUTPUT_BASE        the prefix used for test output files, e.g. test_results/runnable/mytest

SOEXT              platform-specific extension for shared object files (aka. dynamic libraries),
                   e.g. .so, .dll or .dylib

TEST_DIR           the name of the test directory
                   (one of compilable, fail_compilation or runnable)

TEST_NAME          the base name of the test file without the extension, e.g. test15897

Test configuration variables

Sometimes test configuration arguments must be dynamic. For example, the output of all tests should be placed into RESULTS_DIR:

// REQUIRED_ARGS: -mixin=${RESULTS_DIR}/fail_compilation/mixin_test.mixin

Currently these variables are exposed:

RESULTS_DIR       Path to `test_results`

TEST_OUTPUT

A few operations are done on the output of a test before the comparison with TEST_OUTPUT:

  • newlines get unified for consistent TEST_OUTPUT between platforms
  • DMD's debug message (e.g. DMD v2.084.0-255-g86b608a15-dirty DEBUG) gets stripped away
  • paths to test_results will be replaced with {{RESULTS_DIR}}

TEST_OUTPUT offers the following special sequences to match error messages which depend on the current platform and target:

$n$             arbitrary amount of digits

$p:<tail>$      paths ending with <tail> (which must refer to an existing file or directory)

$?:<choices>$   selection based on the current environment where a choice is either
                conditional `<condition>=<content>` or a fallback value `<default>`.
                Multiple choices are separated by `|` and the leftmost satisfied condition
                or fallback is chosen if multiple choices apply.

                Supported conditions:
                - OS: posix, windows, ...
                - Model: 64, 32

$r:<regex>$     any text matching <regex> (using $ inside of <regex> is not
                supported, use multiple regexes instead)

Both stderr and stdout of the DMD are captured for output comparison.

Test Coding Practices

The purpose of the test suite is to test the compiler only. This means:

  • do not import modules from Phobos
  • keep imports from druntime to the interface to the C standard library
  • use core.stdc.stdio.printf, not std.stdio.writef

In order to make the test suite run faster, multiple unrelated tests can be aggregated into a single file, for example test/runnable/test42.d

Each test should be in the following form:

/*******************************/
// https://issues.dlang.org/show_bug.cgi?id=NNNN

void testNNNN()
{
}

The NNNN is the bugzilla issue number this test ensures is fixed. The test code should be self-contained. The test code should be minimized to focus on the test.

As usual, test source code should be LF terminated lines, and not contain any tab characters.