@open-xchange/codecept-runner

A runner script for E2E tests. Supports parallel execution, rerunning failed tests, and continuation.

Help

The switch --help (or -h) prints an overview with all supported switches.
Example:

``sh yarn codecept-runner --help`

`CodeceptJS Configuration File`

The runner needs to know the location of the CodeceptJS configuration file. If it is not located in the project root directory, the switch --config (or -c) allows to specify an alternative path.

Like CodeceptJS itself, the runner will try all well-known names for the configuration file:

- codecept.config.js-codecept.conf.js-codecept.json-codecept.config.ts-codecept.conf.ts

Example:

`sh

`by exact file name`


yarn codecept-runner --config=./e2e/special.config.ts
by path with auto-detection

yarn codecept-runner --config=./e2e


Filter by Test Titles

The tests to be executed can be selected by full test titles (including scenario names and tags) using the switches --grep (or -g) and --ignore (or -i). Both switches expect a regular expression.

- If --grepis omitted, all existing tests will be selected. - If--ignoreis omitted, no selected test will be skipped. - If--grep and --ignore are specified, the tests selected by both will be skipped.

Example:

`sh yarn codecept-runner --grep=@stable --ignore=@experimental`

will execute all tests with the tag "stable" that do not contain the tag "experimental".

`Retry Failed Tests`

Failed tests can be rerun automatically using the switch --retries (or -r). The number after the switch specifies the number of _additional_ test runs after the initial test run.

Example:

`sh $ yarn codecept-runner --retries=2 running 1000 tests ... 20 tests failed

retrying 20 tests ... 2 tests failed

retrying 2 tests ... all tests passed`

`Parallel Execution`

Tests will be executed in parallel with multiple workers.

By default, the initial number of workers is half of the number of processors available on the system. For every retry run, the number of workers will be halved to compensate for shaky tests caused by system load.

Example (on a system with 16 reported processors):

`sh $ yarn codecept-runner --retries=2 running 1000 tests with 8 workers ... 20 tests failed

retrying 20 tests with 4 workers ... 3 tests failed

retrying 3 tests with 2 workers ... all tests passed`

The number of workers can be changed using the switch --workers (or -w). The additional switch --retry-workers (or uppercase -W) allows to set a different number of workers for retrying failed tests.

Example:

`sh $ yarn codecept-runner --workers=4 --retries=1 --retry-workers=3 running 1000 tests with 4 workers ... 20 tests failed

retrying 20 tests with 3 workers ... all tests passed`

The option --retry-workers supports the keywords same and halve instead of a number. The keyword same causes to use the initial number of workers for all retry runs. The keyword halve (which is the default) causes to halve the number of workers for every retry run.

Example (on a system with 16 reported processors):

`sh $ yarn codecept-runner --retries=1 --retry-workers=same running 1000 tests with 8 workers ... 20 tests failed

retrying 20 tests with 8 workers ... all tests passed`

`Partitioning`

A chunk of all matching tests can be executed with the option --partition (or -P). Its value consists of the one-based chunk index and of the total number of chunks, separated with a forward slash.

`sh $ yarn codecept-runner --partition=2/10 found 1000 matching tests running chunk 2 of 10 with 100 tests all tests passed`

`Continuation`

The results of a test run will be saved in a JSON file in the configured output directory. After a test run finishes with failed tests, or has been interrupted, the remaining tests (failed and pending) can be restarted using the switch --continue (or uppercase -C).

Example:

`sh $ yarn codecept-runner --grep=@stable running 1000 tests ... 20 tests failed

$ yarn codecept-runner --continue running 20 tests ... all tests passed`

The path to the JSON file can be customized with the option --state-file (or uppercase -S).

Example:

`sh $ yarn codecept-runner --grep=@stable --state-file=./.temp/test-state.json running 1000 tests ... 20 tests failed

$ yarn codecept-runner --continue --state-file=./.temp/test-state.json running 20 tests ... all tests passed`

`Dry Run`

Using the switch --dry-run (or -d) the runner will print log messages but will not actually run any test.

Example:

`sh $ yarn codecept-runner --workers=4 --retries=1 --dry-run running 1000 tests with 4 workers ... 1000 tests skipped``

@open-xchange/codecept-runner

A runner script for E2E tests. Supports parallel execution, rerunning failed tests, and continuation.

Help

The switch --help (or -h) prints an overview with all supported switches.
Example:

``sh yarn codecept-runner --help`

`CodeceptJS Configuration File`

Like CodeceptJS itself, the runner will try all well-known names for the configuration file:

- codecept.config.js-codecept.conf.js-codecept.json-codecept.config.ts-codecept.conf.ts

Example:

`sh

`by exact file name`


yarn codecept-runner --config=./e2e/special.config.ts
by path with auto-detection

yarn codecept-runner --config=./e2e


Filter by Test Titles

Example:

`sh yarn codecept-runner --grep=@stable --ignore=@experimental`

will execute all tests with the tag "stable" that do not contain the tag "experimental".

`Retry Failed Tests`

Failed tests can be rerun automatically using the switch --retries (or -r). The number after the switch specifies the number of _additional_ test runs after the initial test run.

Example:

`sh $ yarn codecept-runner --retries=2 running 1000 tests ... 20 tests failed

retrying 20 tests ... 2 tests failed

retrying 2 tests ... all tests passed`

`Parallel Execution`

Tests will be executed in parallel with multiple workers.

Example (on a system with 16 reported processors):

`sh $ yarn codecept-runner --retries=2 running 1000 tests with 8 workers ... 20 tests failed

retrying 20 tests with 4 workers ... 3 tests failed

retrying 3 tests with 2 workers ... all tests passed`

Example:

`sh $ yarn codecept-runner --workers=4 --retries=1 --retry-workers=3 running 1000 tests with 4 workers ... 20 tests failed

retrying 20 tests with 3 workers ... all tests passed`

Example (on a system with 16 reported processors):

`sh $ yarn codecept-runner --retries=1 --retry-workers=same running 1000 tests with 8 workers ... 20 tests failed

retrying 20 tests with 8 workers ... all tests passed`

`Partitioning`

`sh $ yarn codecept-runner --partition=2/10 found 1000 matching tests running chunk 2 of 10 with 100 tests all tests passed`

`Continuation`

Example:

`sh $ yarn codecept-runner --grep=@stable running 1000 tests ... 20 tests failed

$ yarn codecept-runner --continue running 20 tests ... all tests passed`

The path to the JSON file can be customized with the option --state-file (or uppercase -S).

Example:

`sh $ yarn codecept-runner --grep=@stable --state-file=./.temp/test-state.json running 1000 tests ... 20 tests failed

$ yarn codecept-runner --continue --state-file=./.temp/test-state.json running 20 tests ... all tests passed`

`Dry Run`

Using the switch --dry-run (or -d) the runner will print log messages but will not actually run any test.

Example:

`sh $ yarn codecept-runner --workers=4 --retries=1 --dry-run running 1000 tests with 4 workers ... 1000 tests skipped``