@testledger/cli

Test orchestration CLI for Test Ledger - automatically skip flaky and quarantined tests.

Installation

``bash
npm install -g @testledger/cli
`

Quick Start

`bash


Login with your Test Ledger credentials

testledger login --api-token YOUR_API_TOKEN

Check project status (flaky tests, quarantined tests)

testledger status --project-id 123

Run tests, automatically skipping flaky tests

testledger run --project-id 123 -- npx wdio run wdio.conf.js
`

Features

- Skip Flaky Tests: Automatically skip tests that have been flagged as flaky in Test Ledger
- Quarantine Tests: Skip tests that are quarantined/broken
- WebDriverIO Support: Full support for WebDriverIO test framework
- Flaky Mode Options: Choose to skip, warn, or fail on flaky tests

> Note: Playwright and Cypress support coming soon.

Commands

$3

Authenticate with Test Ledger.

`bash
testledger login --api-token YOUR_API_TOKEN
`

Options:
- -t, --api-token - Your API token
- --api-url - Custom API URL (default: https://app-api.testledger.dev)

$3

Show project status including flaky and quarantined tests.

`bash
testledger status --project-id 123

With custom flaky thresholds


testledger status --project-id 123 --min-flaky-percent 30 --min-flaky-count 5
`

Options:
- -p, --project-id - Project ID
- -v, --version - Filter by app version
- --min-flaky-count - Minimum flaky occurrences to be considered flaky (default: 3)
- --min-flaky-percent - Minimum percentage of runs that are flaky (default: 20)
- --min-total-runs - Minimum runs for statistical significance (default: 5)

$3

Run tests with automatic flaky/quarantine test skipping.

`bash


Basic usage - skip flaky tests

testledger run --project-id 123 -- npx wdio run wdio.conf.js

With flaky mode options

testledger run --project-id 123 --flaky-mode=skip -- npx wdio # Skip flaky tests (default)
testledger run --project-id 123 --flaky-mode=warn -- npx wdio # Run flaky tests, warn on failure
testledger run --project-id 123 --flaky-mode=fail -- npx wdio # Run flaky tests normally

Include quarantined tests

testledger run --project-id 123 --include-quarantined -- npx wdio

Dry run (show what would be excluded)

testledger run --project-id 123 --dry-run -- npx wdio
`

Options:
- -p, --project-id - Project ID
- -v, --version - App version
- --flaky-mode - How to handle flaky tests: skip, warn, fail (default: skip)
- --include-quarantined - Run quarantined tests anyway
- --framework - Force framework: wdio, playwright, cypress
- --dry-run - Show what would be excluded without running tests
- --min-flaky-count - Minimum flaky occurrences to be considered flaky (default: 3)
- --min-flaky-percent - Minimum percentage of runs that are flaky (default: 20)
- --min-total-runs - Minimum runs for statistical significance (default: 5)

Parallel Execution

For parallel test execution, use your test framework's built-in sharding. The CLI will apply the same flaky/quarantine exclusions to each shard.

$3

`bash


Shard 1 of 3

testledger run --project-id 123 -- npx wdio run wdio.conf.js --shard 1/3

Shard 2 of 3

testledger run --project-id 123 -- npx wdio run wdio.conf.js --shard 2/3

Shard 3 of 3

testledger run --project-id 123 -- npx wdio run wdio.conf.js --shard 3/3
`

$3

`yaml
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2, 3]
env:
TESTLEDGER_API_TOKEN: ${{ secrets.TESTLEDGER_TOKEN }}
TESTLEDGER_PROJECT_ID: 123
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
- run: npm ci
- run: npm install -g @testledger/cli
- name: Run tests
run: testledger run -- npx wdio run wdio.conf.js --shard ${{ matrix.shard }}/3
`

Framework Support

$3

The CLI automatically detects WebDriverIO projects and uses the --exclude flag to skip specs.

`bash
testledger run --project-id 123 -- npx wdio run wdio.conf.js
`

$3

The CLI will use --grep-invert to exclude tests by pattern.

`bash
testledger run --project-id 123 -- npx playwright test
`

$3

The CLI will pass exclusions via the TESTLEDGER_EXCLUDE environment variable. Add this to your cypress.config.js:

`javascript
const { defineConfig } = require('cypress');

module.exports = defineConfig({
e2e: {
excludeSpecPattern: process.env.TESTLEDGER_EXCLUDE
? process.env.TESTLEDGER_EXCLUDE.split(',')
: []
}
});
`

Then run:

`bash
testledger run --project-id 123 -- npx cypress run
`

Configuration

Credentials are stored in ~/.config/testledger-cli/config.json (Linux/Mac) or the appropriate config directory on Windows.

Environment Variables

- TESTLEDGER_API_TOKEN - API token (alternative to --api-token or testledger login)
- TESTLEDGER_PROJECT_ID - Default project ID (alternative to --project-id)
- TESTLEDGER_API_URL - Custom API URL
- DEBUG=1` - Enable debug logging

License

MIT