Core lib and cli for Playwright test orchestration
npm install @dionlarson/playwright-orchestrator-coreA CLI tool for orchestrating and managing Playwright test execution.
Helps to orchestrate Playwright test execution through smart sharding using existing database.
The project provides tooling to analyze and orchestrate Playwright tests using available storage plugin options. Currently available plugin options:
- file: Basic storage that uses a local file as storage.
- dynamo-db: Amazon's DynamoDB adapter.
- pg: PostgreSQL adapter.
- mysql: MySQL adapter.
- mongo: MongoDB adapter.
Tests are ordered as follows:
1. If there are no previous test runs, use the default timeout.
2. Take the EMA (Exponential Moving Average) of the test duration. The window size can be changed in create command, default value is 10.
3. If there was a failed test in the chosen window, it is more likely to fail again. Therefore, the formula is adjusted as follows:
- Calculate the EMA of the test duration.
- Adjust the EMA by adding a factor based on the percentage of failed tests in the window.
- The final formula is: EMA + EMA * (% of failed tests in window)
Example:
- If the EMA of the test duration is 2 minutes and 4 of 10 tests in the window failed, the adjusted duration would be:
2 + 2 * 0.4 = 2.8 minutes
Serial tests duration is a sum of all included tests.
In order to keep history of tests, they need to be identified. There are multiple possible cases:
1. Default id: [{project}] {file} > {title of the test}.
2. Serial test id: [{project}] {file} > {title of the parent group}.
3. Serial test defined at the file level: [{project}] {file}.
4. Custom Id: [{project}] {custom id}.
In case some of these attributes changed, test history would be recreated. To preserve test history between changes, you can add a static attribute. Adding an id to an existing test would recreate the history as well.
✅ Examples
``
import { id } from '@playwright-orchestrator/core/annotations';
test('test', { annotation: id('#custom_id') }, () => {
// test code
}
`
import { id } from '@playwright-orchestrator/core/annotations';
test.describe('serial tests', { annotation: id('#custom_id') }, () => {
// test code
}
❌ This won't work
`
import { id } from '@playwright-orchestrator/core/annotations';
test.describe('test', () => {
test.info().annotations.push(id('#custom_id'));
// test code
}
Make sure Playwright is installed by following Playwright's installation guide.
`bash
npm install @playwright-orchestrator/core --save-dev
npm install @playwright-orchestrator/
`
Run the CLI tool:
`bash`
npx playwright-orchestrator
- Mongo
- DynamoDB
1. Run the init command. Required to run once to set up storage. Make sure that executing credentials have all permissions.
`bash`
npx playwright-orchestrator init pg --connection-string "postgres://username:password@localhost:5432/postgres"
2. Run the create command. Required to run once per run.
`bash`
npx playwright-orchestrator create pg --connection-string "postgres://username:password@localhost:5432/postgres" --workers 2
> 019464f7-1488-75d1-b5c0-5e7d3dde9195
3. Start the desired count of shards using the run command. Run ID is generated in the previous step. All Playwright options are already saved in the previous step.
`bash`
npx playwright-orchestrator run pg --connection-string "postgres://username:password@localhost:5432/postgres" --run-id 019464f7-1488-75d1-b5c0-5e7d3dde9195
4. Failed tests can be started again using step 3.
5. Merge report using Playwright's Merge-reports CLI
`bash`
npx playwright merge-reports ./blob-reports --reporter html
Seeds data storage with necessary tables and initial configuration.
No additional options.
Creates and configures a new test run. Outputs created run ID. Supports most of playwright's options.
| Option | Description | Type | Default | Required? |
| ------------------ | ----------------------------------------------------------------------------------- | -------- | ------- | --------- |
| --history-window | Count of runs history kept and window for average duration. More here | number | 10 | no |
Starts a test shard for the provided test run. If used with a finished run, it will only start failed tests.
Command generates blob reports into --output directory. To merge them use Playwright's Merge-reports CLI
#### Setup Phase
Each shard automatically runs these setup operations once before executing tests:
1. globalSetup - Runs your globalSetup file (if configured in playwright.config.ts)webServer
2. webServer - Starts your (if configured) and waits for it to be ready
3. Setup projects - Runs any projects that are dependencies but have no dependencies themselves
After all tests complete (or on failure), teardown runs in reverse order: stop webServer → run globalTeardown.
Note: Each shard is independent (runs on different machines), so setup runs once per shard, not once globally.
| Option | Description | Type | Default | Required? |
| ------------------------ | -------------------------------------------------- | -------- | -------------- | --------- |
| --run-id | Run ID generated by create command | string | - | yes |-o, --output
| | Directory for artifacts produced by tests | string | blob_reports | no |--skip-global-setup
| | Skip globalSetup (use when managed externally) | - | - | no |--skip-web-server
| | Skip webServer startup (use when already running) | - | - | no |--skip-setup-projects
| | Skip setup project execution | - | - | no |
Generates report for provided test run id.
| Option | Description | Type | Default | Required? |
| ------------ | ------------------------------------ | ----------------- | ------- | --------- |
| --run-id | Run ID generated by create command | string | - | yes |--reporter
| | Type of reporter | 'json' \| 'gha' | json | no |
GHA Report Example
Each command has corresponding subcommands for installed storages.
Each storage option can be parsed from env variable. For example, table-name-prefix -> TABLE_NAME_PREFIX.
Use as a storage locally created file
| Option | Description | Type | Default | Required? |
| ------------- | -------------------------------- | -------- | --------- | --------- |
| --directory | Directory to store test run data | string | test-runs | no |
Use Amazon's DynamoDB as storage. Credentials are taken from AWS profile
| Option | Description | Type | Default | Required? |
| --------------------- | --------------------- | -------- | ------------------------- | --------- |
| --table-name-prefix | Table(s) name prefix | string | 'playwright-orchestrator' | no |--ttl
| | TTL in days | number | 30 | no |--endpoint-url
| | DynamoDB endpoint URL | string | - | no |
Use PostgreSQL as storage.
| Option | Description | Type | Default | Required? |
| --------------------- | -------------------- | -------- | ------------------------- | --------- |
| --connection-string | Connection string | string | - | yes |--table-name-prefix
| | Table(s) name prefix | string | 'playwright-orchestrator' | no |--ssl-ca
| | SSL CA | string | - | no |--ssl-cert
| | SSL certificate | string | - | no |--ssl-key
| | SSL key | string | - | no |
Use MySQL as storage.
| Option | Description | Type | Default | Required? |
| --------------------------------- | -------------------------------------------- | -------- | ------------------------- | --------- |
| --connection-string | Connection string | string | - | yes |--table-name-prefix
| | Table(s) name prefix | string | 'playwright-orchestrator' | no |--ssl-profile
| | The SSL profile overrides other SSL options. | string | - | no |--ssl-ca
| | SSL CA | string | - | no |--ssl-cert
| | SSL certificate | string | - | no |--ssl-key
| | SSL key | string | - | no |--ssl-passphrase
| | SSL passphrase | string | - | no |--ssl-reject-unauthorized
| | SSL reject unauthorized | - | - | no |--ssl-verify-server-certificate
| | SSL verify server certificate | - | - | no |
Use MongoDB as storage.
| Option | Description | Type | Default | Required? |
| ---------------------------------- | ------------------------------------- | -------- | ------------------------- | --------- |
| --connection-string | Connection string | string | - | yes |--db
| | Database name | string | - | yes |--collection-name-prefix
| | Table(s) name prefix | string | 'playwright-orchestrator' | no |--tls
| | Enable TLS | - | - | no |--tls-ca
| | SSL CA | string | - | no |--tls-key
| | SSL key | string | - | no |--tls-key-password
| | SSL key password | string | - | no |--tls-passphrase
| | SSL passphrase | string | - | no |--tls-allow-invalid-certificates
| | Allow invalid certificates | - | - | no |--tls-allow-invalid-hostnames
| | Allow invalid hostnames | - | - | no |--tls-allow-invalid-certificates
| | Allow invalid certificates | - | - | no |--tls-insecure
| | Allow insecure | - | - | no |--debug
| | Add extra fields for some collections | string | - | no |
Make sure podman and compose is installed. They used for tests and local development.
Build or use npm run watch.
Make sure you run npm run cli-permissions and npm run link-packages`
See packages.json .scripts section for more commands.
Licensed under the Apache License 2.0. See LICENSE.md for details.
- ✅ ~~Tests~~
- ✅ ~~Better Error Handling~~
- ✅ ~~MySQL adapter~~
- ✅ ~~MongoDB adapter~~
- ✅ ~~Tests improvements~~
- ✅ ~~Better Logging~~
- ✅ ~~Test History statistics (test duration trends, count of test failures for past n days, etc.)~~
- ✅ ~~Smarter test ordering based on previous execution duration~~
- ✅ ~~GHA reporter~~
- ⬜ Redis adapter
- ⬜ Even more adapters (by request)
- ⬜ More examples
- ⬜ Create Documentation site.
- ? Restore unfinished tests in case shard terminated (Can be simply fixed by creating new run)
This library was created in a couple weeks of free time, so issues may occur, but I try to address them as quickly as I can. Don't hesitate to create an issue report or contribute.
For now, the best way to support this project is to star it on GitHub and share it with your colleagues or the community.