integ-runner

---

> The APIs of higher level constructs in this module are experimental and under active development.
> They are subject to non-backward compatible changes or removal in any future version. These are
> not subject to the Semantic Versioning model and breaking changes will be
> announced in the release notes. This means that while you may use them, you may need to update
> your source code when upgrading to a newer version of this package.

---

Overview

This tool has been created to be used initially by this repo (aws/aws-cdk). Long term the goal is
for this tool to be a general tool that can be used for running CDK integration tests. We are
publishing this tool so that it can be used by the community and we would love to receive feedback
on use cases that the tool should support, or issues that prevent the tool from being used in your
library.

This tool is meant to be used with the integ-tests library.

Usage

- Run all integration tests in test directory

``bash integ-runner [ARGS] [TEST...]`

This will look for all files that match the naming convention of /integ..(js|ts)$/ or /integ_.py. Each of these files represents a self contained AWS CDK app, defining test cases and assertions using the integ-tests library.

By default, executing integ-runner will do the following for each file (app):

1. Synth each integration test to create a new snapshot in memory 2. Compare the snapshot to the previous version (i.e. files in.snapshot/*), and fail on any differences.\ For new tests this will always fail.

To accept a snapshot update, the integreation test has to be deployed and assertions have to pass.\ Execute the runner again, this time withinteg-runner --update-on-fail and the following will happen for each file:

1. Deploy the previous snapshot 2. Deploy the new version as a Stack update and run assertions 3. If successful, update the snapshots

All snapshot files (i.e. .snapshot/*) must be checked-in to version control. If not, changes cannot be compared across systems and the update workflow cannot be used.

`$3`

- --update-on-failed (default=false) Rerun integration tests if snapshot fails ---clean (default=true) Destroy stacks after deploy (use--no-cleanfor debugging) ---verbose (default=false) verbose logging, including integration test metrics (specify multiple times to increase verbosity) ---parallel-regions (default=us-east-1,us-east-2, us-west-2) List of regions to run tests in. If this is provided then all tests will be run in parallel across these regions ---directory (default=test) Search for integration tests recursively from this starting directory ---force (default=false) Rerun integration test even if the test passes ---profilesList of AWS Profiles to use when running tests in parallel ---exclude (default=false) If this is set totruethen the list of tests provided will be excluded ---strict (default=false) Fail if any specified tests are not found. Cannot be used with--exclude---from-fileRead the list of tests from this file ---disable-update-workflow (default=false) If this is set totruethen the update workflow will be disabled ---appThe custom CLI command that will be used to run the test files. You can include {filePath} to specify where in the command the test file path should be inserted. Example: --app="python3.8 {filePath}".

Use together with --test-regex to fully customize how tests are run, or use with a single --languagepreset to change the command used for this language. ---test-regexDetect integration test files matching this JavaScript regex pattern. If used multiple times, all files matching any one of the patterns are detected.

- --watchRun a single integration test in watch mode. In watch mode the integ-runner will not save any snapshots.

Use together with --app to fully customize how tests are run, or use with a single --languagepreset to change which files are detected for this language. ---languageThe language presets to use. You can discover and run tests written in multiple languages by passing this flag multiple times (--language typescript --language python). Defaults to all supported languages. Currently supported language presets are: -javascript: - File RegExp:^integ\..*\.js$- App run command:node {filePath}-typescript:\ Note that for TypeScript files compiled to JavaScript, the JS tests will take precedence and the TS ones won't be evaluated. - File RegExp:^integ\..*(?
- App run command: node -r ts-node/register {filePath}-python: - File RegExp:^integ_.*\.py$- App run command:python {filePath}-go: - File RegExp:^integ_.*\.go$- App run command:go run {filePath}---list (default=false) List tests instead of running them. ---inspect-failures (default=false) Keep generated snapshots when differences exist in snapshot comparisons. ---max-workers (default=16) The max number of workerpool workers to use when running integration tests concurrently.

Example:

`bash integ-runner --update-on-failed --parallel-regions us-east-1 --parallel-regions us-east-2 --parallel-regions us-west-2 --directory ./ --language python`

This will search for python integration tests recursively from the current directory and then execute them in parallel across us-east-1, us-east-2, & us-west-2.

If you are providing a list of tests to execute, either as CLI arguments or from a file, the name of the test needs to be relative to the directory. For example, if there is a testaws-iam/test/integ.policy.js and the current working directory is aws-iam you would provide integ.policy.js

`bash integ-runner integ.policy.js`

`$3`

A common workflow to use when running integration tests is to first run the integration tests to see if there are any snapshot differences.

`bash integ-runner`

If there are any differences you might see an output like the output below.

`bash integ-runner

Verifying integration test snapshots...

eks-cluster-private-endpoint No Change! eks-inference No Change! alb-controller No Change! eks-oidc-provider No Change! eks-bottlerocket-ng No Change! eks-cluster No Change! fargate-cluster No Change! eks-cluster-handlers-vpc No Change! eks-helm-asset - Snapshot changed! Resources [+] Custom::AWSCDK-EKS-HelmChart Clustercharttestocichart9C188967

Snapshot Results:

Tests: 1 failed, 9 total Error: Some snapshot tests failed! To re-run failed tests run: integ-runner --update-on-failed at main (packages/@aws-cdk/integ-runner/lib/cli.js:90:15) error Command failed with exit code 1.`

To re-run the integration test for the failed tests you would then run:

`bash integ-runner --update-on-failed`

This will run the snapshot tests and collect all the failed tests. It will then re-execute the integration test for the failed tests and if successful, save the new snapshot.

`bash integ-runner --update-on-failed

Verifying integration test snapshots...

Snapshot Results:

Tests: 1 failed, 9 total

Running integration tests for failed tests...

Running in parallel across: us-east-1, us-east-2, us-west-2 Running test test/integ.eks-helm-asset.js in us-east-1 eks-helm-asset Test Succeeded!

Test Results:

Tests: 1 passed, 1 total`

Nested stack templates are also compared as part of the snapshot. However asset hashes are ignored by default. To enable diff for asset hashes, set diffAssets: true of IntegTestProps.

#### Update Workflow

By default, integration tests are run with the "update workflow" enabled. This can be disabled by using the --disable-update-workflow command line option.

If an existing snapshot is being updated, the integration test runner will first deploy the existing snapshot and then perform a stack update with the new changes. This is to test for cases where an update would cause a breaking change only on a stack update.

The integ-runner will also attempt to warn you if you are making any destructive changes with a message like:

`bash !!! This test contains destructive changes !!! Stack: aws-cdk-lambda-1 - Resource: MyLambdaServiceRole4539ECB6 - Impact: WILL_DESTROY Stack: aws-cdk-lambda-1 - Resource: AliasAliasPermissionAF30F9E8 - Impact: WILL_REPLACE Stack: aws-cdk-lambda-1 - Resource: AliasFunctionUrlDC6EC566 - Impact: WILL_REPLACE Stack: aws-cdk-lambda-1 - Resource: Aliasinvokefunctionurl4CA9917B - Impact: WILL_REPLACE !!! If these destructive changes are necessary, please indicate this on the PR !!!`

If the destructive changes are expected (and required) then please indicate this on your PR.

##### New tests

If you are adding a new test which creates a new snapshot then you should run that specific test with --disable-update-workflow. For example, if you are working on a new testinteg.new-test.js then you would run:

`bash integ-runner --update-on-failed --disable-update-workflow integ.new-test.js`

This is because for a new test we do not need to test the update workflow (there is nothing to update).

`$3`

It can be useful to run an integration test in watch mode when you are iterating on a specific test.

`console integ-runner integ.new-test.js --watch`

In watch mode the integ test will run similar to cdk deploy --watchwith the addition of also displaying the assertion results. By default the output will only show the assertion results.

- To show the console output from watch run with -v- To also stream the CloudWatch logs (i.e.cdk deploy --watch --logs) run with -vv

When running in watch mode most of the integ-runner functionality will be turned off.

- Snapshots will not be created - Update workflow will not be run - Stacks will not be cleaned up (you must manually clean up the stacks) - Only a single test can be run

Once you are done iterating using watch and want to create the snapshot you can run the integ test like normal to create the snapshot and clean up the test.

#### cdk.context.json

cdk watch depends on a cdk.context.json file existing with a watchkey. The integ-runner will create a defaultcdk.context.jsonfile if one does not exist.

`json { "watch": {} }`

You can further edit this file after it is created and add additional watchfields. For example:

`json { "watch": { "include": ["*/.js"] } }`

`$3`

See @aws-cdk/cloud-assembly-schema/lib/integ-tests/schema.ts

`$3`

See the @aws-cdk/integ-testsmodule for information on how to define integration tests for the runner to exercise.

`$3`

All options can be configured via the integ.config.json configuration file in the current working directory.

`json { "maxWorkers": 10, "parallelRegions": [ "eu-west-1", "ap-southeast-2" ] }`

Available options can be listed by running the following command:

`sh integ-runner --help`

To use a different config file, provide the --config` command-line option.

integ-runner

---

!cdk-constructs: Experimental

---

Overview

This tool is meant to be used with the integ-tests library.

Usage

- Run all integration tests in test directory

``bash integ-runner [ARGS] [TEST...]`

By default, executing integ-runner will do the following for each file (app):

1. Deploy the previous snapshot 2. Deploy the new version as a Stack update and run assertions 3. If successful, update the snapshots

All snapshot files (i.e. .snapshot/*) must be checked-in to version control. If not, changes cannot be compared across systems and the update workflow cannot be used.

`$3`

- --watchRun a single integration test in watch mode. In watch mode the integ-runner will not save any snapshots.

Example:

`bash integ-runner --update-on-failed --parallel-regions us-east-1 --parallel-regions us-east-2 --parallel-regions us-west-2 --directory ./ --language python`

This will search for python integration tests recursively from the current directory and then execute them in parallel across us-east-1, us-east-2, & us-west-2.

`bash integ-runner integ.policy.js`

`$3`

A common workflow to use when running integration tests is to first run the integration tests to see if there are any snapshot differences.

`bash integ-runner`

If there are any differences you might see an output like the output below.

`bash integ-runner

Verifying integration test snapshots...

Snapshot Results:

To re-run the integration test for the failed tests you would then run:

`bash integ-runner --update-on-failed`

This will run the snapshot tests and collect all the failed tests. It will then re-execute the integration test for the failed tests and if successful, save the new snapshot.

`bash integ-runner --update-on-failed

Verifying integration test snapshots...

Snapshot Results:

Tests: 1 failed, 9 total

Running integration tests for failed tests...

Running in parallel across: us-east-1, us-east-2, us-west-2 Running test test/integ.eks-helm-asset.js in us-east-1 eks-helm-asset Test Succeeded!

Test Results:

Tests: 1 passed, 1 total`

Nested stack templates are also compared as part of the snapshot. However asset hashes are ignored by default. To enable diff for asset hashes, set diffAssets: true of IntegTestProps.

#### Update Workflow

By default, integration tests are run with the "update workflow" enabled. This can be disabled by using the --disable-update-workflow command line option.

The integ-runner will also attempt to warn you if you are making any destructive changes with a message like:

If the destructive changes are expected (and required) then please indicate this on your PR.

##### New tests

`bash integ-runner --update-on-failed --disable-update-workflow integ.new-test.js`

This is because for a new test we do not need to test the update workflow (there is nothing to update).

`$3`

It can be useful to run an integration test in watch mode when you are iterating on a specific test.

`console integ-runner integ.new-test.js --watch`

In watch mode the integ test will run similar to cdk deploy --watchwith the addition of also displaying the assertion results. By default the output will only show the assertion results.

- To show the console output from watch run with -v- To also stream the CloudWatch logs (i.e.cdk deploy --watch --logs) run with -vv

When running in watch mode most of the integ-runner functionality will be turned off.

- Snapshots will not be created - Update workflow will not be run - Stacks will not be cleaned up (you must manually clean up the stacks) - Only a single test can be run

Once you are done iterating using watch and want to create the snapshot you can run the integ test like normal to create the snapshot and clean up the test.

#### cdk.context.json

cdk watch depends on a cdk.context.json file existing with a watchkey. The integ-runner will create a defaultcdk.context.jsonfile if one does not exist.

`json { "watch": {} }`

You can further edit this file after it is created and add additional watchfields. For example:

`json { "watch": { "include": ["*/.js"] } }`

`$3`

See @aws-cdk/cloud-assembly-schema/lib/integ-tests/schema.ts

`$3`

See the @aws-cdk/integ-testsmodule for information on how to define integration tests for the runner to exercise.

`$3`

All options can be configured via the integ.config.json configuration file in the current working directory.

`json { "maxWorkers": 10, "parallelRegions": [ "eu-west-1", "ap-southeast-2" ] }`

Available options can be listed by running the following command:

`sh integ-runner --help`

To use a different config file, provide the --config` command-line option.

@aws-cdk/integ-runner

integ-runner

Overview

Usage

$3

$3

$3

$3

$3

$3

@aws-cdk/integ-runner

integ-runner

Overview

Usage

$3

$3

$3

$3

$3

$3

Dist Tags

`$3`

`$3`

`$3`

`$3`

`$3`

`$3`

`$3`

`$3`

`$3`

`$3`

`$3`

`$3`