Nx s3 cache using GNU tar, pigz and multipart s3 uploads/downloads
npm install @uizard/nx-fast-s3-cachePopular caching libraries prioritize portability over speed, but this prevents Nx users from caching humongous folders, like node_modules. This module prioritizes speed over anything else:
1. Uses GNU tar with pigz for native, multithreaded compression, instead of using the npm tar package.
2. Uses multipart uploads and downloads for larger files when communicating with S3
1. GNU tar (or compatible) executable at /usr/bin/tar
2. The pigz executable should be in PATH so tar can find it
``bash`
npm install @uizard/nx-fast-s3-cache --save-dev
| Parameter | Description | Environment Variable / .env | nx.json |NXCACHE_S3_ENDPOINT
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ---------------- |
| Endpoint | Optional. The fully qualified endpoint of the webservice. This is only required when using a custom (non-AWS) endpoint. | | endpoint |NXCACHE_S3_BUCKET
| Bucket | Optional. Specify which bucket should be used for storing the cache. | | bucket |NXCACHE_S3_PREFIX
| Prefix | Optional. Specify prefix path of target object key. | | prefix |NXCACHE_S3_REGION
| Region | Optional. The AWS region to which this client will send requests. | | region |NXCACHE_S3_PROFILE
| Profile | Optional. The AWS profile to use to authenticate. | | profile |https://s3.amazonaws.com/
| Force Path Style | Optional. Whether to force path style URLs for S3 objects (e.g., instead of https:// | NXCACHE_S3_FORCE_PATH_STYLE | forcePathStyle |NXCACHE_S3_READ_ONLY
| Read Only | Optional. Disable writing cache to the S3 bucket. This may be useful if you only want to write to the cache from a CI but not localhost. | | readOnly |NXCACHE_S3_WRITE_ONLY
| Write Only | Optional. Disable reading cache from the S3 bucket. This may be useful if your builds depend on external factors that you cannot easily take into account during build and you want to restart a build without automatically fetching the cached results. | | writeOnly |
nx.json:
`json`
{
"tasksRunnerOptions": {
"ci-runner": {
"runner": "@uizard/nx-fast-s3-cache",
"options": {
"cacheableOperations": ["init", "build", "test", "lint", "e2e"],
"endpoint": "https://some-endpoint.com",
"bucket": "name-of-bucket",
"prefix": "prefix/",
"region": "us-west-000",
"profile": "name-of-aws-profile",
"forcePathStyle": true,
"readOnly": false
}
},
"default": {
"runner": "@nx/workspace/tasks-runners/default",
"options": {
"cacheableOperations": []
}
}
},
"namedInputs": {
"dependency-files-collection": [
"{projectRoot}/package-lock.json",
"{projectRoot}/yarn.lock",
"{projectRoot}/requirements*.txt",
"{projectRoot}/pyproject.toml",
"{projectRoot}/.nvmrc",
"{projectRoot}/.python-version"
]
},
"targetDefaults": {
"init": {
"inputs": ["dependency-files-collection"],
"outputs": ["{projectRoot}/venv", "{projectRoot}/node_modules"]
}
}
}
Authentication is handled by @aws-sdk/credential-provider-node, so credentials will be attempted to be found from the following sources (listed in order of precedence):
- Environment variables exposed via process.env (example: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
- SSO credentials from token cache
- Web identity token credentials
- Shared credentials and config ini files
- The EC2/ECS Instance Metadata Service
Running tasks should now show the storage or retrieval from the remote cache, with additional information on the duration of each step:
`
nx run --runner=ci-runner
...
...
------------------------------------------------------------------------------
Stored to remote cache: S3 (total:13163ms/compress:10104ms/upload:3059ms)
File: d5b82b049e031a312104d3f196f459a6c551c5e1a4a510c1f8a9bfe0f391d998.tar.gz
------------------------------------------------------------------------------
`
This package couldn't depend on the packages it got inspiration from because of their hardcoded dependencies to npm tar` or their simple (non-multipart) communication with s3. Still, we have to recognize that most of the code of this package (including this README) comes from these 2 npm packages: