Data Archive Manager
npm install @itk-wasm/damData Archive Manager (dam): a tool to prepare and download source code repository testing, example, and website asset data.
Data is stored in .tar.gz archives and can be hosted at multiple, redundant https URLs. A simpler, more flexible alternative to Git LFS, etc.
- Download and unpack testing and example data .tar.gz archives
- CLI for NPM scripts
- Verify with an expected hash compatible with web3.storage
- Cache archive downloads based on the expected hash
- Support for multiple storage urls
- Automatic retries
``sh`
npm install -g @itk-wasm/dam
Store your data in a directory. For example:
`sh`
> mkdir -p test/data
> touch test/data/file1.txt
> touch test/data/file2.txt
Pack the current version of the data into a compressed archive:
`sh`
> dam pack test/data test/data.tar.gz
CID: bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
Upload the test/data.tar.gz file to hosting services such as web3.storage, GitHub Releases, AWS s3, a local http server, etc. For web3.storage, upload with the w3 CLI:
``
> npm install --global @web3-storage/w3
> w3 token
> w3 put --name dam-test-data --no-wrap test/data.tar.gzPacked 1 file (0.0MB)
bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
ā Stored 1 file
ā https://w3s.link/ipfs/bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
Note the --no-wrap flag.
In your NPM scripts, add
`sh`
> dam download -v test/data test/data.tar.gz bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy https://w3s.link/ipfs/bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
If the data is stored at multiple locations, add all URLs to the end of the download` subcommand.
The first time this command is called the data will be downloaded and unpacked.
When data updates are required, repeat the pack / upload process, and update the content identifer (CID) hash.