@itk-wasm/dam

Data Archive Manager (dam): a tool to prepare and download source code repository testing, example, and website asset data.

Data is stored in .tar.gz archives and can be hosted at multiple, redundant https URLs. A simpler, more flexible alternative to Git LFS, etc.

Features

- Download and unpack testing and example data .tar.gz archives
- CLI for NPM scripts
- Verify with an expected hash compatible with web3.storage
- Cache archive downloads based on the expected hash
- Support for multiple storage urls
- Automatic retries

Installation

``sh
npm install -g @itk-wasm/dam
`

Usage

Store your data in a directory. For example:

`sh
> mkdir -p test/data
> touch test/data/file1.txt
> touch test/data/file2.txt
`

Pack the current version of the data into a compressed archive:

`sh
> dam pack test/data test/data.tar.gz
CID: bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
`

Upload the test/data.tar.gz file to hosting services such as web3.storage, GitHub Releases, AWS s3, a local http server, etc. For web3.storage, upload with the w3 CLI:

`
> npm install --global @web3-storage/w3
> w3 token
> w3 put --name dam-test-data --no-wrap test/data.tar.gz


Packed 1 file (0.0MB)

bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy

⁂ Stored 1 file
⁂ https://w3s.link/ipfs/bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
`

Note the --no-wrap flag.

In your NPM scripts, add

`sh
> dam download -v test/data test/data.tar.gz bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy https://w3s.link/ipfs/bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
`

If the data is stored at multiple locations, add all URLs to the end of the download` subcommand.

The first time this command is called the data will be downloaded and unpacked.

When data updates are required, repeat the pack / upload process, and update the content identifer (CID) hash.