js-wacz

  

JavaScript module and CLI tool for working with web archive data using the WACZ format specification, similar to Webrecorder's py-wacz.

It can be used to combine a set of .warc / .warc.gz files into a single .wacz file:

... programmatically (Node.js):
``javascript
import { WACZ } from '@harvard-lil/js-wacz'

const archive = new WACZ({
input: 'collection/*.warc.gz',
output: 'collection.wacz',
})

await archive.process() // "my-collection.wacz" is ready!
`

... or via the command line:
`bash
js-wacz create -f "collection/*.warc.gz" -o "collection.wacz"
`

js-wacz makes use of workers to process as many WARC files in parallel as the host machine can handle.

---

Summary


- Install
- CLI: create command
- Programmatic use
- Feature parity with py-wacz
- Development

---

Install

js-wacz requires Node JS 18+.

npm can be used to install this package and make the js-wacz command accessible system-wide:

`bash
npm install -g @harvard-lil/js-wacz
`

👆 Back to summary

---

CLI: create command

The create command helps combine one or multiple .warc or .warc.gz files into a single .wacz file.

`bash
js-wacz create -f "collection/*.warc.gz" -o "collection.wacz"
`

js-wacz accepts the following options and arguments for customizing how the WACZ file is assembled.

$3

This is the only required argument, which indicates what file(s) should be processed and added to the resulting WACZ file.

The target can be a single file, or a glob pattern such as folder/*.warc.gz.

`bash


Single file:


js-wacz create --file archive.warc
`

`bash


Collection:


js-wacz create --file "collection/*.warc"
`

Note: When using globs, make sure to surround the path with quotation marks.

$3

Specify where the resulting .wacz file should be created, and what its filename should be.

Defaults to archive.wacz in the current directory if not provided.

`bash
js-wacz create --file cool-beans.warc --output cool-beans.wacz
`

$3

Path to a folder containing pages.jsonl files (pages.jsonl, extraPages.jsonl ...).

If not provided, js-wacz is going to attempt to detect pages in WARC records to build its own pages.jsonl index.

`bash


Assuming the following file exists: /collections/pages/pages.jsonl


js-wacz create -f "collection/*.warc.gz" --pages collection/pages/
`

$3

Pass a directory of existing CDXJ files, rather than indexing from WARCs. Must be used in combination with --pages.

`bash
js-wacz create -f "collection/*.warc.gz" --pages collection/pages.jsonl --cdxj collection/indexes/
`

$3

If provided, will be used as the mainPageUrl attribute for datapackage.json.

Must be a valid URL.

`bash
js-wacz create -f "collection/*.warc.gz" --url "https://lil.law.harvard.edu"
`

$3

If provided, will be used as the mainPageDate attribute for datapackage.json.

Can be any value that can be parsed by JavaScript's Date() constructor.

`bash
js-wacz create -f "collection/*.warc.gz" --ts "2023-02-22T12:00:00.000Z"
`

$3

If provided, will be used as the title attribute for datapackage.json.

`bash
js-wacz create -f "collection/*.warc.gz" --title "My collection."
`

$3

If provided, will be used as the description attribute for datapackage.json.

`bash
js-wacz create -f "collection/*.warc.gz" --desc "My cool collection of web archives."
`

$3

If provided, will be used as an API endpoint for applying a cryptographic signature to the resulting WACZ file.

This endpoint is expected to be authsign-compatible.

`bash
js-wacz create -f "collection/*.warc.gz" --signing-url "https://example.com/sign"
`

$3

Used conjointly with --signing-url if provided, in case the signing server requires authentication.

`bash
js-wacz create -f "collection/*.warc.gz" --signing-url "https://example.com/sign" --signing-token "FOO-BAR"
`

$3

Can be used to determine how verbose js-wacz needs to be.

- Possible values are: silent, trace, debug, info, warn, error
- Default is: info

`bash
js-wacz create -f "collection/*.warc.gz" --log-level trace
`


👆 Back to summary

---

Programmatic use

js-wacz's CLI and underlying logic are decoupled, and it can therefore be consumed as a JavaScript module _(currently only with Node.js)_.

Example: Creating a signed WACZ programmatically
`javascript
import { WACZ } from '@harvard-lil/js-wacz'

try {
const archive = new WACZ({
file: 'collection/*.warc.gz',
output: 'collection.wacz',
signingUrl: 'https://example.com/sign',
signingToken: 'FOO-BAR',
}

await archive.process()

// collection.wacz is ready
} catch(err) {
// ...
}
`

Although a process() convenience method is made available, every step of said process can be run individually and the archive's state inspected / edited throughout.

$3


- WACZ.addPage() allows for manually adding an entry to pages.jsonl.
- WACZ.addFileToZip() allows for manually adding any additional data to the final WACZ file.
- The datapackageExtras option allows for adding an arbitrary JSON-serializable object to datapackage.json under extras.

$3

- WACZ Class
- Available options

👆 Back to summary

---

Feature parity with py-wacz

js-wacz is aiming at partial feature parity with webrecorder's py-wacz, similar to Webrecorder's py-wacz.

This section lists notable differences in implementation that might affect interoperability.

Main differences in currently implemented features:
- CLI: create --detect-pages: --detect-pages is implied in js-wacz unless --pages is provided.
- CLI: create --file: that argument can be implied in py-wacz, it is always explicit in js-wacz.

👆 Back to summary

---

Development

$3


This codebase uses the Standard JS coding style.
- npm run lint can be used to check formatting.
- npm run lint-autofix can be used to check formatting _and_ automatically edit files accordingly when possible.
- Most IDEs can be configured to automatically check and enforce this coding style.

$3

JSDoc is used for both documentation and loose type checking purposes on this project.

$3

This project uses Node.js' built-in test runner.

`bash
npm run test
`

#### Tests-specific environment variables
The following environment variables allow for testing features requiring access to a third-party server.

These are optional, and can be added to a local .env file which will be automatically interpreted by the test runner.

| Name | Description |
| --- | --- |
| TEST_SIGNING_URL | URL of an authsign-compatible endpoint for signing WACZ files.
To run such an endpoint locally, use npm run dev-signer, which will overwrite .env and set this variable to http://localhost:5000/sign; see .services/signer.|
| TEST_SIGNING_TOKEN | If required by the server at TEST_SIGNING_URL, an authentication token. |




$3

`bash


Runs test suite

npm run test

Runs linter

npm run lint

Runs linter and attempts to automatically fix issues

npm run lint-autofix

Step-by-step NPM publishing helper

npm run publish-util

Runs a local instance of wacz-signer for test purposes (see "Testing" section)

npm run dev-signer
``

👆 Back to summary