node-pkware

Node JS implementation of StormLib's Pkware compression/decompression algorithm, which is not the same as the zip pkware
format that is commonly used today

It was the de-facto compression for games from around Y2K, like Arx Fatalis

installation / update existing version

npm i -g node-pkware

- minimum required node version: 18.0.0
- recommended node version (to be able to run the tests): 20.6.0

command line interface (CLI)

explode [] [--offset=] [--drop-before-offset] [--output= [--verbose]] - decompresses a file or a stream

implode [] [--offset=] [--drop-before-offset] [--output= [--verbose]] - compresses a file or a stream

, --output or both can be omitted when the input is being piped from stdin or when the output is being piped into stdout

The --offset can have a numeric value in either decimal or hexadecimal format which tells explode or implode to start decompression at a later point.
This is useful for partially compressed files where the initial header part is uncompressed while the remaining part is compressed.

The --drop-before-offset flag tells node-pkware to drop the portion before --offset, otherwise it will keep it untouched and attach it to the output file.

The --verbose flag will display additional information while running the commands

For implode can either be --ascii or --binary

For implode can either be --small, --medium or --large

Calling either explode or implode with only the -v or --version flag will display the package's version

examples

explode test/files/fast.fts --output=C:/fast.fts.decompressed --offset=1816

explode test/files/fast.fts --output=C:/fast.fts.decompressed --offset=0x718

implode test/files/fast.fts.unpacked --output=C:/fast.fts --binary --large --offset=1816

$3

cat c:/arx/level8.llf | explode > c:/arx/level8.llf.unpacked

explode c:/arx/level8.llf > c:/arx/level8.llf.unpacked

cat c:/arx/level8.llf | explode --output=c:/arx/level8.llf.unpacked

cat e:/piping/level8.llf.unpacked | implode --binary --large > e:/piping/level8.llf

implode e:/piping/level8.llf.unpacked --binary --large > e:/piping/level8.llf.comp

cat e:/piping/level8.llf.unpacked | implode --binary --large --output="e:/piping/level8.llf"

using as a library

$3

explode(config: object): transform._transform - decompresses stream

decompress(config: object): transform._transform - alias for explode

Returns a function, that you can use as a transform.\_transform method.
The returned function has the (chunk: Buffer, encoding: string, callback: function) parameter signature.

Takes an optional config object, which has the following properties:

``js
{
verbose: boolean // whether the code should display extra debug messages on the console or not (default = false)
}
`

implode(compressionType: int, dictionarySize: int, config: object): transform._transform - compresses stream

compress(compressionType: int, dictionarySize: int, config: object): transform._transform - alias for implode

Takes an optional config object, which has the following properties:

`js
{
verbose: boolean // whether the code should display extra debug messages on the console or not (default = false)
}
`

Returns a function, that you can use as a transform.\_transform method.
The returned function has the (chunk: Buffer, encoding: string, callback: function) parameter signature.

stream - an object of helper functions for channeling streams to and from explode/implode

stream.through(transformer: function): Transform - a function, which takes a transform._transform type function and turns it into a Transform stream instance

stream.transformEmpty(chunk: Buffer, encoding: string, callback: function) - a transform._transform type function, which for every input chunk will output an empty buffer

stream.transformIdentity(chunk: Buffer, encoding: string, callback: function) - a transform._transform type function, which lets the input chunks through without any change

stream.splitAt(index: int): function - creates a "predicate" function, that awaits Buffers, keeps an internal counter of the bytes from them and splits the appropriate buffer at the given index. Splitting is done by returning an array with [left: Buffer, right: Buffer, isLeftDone: bool]. If you want to split data at the 100th byte and you keep feeding 60 byte long buffers to the function returned by splitAt(100), then it will return arrays in the following manner:

`
1) [inputBuffer, emptyBuffer, false]
2) [inputBuffer.subarray(0, 40), inputBuffer.subarray(40, 60), true]
3) [emptyBuffer, inputBuffer, true]
4) [emptyBuffer, inputBuffer, true]
... and so on
`

stream.transformSplitBy(predicate: predicate, left: transform._transform, right: transform._transform): transform._transform - higher order function for introducing conditional logic to transform.\_transform functions. This is used internally to handle offsets for explode()

stream.toBuffer(callback: function): writable._write - data can be piped to the returned function from a stream and it will concatenate all chunks into a single buffer. Takes a callback function, which will receive the concatenated buffer as a parameter

constants.Compression.Binary and constants.Compression.Ascii - compression types for implode

constants.DictionarySize.Small, constants.DictionarySize.Medium and constants.DictionarySize.Large - dictionary sizes for implode, determines how well the file get compressed. Small dictionary size means less memory to lookback in data for repetitions, meaning it will be less effective, the file stays larger, less compressed. On the other hand, large compression allows more lookback allowing more effective compression, thus generating smaller, more compressed files. The original C library used less memory when the dictionary size was smaller, plus there might be files out there which only support smaller dictionary sizes

errors.InvalidDictionarySizeError - thrown by implode when invalid dictionary size was specified or by explode when it encounters invalid data in the header section (the first 2 bytes of a compressed files)

errors.InvalidCompressionTypeError - thrown by implode when invalid compression type was specified or by explode when it encounters invalid data in the header section (the first 2 bytes of a compressed files)

errors.AbortedError - thrown by explode when compressed data ends without reaching the end literal or in mid decompression

$3

#### decompressing file with no offset into a file

`js
const fs = require('node:fs')
const { explode, stream } = require('node-pkware')
const { through } = stream

fs.createReadStream(path-to-compressed-file)
.pipe(through(explode()))
.pipe(fs.createWriteStream(path-to-write-decompressed-data))
`

#### decompressing buffer with no offset into a buffer

`js
const { Readable } = require('node:stream')
const { explode, stream } = require('node-pkware')
const { through, toBuffer } = stream

Readable.from(buffer) // buffer is of type Buffer with compressed data
.pipe(through(explode()))
.pipe(
toBuffer((decompressedData) => {
// decompressedData holds the decompressed buffer
}),
)
`

#### decompressing file with offset into a file, keeping initial part intact

`js
const fs = require('node:fs')
const { explode, stream } = require('node-pkware')
const { through, transformSplitBy, splitAt, transformIdentity } = stream

const offset = 150 // 150 bytes of data will be skipped and explode will decompress the data that comes afterwards

fs.createReadStream(path-to-compressed-file)
.pipe(through(transformSplitBy(splitAt(offset), transformIdentity(), explode())))
.pipe(fs.createWriteStream(path-to-write-decompressed-data))
`

#### decompressing file with offset into a file, discarding initial part

`js
const fs = require('node:fs')
const { explode, stream } = require('node-pkware')
const { through, transformSplitBy, splitAt, transformEmpty } = stream

const offset = 150 // 150 bytes of data will be skipped and explode will decompress the data that comes afterwards

fs.createReadStream(path-to-compressed-file)
.pipe(through(transformSplitBy(splitAt(offset), transformEmpty(), explode())))
.pipe(fs.createWriteStream(path-to-write-decompressed-data))
`

#### Non-stream compression (also works in browser)

`js
import * as fs from 'node:fs/promises'
import { implode } from 'node-pkware/simple'

const inputString = 'hello pkware!'

// create an ArrayBuffer from inputString
const encoder = new TextEncoder()
const input = encoder.encode(inputString).buffer

// compress it and get the result in another ArrayBuffer
const output = implode(input, 'ascii', 'large')

// write the output to a file
await fs.writeFile('/tmp/compressedHello', new Uint8Array(output))
`

$3

`js
const fs = require('node:fs')
const { explode, stream } = require('node-pkware')
const { through } = stream

fs.createReadStream(path-to-compressed-file)
.on('error', (err) => {
console.error('readstream error')
})
.pipe(
through(explode()).on('error', (err) => {
console.error('explode error')
}),
)
.pipe(
fs.createWriteStream(path-to-write-decompressed-data).on('error', (err) => {
console.error('writestream error')
}),
)
``

Useful links

$3

https://github.com/arx-tools/pkware-test-files

$3

- https://github.com/ladislav-zezula/StormLib/tree/master/src/pklib
- https://github.com/ShieldBattery/implode-decoder
- https://github.com/TheNitesWhoSay/lawine/blob/master/lawine/misc/implode.c - nice find, @alexpineda!
- https://github.com/arx/ArxLibertatis/blob/master/src/io/Blast.cpp
- https://github.com/arx/ArxLibertatis/blob/229d55b1c537c137ac50096221fa486df18ba0d2/src/io/Implode.cpp

Implode was removed from Arx Libertatis at this commit: https://github.com/arx/ArxLibertatis/commit/2db9f0dd023fdd5d4da6f06c08a92d932e218187

$3

- https://stackoverflow.com/questions/2094666/pointers-in-c-when-to-use-the-ampersand-and-the-asterisk
- https://stackoverflow.com/a/49394095/1806628
- https://nodejs.org/api/stream.html