Encode data with built-in or custom encodings
npm install level-transcoderEncode data with built-in or custom encodings. The successor to [level-codec][level-codec] that transcodes encodings from and to internal data formats supported by a database. This allows a database to store data in a format of its choice (Buffer, Uint8Array or String) with zero-effort support of known encodings. That includes other encoding interfaces in the ecosystem like [abstract-encoding][abstract-enc] and [multiformats][blockcodec].
[![level badge][level-badge]](https://github.com/Level/awesome)
Create a transcoder, passing a desired format:
``js
const { Transcoder } = require('level-transcoder')
const transcoder1 = new Transcoder(['view'])
const transcoder2 = new Transcoder(['buffer'])
const transcoder3 = new Transcoder(['utf8'])
`
Then select an encoding and encode some data:
`js
// Uint8Array(3) [ 49, 50, 51 ]
console.log(transcoder1.encoding('json').encode(123))
//
console.log(transcoder2.encoding('json').encode(123))
// '123'
console.log(transcoder3.encoding('json').encode(123))
`
If the Transcoder constructor is given multiple formats then Transcoder#encoding() selects an encoding with the best fitting format. Consider a database like [leveldown][leveldown] which has the ability to return data as a Buffer or string. If an encoding.decode(data) function needs a string, we'll want to fetch that data from the database as a string. This avoids the cost of having to convert a Buffer to a string. So we'd use the following transcoder:
`js`
const transcoder = new Transcoder(['buffer', 'utf8'])
Then, knowing for example that the return value of JSON.stringify(data) is a UTF-8 string which matches one of the given formats, the 'json' encoding will return a string here:
`js`
// '123'
console.log(transcoder.encoding('json').encode(123))
In contrast, data encoded as a 'view' (for now that just means Uint8Array) would get transcoded into the 'buffer' encoding. Copying of data is avoided where possible, like how the underlying ArrayBuffer of a view can be passed to Buffer.from(..) without a copy.
Lastly, encodings returned by Transcoder#encoding() have a format property to be used to forward information to an underlying store. For example: an input value of { x: 3 } using the 'json' encoding which has a format of 'utf8', can be forwarded as value '{"x":3}' with encoding 'utf8'. Vice versa for output.
These encodings can be used out of the box and are to be selected by name.
In this table, the _input_ is what encode() accepts. The _format_ is what encode() returns as well as what decode() accepts. The _output_ is what decode() returns. The TypeScript typings of level-transcoder have generic type parameters with matching names: TIn, TFormat and TOut.
| Name | Input | Format | Output |
| :---------------------- | :------------------------- | :--------- | :-------------- |
| 'buffer' 1 | Buffer, Uint8Array, String | 'buffer' | Buffer |'view'
| | Uint8Array, Buffer, String | 'view' | Uint8Array |'utf8'
| | String, Buffer, Uint8Array | 'utf8' | String |'json'
| | Any JSON type | 'utf8' | As input |'hex'
| | String (hex), Buffer | 'buffer' | String (hex) |'base64'
| | String (base64), Buffer | 'buffer' | String (base64) |
1 Aliased as 'binary'. Use of this alias does not affect the ability to transcode.
It's not necessary to use or reference the below encodings directly. They're listed here for implementation notes and to show how input and output is the same; it's the format that differs.
Custom encodings are transcoded in the same way and require no additional setup. For example: if a custom encoding has { name: 'example', format: 'utf8' } then level-transcoder will create transcoder encodings on demand with names 'example+buffer' and 'example+view'.
| Name | Input | Format | Output |
| :--------------------------- | :------------------------- | :--------- | :-------------- |
| 'buffer+view' | Buffer, Uint8Array, String | 'view' | Buffer |'view+buffer'
| | Uint8Array, Buffer, String | 'buffer' | Uint8Array |'utf8+view'
| | String, Buffer, Uint8Array | 'view' | String |'utf8+buffer'
| | String, Buffer, Uint8Array | 'buffer' | String |'json+view'
| | Any JSON type | 'view' | As input |'json+buffer'
| | Any JSON type | 'buffer' | As input |'hex+view'
| 1 | String (hex), Buffer | 'view' | String (hex) |'base64+view'
| 1 | String (base64), Buffer | 'view' | String (base64) |
1 Unlike other encodings that transcode to 'view', these depend on Buffer at the moment and thus don't work in browsers if a shim is not included by JavaScript bundlers like Webpack and Browserify.
Various modules in the ecosystem, in and outside of Level, can be used with level-transcoder although they follow different interfaces. Common between the interfaces is that they have encode() and decode() methods. The terms "codec" and "encoding" are used interchangeably in the ecosystem. Passing these encodings through Transcoder#encoding() (which is done implicitly when used in an abstract-level database) results in normalized encoding objects as described further below.
| Module | Format | Interface | Named |
| :----------------------------------------- | :--------------- | :---------------------------------- | :---- |
| [protocol-buffers][protocol-buffers] | buffer | [level-codec][level-codec] | ❌ |charwise
| [][charwise] | utf8 | [level-codec][level-codec] | ✅ |bytewise
| [][bytewise] | buffer | [level-codec][level-codec] | ✅ |lexicographic-integer-encoding
| [][lexint] | buffer, utf8 | [level-codec][level-codec] | ✅ |abstract-encoding
| [][abstract-enc] | buffer | [abstract-encoding][abstract-enc] | ❌ |multiformats
| [][js-multiformats] | view | [multiformats][blockcodec] | ✅ |
Those marked as not named are modules that export or generate encodings that don't have a name property (or type as an alias). We call these _anonymous encodings_. They can only be used as objects and not by name. Passing an anonymous encoding through Transcoder#encoding() does give it a name property for compatibility, but the value of name is not deterministic.
#### transcoder = new Transcoder(formats)
Create a new transcoder, providing the formats that are supported by a database (or other). The formats argument must be an array containing one or more of 'buffer', 'view', 'utf8'. The returned transcoder instance is stateful, in that it contains a set of cached encoding objects.
#### encoding = transcoder.encoding(encoding)
Returns the given encoding argument as a normalized encoding object that follows the level-transcoder encoding interface. The encoding argument may be:
- A string to select a known encoding by its name
- An object that follows one of the following interfaces: level-transcoder, level-codec, [abstract-encoding][abstract-enc], [multiformats][blockcodec]encoding(x)
- A previously normalized encoding, such that equals encoding(encoding(x)).
Results are cached. If the encoding argument is an object and it has a name then subsequent calls can refer to that encoding by name.
Depending on the formats provided to the Transcoder constructor, this method may return a _transcoder encoding_ that translates the desired encoding from / to a supported format. Its encode() and decode() methods will have respectively the same input and output types as a non-transcoded encoding, but its name property will differ.
#### encodings = transcoder.encodings()
Get an array of encoding objects. This includes:
- Encodings for the formats that were passed to the Transcoder constructortranscoder.encoding()
- Custom encodings that were passed to
- Transcoder encodings for either.
#### data = encoding.encode(data)
Encode data.
#### data = encoding.decode(data)
Decode data.
#### encoding.name
Unique name. A string.
#### encoding.commonName
Common name, computed from name. If this encoding is a transcoder encoding, name will be for example 'json+view' and commonName will be just 'json'. Else name will equal commonName.
#### encoding.format
Name of the (lower-level) encoding used by the return value of encode(). One of 'buffer', 'view', 'utf8'. If name equals format then the encoding can be assumed to be idempotent, such that encode(x) equals encode(encode(x)).
Custom encodings must follow the following interface:
`ts`
interface IEncoding
name: string
format: 'buffer' | 'view' | 'utf8'
encode: (data: TIn) => TFormat
decode: (data: TFormat) => TOut
}
With npm do:
`
npm install level-transcoder
Level/transcoder` is an OPEN Open Source Project. This means that:
> Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.
See the Contribution Guide for more details.
Support us with a monthly donation on Open Collective and help us continue our work.
[level-badge]: https://leveljs.org/img/badge.svg
[level-codec]: https://github.com/Level/codec
[leveldown]: https://github.com/Level/leveldown
[protocol-buffers]: https://github.com/mafintosh/protocol-buffers
[charwise]: https://github.com/dominictarr/charwise
[bytewise]: https://github.com/deanlandolt/bytewise
[lexint]: https://github.com/vweevers/lexicographic-integer-encoding
[abstract-enc]: https://github.com/mafintosh/abstract-encoding
[js-multiformats]: https://github.com/multiformats/js-multiformats
[blockcodec]: https://github.com/multiformats/js-multiformats/blob/master/src/codecs/interface.ts