token-types

A primitive token library used to read and write from a Uint8Array.
Although it is possible to use this module directly, it is primarily designed to be used with strtok3 tokenizer.

Compatibility

Module: version 5 migrated from CommonJS to pure ECMAScript Module (ESM).
JavaScript is compliant with ECMAScript 2019 (ES10).

Installation

``sh
npm install --save token-types
`
Usually in combination with strtok3:
`sh
npm install --save strtok3
`

Using TypeScript you should also install @tokenizer/token as a development
dependency:

`shell
npm install --save-dev @tokenizer/token
`

Example

`js
import * as strtok3 from 'strtok3';
import * as token from 'token-types';

(async () => {

const tokenizer = await strtok3.fromFile("somefile.bin");
try {
const myNumber = await tokenizer.readToken(token.Float32_BE);
console.log(My number: ${myNumber});
} finally {
tokenizer.close(); // Close the file
}
})();
`

Tokens

$3

token-types supports a wide variety of numeric tokens out of the box:

| Token | Number | Bits | Endianness |
|---------------|------------------|------|----------------|
| UINT8 | Unsigned integer | 8 | n/a |
| UINT16_BE | Unsigned integer | 16 | big endian |
| UINT16_LE | Unsigned integer | 16 | little endian |
| UINT24_BE | Unsigned integer | 24 | big endian |
| UINT24_LE | Unsigned integer | 24 | little endian |
| UINT32_BE | Unsigned integer | 32 | big endian |
| UINT32_LE | Unsigned integer | 32 | little endian |
| UINT64_BE | Unsigned integer | 64 | big endian |
| UINT64_LE* | Unsigned integer | 64 | little endian |
| INT8 | Signed integer | 8 | n/a |
| INT16_BE | Signed integer | 16 | big endian |
| INT16_LE | Signed integer | 16 | little endian |
| INT24_BE | Signed integer | 24 | big endian |
| INT24_LE | Signed integer | 24 | little endian |
| INT32_BE | Signed integer | 32 | big endian |
| INT32_LE | Signed integer | 32 | little endian |
| INT64_BE | Signed integer | 64 | big endian |
| INT64_LE* | Signed integer | 64 | little endian |
| Float16_BE | IEEE 754 float | 16 | big endian |
| Float16_LE | IEEE 754 float | 16 | little endian |
| Float32_BE | IEEE 754 float | 32 | big endian |
| Float32_LE | IEEE 754 float | 32 | little endian |
| Float64_BE | IEEE 754 float | 64 | big endian |
| Float64_LE | IEEE 754 float | 64 | little endian |
| Float80_BE* | IEEE 754 float | 80 | big endian |
| Float80_LE* | IEEE 754 float | 80 | little endian |

(*) The tokens exceed the JavaScript IEEE 754 64-bit Floating Point precision, decoding and encoding is best effort based.

$3

StringType decoding is using @borewit/text-codec.

$3

Custom tokens can be added, suitable for reading binary files or network messages:
`js
ExtendedHeader = {
len: 10,

get: (buf, off) => {
return {
// Extended header size
size: Token.UINT32_BE.get(buf, off),
// Extended Flags
extendedFlags: Token.UINT16_BE.get(buf, off + 4),
// Size of padding
sizeOfPadding: Token.UINT32_BE.get(buf, off + 6),
// CRC data present
crcDataPresent: common.strtokBITSET.get(buf, off + 4, 31)
};
}
};
``

Licence

This project is licensed under the MIT License. Feel free to use, modify, and distribute as needed.