Signed and encrypted streams with JOSE (JWE, JWS, JWK) and JSONL
npm install jose-streamSigned and encrypted streams with
JOSE
(JWE,
JWS,
JWK) and
JSONL/NDJSON
JWE and JWS have become popular formats for the exchange of encrypted and signed
data on the web. While they are well-suited for encryption and signing of short
messages, they're less suitable for long messages and large files owing to the
lack of streaming JSON or JWE parsers. Typically an entire JWE or JWS is held in
memory during encryption/decryption/signing/verification, which imposes limits
on scalability in memory-constrained and server environments. In addition,
multiple layers of Base64 encoding result in additional bandwidth and storage
overhead when utilizing sign-encrypt or sign-encrypt-sign pipelines.
JOSE-Stream proposes a streaming signing and encryption format based on JWS and
JWE. Plaintext is signed, compressed, and split into fixed-length chunks. Chunks
are encrypted with JWE and—along with a JWE header and JWS signatures—are
streamed in the JSONL line-delimited JSON format. Compression and signatures are
both optional.
For implementation examples see
JOST,
a command line tool for working with JOSE streams, which depends upon
this jose-stream package.
- Beta
- Until it hits 1.0 there will be frequent changes to API and format
- Some features not functional or buggy
- Bug reports welcome
- PRs welcome but get in touch first
``bash`
$ npm install jose-stream
- Newline-delimited UTF-8, one JSON per line.
- LF is preferred, but CRLF is allowed, as JSON parsers will eat the CR as
whitespace
- Each line contains one complete JWE or JWS serialization, which itself must
not contain any unescaped newline characters
- All JWE or JWS instances must contain a "seq" sequence property in their
protected headers
- The value of the "seq" sequence property is an unsigned integer, starting at
0, and incrementing by 1 with each subsequent JWE or JWS instance
- A JWE or JWS instance with a missing or out-of-order "seq" sequence value
must invalidate the entire JSON-Stream
The header is a JWE using the
fully general JWE JSON Serialization syntax.
Its "typ" property is "jose-stream". It carries:
- In the ciphertext, the secret key used to encrypt all subsequent JWE instances
- (Optionally) the public key corresponding to the private key used to sign all
subsequent JWS instances
- The identities of the digest, compression, and encryption algorithms used with
subsequent JWE and JWS instances, in the "dig", "cmp", and "enc" properties,
respectively
A tag signature is a JWS signature of the digest of the concatenation of
base64url decoded authentication tags from all prior JWEs. In the case of
the header tag signature, there is only one prior JWE — the header.
The header tag signature is a JWS using the
flattened JWS JSON Serialization syntax.
Its "typ" property is "tag". It is produced by signing the base64url decoded
"tag" value of the preceding header instance using the private key corresponding
to the "pub" public-key included in the header instance. It is conditional, and
is required if the header instance includes a public key in its "pub" value.
Otherwise it must not appear in the jose-stream.
A body instance is a JWE using the
flattened JWE JSON Serialization syntax.
Its "typ" property is "bdy". It is produced by encrypting a chunk of the
(optionally compressed) plaintext using the secret key encrypted and base64url
encoded in the ciphertext property of the header instance. Multiple body
instances are allowed, one for each fixed-size chunk of the compressed
plaintext. The final body instance in the jose-stream must include the boolean
value true in its "end" property.
The content signature is a JWE using the
flattened JWE JSON Serialization syntax.
Its "typ" property is "sig". It is produced by encrypting a JWS instance using
the secret key encrypted and base64url encoded in the ciphertext property of the
header instance. The JWS instance is produced by signing a digest of the
plaintext using the private key corresponding to the "pub" public-key included
in the header instance. It is conditional, and is required if the header
instance includes a public key in its "pub" value. Otherwise it must not appear
in the jose-stream.
A tag signature is a JWS signature of the digest of the concatenation of
base64url decoded authentication tags from all prior JWEs.
The final tag signature is a JWS using the
flattened JWS JSON Serialization syntax.
Its "typ" property is "tag". It is produced by signing the base64url decoded and
concatenated "tag" values of all preceding JWE instances using the private key
corresponding to the "pub" public-key included in the header instance. It is
conditional, and is required if the header instance includes a public key in its
"pub" value. Otherwise it must not appear in the jose-stream.
Here's an incomplete and non-standard
EBNF-ish grammar
describing the format:
jose-stream = header, [ tag-signature ], { body }, body-end,
[ content-signature ], [tag-signature ];header = general-jwe( {
protected: {
typ: "jose-stream", pub: public-key,
dig: digest-type, cmp: compression, enc: encryption,
seq: 0
},
ciphertext: base64url( encrypt( body-key ) )
} ), newline;tag-signature = flattened-jws( {
protected: {
typ: "tag", seq: sequence, b64: false
},
signature: base64url( sign( digest( tag, { tag } ) ) )
} ), newline;body-end = body( {
protected: { end: true }
} );body = flattened-jwe( {
protected: {
typ: "bdy", seq: sequence
},
ciphertext: base64url( encrypt( chunk( compress(
plaintext
) ) ) )
} ), newline;content-signature = flattened-jwe( {
protected: {
typ: "sig", alg: "dir", enc: encryption, seq: sequence
},
ciphertext: base64url( encrypt( flattened-jws( {
protected: { b64: false },
signature: base64url( sign( digest( plaintext ) ) )
} ) ) )
} ), newline;general-jwe = ? RFC 7516 § 7.2.1 General JWE JSON Serialization ?;
flattened-jwe = ? RFC 7516 § 7.2.2 Flattened JWE JSON Serialization ?;
flattened-jws = ? RFC 7515 § 7.2.2 Flattened JWS JSON Serialization ?;
jwk = ? RFC 7517 § 4 JSON Web Key (JWK) Format ?;
public-key = jwk;
body-key = random( encryption );
tag = ? base64urldecode(jwe.tag) of preceding JWE instance ?;
digest-type = "sha256" | "sha384" | "sha512" | "sha512-256" |
"blake2b512" | "blake2s256";
compression = "DEF" | "GZ" | "BR";
encryption = "A128CBC-HS256" | "A192CBC-HS384" | "A256CBC-HS512" |
"A128GCM" | "A192GCM" | "A256GCM";
sequence = ? Unsigned integer. Previous protected.seq + 1 ?;
base64url = ? RFC 4648 § 5 Base 64 Encoding with URL Safe Alphabet ?;
base64urldecode = ? decode of base64url to byte array ?;
encrypt = ? JWE encryption method based on protected.enc in
jose-stream header ?;
sign = ? JWS signature method based on protected.pub in jose-stream
header ?;
random = ? cryptographically secure random byte array with length
based on requirements of ( encryption ) algorithm ?;
chunk = ? split of input byte array into fixed-length chunks, not
to exceed 1.5 MiB in length ?;
digest = ? cryptographic hash function defined by protected.dig in
jose-stream header ?;
compress = ? compression function defined by protected.cmp in
jose-stream header ?;
newline = "\n";
Magnifying glass not included
An example of a jose-stream formatted file with a single recipient, with
protected headers decoded from base64url to JSON, pretty-printed, and with
each newline marked explicitly:
`json``
{
"protected": {
"typ": "jose-stream",
"pub": {
"crv": "Ed25519",
"x": "cXbDRvACe2NSsaTpOOWUZv_mH1wiPoE6Y5Jff4IyWiM",
"kty": "OKP"
},
"dig": "blake2b512",
"cmp": "DEF",
"enc": "A256GCM",
"seq": 0,
"epk": {
"x": "qBBa0dpSYokFMmHt6s0KIKs1cFfqXtfJnXKV8y169T0",
"crv": "X25519",
"kty": "OKP"
}
},
"recipients": [
{
"encrypted_key": "P2MEQVOueTL7GLCawJJCp0_hvDks78dYKkWQzOa6tf1AsMfHsqGEGQ",
"header": {
"alg": "ECDH-ES+A256KW",
"kid": "OhHmvNaYntMdpoH9LlPyUg9svcMzp3Jqj6zCjKK_rGs"
}
}
],
"iv": "OnFMc_YacqlaF7ON",
"ciphertext": "b-Lr_JteXKj9yt22cMTP37n1E9yrPLhqK5l0pdfEof_lg8PHe2TqRG5hSPNpzCOhG0iOMMb-VMBV4KjBFwg9",
"tag": "46GkceBB8ALz1kE7HwbrCQ"
}
{
"protected": {
"typ": "tag",
"alg": "EdDSA",
"crv": "Ed25519",
"b64": false,
"seq": 1
},
"signature": "7UoTDnGuC-RYE2pI1lUgbcWSn057GY5vaugPXijKmDVR_n9iRdwa0G36KAYWx7dLNCT93yYIlslgAgFrZIh9Dw"
}
{
"protected": {
"typ": "bdy",
"alg": "dir",
"enc": "A256GCM",
"end": true,
"seq": 2
},
"iv": "kgVytUX9Xx24SgaG",
"ciphertext": "aWkkgmPgGUspNW_kWjW_tL3G947dD6IUA3-RTPeg2ssjqDYBbhGQzlj1SPdtZdMN0Tt2g4xAEkeqUjH2Q393h-FZ5ZUux_P8ARqba__Keqn6mJukEnMNqlVPZDaOersUSZ3lBxGMI9pUWFbl-9mYQEDxK1xt0UwUIpXwnRSdMOJynyWWhMrKzFNvUTIQ3UMwDOTB33vH8yj-8LtlTrvFwHJH_Lw6mrPTJSmd1QyTY8lvMgECMsqEGGBqsISljGRWMA4j5D-wpfLozaiHyd4G6MTjMBHZdg",
"tag": "iV2ghlZlF3SlLEw6DDZ5xg"
}
{
"protected": {
"typ": "sig",
"alg": "dir",
"enc": "A256GCM",
"seq": 3
},
"iv": "zTXmMgqpkB-0-t2R",
"ciphertext": "gZ3ilSyj1mWSg4kH0JiscyeP8U1V9UX1sTlijc9IaoUOYN2BaH4_In7Pzn1pHBWNCyV-zyhKBj57iayLlb2v_Ne4zAV1adt7E0soF70rNcjjlncPi67zPgXnYYLICJ_4Xg4l1UEwbaeGP2eTIQtDA8WQdAvPnfea4e6RSwy3_358y0EZBctQF-6S4LLlcyqpOMT_j8rqpzbIJTq6sSKbIbpnnF_6Ygl307WLVFLR9Q",
"tag": "JVN9qtLtm75u9crHcB4RXg"
}
{
"protected": {
"typ": "tag",
"alg": "EdDSA",
"crv": "Ed25519",
"b64": false,
"seq": 4
},
"signature": "swqq_9RkAsUkRjcrfs979UlqZOix35C1D-dFGzcbo7h4cTDMxq07Ee7N4x983uvG-DDgdnMoYwjBJsBgpTxMCg"
}
- Be secure. Follow best practices.
- Signing always signs both plaintext and ciphertext.
Why?
- Work within the standards: Read and write JOSE-Stream encoded streams
utilizing existing JOSE framework libraries. Try to invent as little as
possible.
- Allow stream readers to invalidate a stream as early as is practical, e.g.
after reading a header without a valid public key, or a missing header tag
signature, or a body chunk that can't be decrypted.
- Eventual binary format(s) (BSON? protobuf?). This should not be difficult
given that JOSE uses base64url encoding to establish boundaries between
format-sensitive and format-insensitive portions of the standard
(see: RFC 7165 § 6.3 D2 Avoid JSON canonicalization to the extent possible.)
- libsodium secretstream
- Tink Streaming AEAD
- Miscreant STREAM
- age STREAM