Content Addressable aRchive format reader and writer
npm install @ipld/car
> Content Addressable aRchive format reader and writer
``console`
$ npm i @ipld/car
See also:
- Original Go implementation
- CAR specification
- IPLD
- Example
- Usage
- API
- License
`js
// Create a simple .car file with a single block and that block's CID as the
// single root. Then read the .car and fetch the block again.
import fs from 'fs'
import { Readable } from 'stream'
import { CarReader, CarWriter } from '@ipld/car'
import * as raw from 'multiformats/codecs/raw'
import { CID } from 'multiformats/cid'
import { sha256 } from 'multiformats/hashes/sha2'
async function example() {
const bytes = new TextEncoder().encode('random meaningless bytes')
const hash = await sha256.digest(raw.encode(bytes))
const cid = CID.create(1, raw.code, hash)
// create the writer and set the header with a single root
const { writer, out } = await CarWriter.create([cid])
Readable.from(out).pipe(fs.createWriteStream('example.car'))
// store a new block, creates a new file entry in the CAR archive
await writer.put({ cid, bytes })
await writer.close()
const inStream = fs.createReadStream('example.car')
// read and parse the entire stream in one go, this will cache the contents of
// the car in memory so is not suitable for large files.
const reader = await CarReader.fromIterable(inStream)
// read the list of roots from the header
const roots = await reader.getRoots()
// retrieve a block, as a { cid:CID, bytes:UInt8Array } pair from the archive
const got = await reader.get(roots[0])
// also possible: for await (const { cid, bytes } of CarIterator.fromIterable(inStream)) { ... }
console.log(
'Retrieved [%s] from example.car with CID [%s]',
new TextDecoder().decode(got.bytes),
roots[0].toString()
)
}
example().catch((err) => {
console.error(err)
process.exit(1)
})
`
Will output:
`
Retrieved [random meaningless bytes] from example.car with CID [bafkreihwkf6mtnjobdqrkiksr7qhp6tiiqywux64aylunbvmfhzeql2coa]
See the examples directory for more.
@ipld/car is consumed through factory methods on its different classes. Each
class represents a discrete set of functionality. You should select the classes
that make the most sense for your use-case.
Please be aware that @ipld/car does not validate that block data matches
the paired CIDs when reading a CAR. See the
verify-car.js example for one possible approach to
validating blocks as they are read. Any CID verification requires that the hash
function that was used to generate the CID be available, the CAR format does
not restrict the allowable multihashes.
The basic CarReader class is consumed via:
`js`
import { CarReader } from '@ipld/car/reader'
import { CarBufferReader } from '@ipld/car/buffer-reader'
Or alternatively: import { CarReader } from '@ipld/car'. CommonJS require
will also work for the same import paths and references.
CarReader is useful for relatively small CAR archives as it buffers theCarReader
entirety of the archive in memory to provide access to its data. This class is
also suitable in a browser environment. The class providesget(key)
random-access and has(key)
methods as well as iterators for ] andcids()].
CarReader can be instantiated from aUint8Array
single or fromAsyncIterable
an of Uint8Arrays (note thatAsyncIterable
Node.js streams are s and can be consumed in this way).
CarBufferReader works exactly the same way as CarReader but all methods are synchronous.
The CarIndexedReader class is a special form of CarReader and can be
consumed in Node.js only (not in the browser) via:
`js`
import { CarIndexedReader } from '@ipld/car/indexed-reader'
Or alternatively: import { CarIndexedReader } from '@ipld/car'. CommonJSrequire will also work for the same import paths and references.
A CarIndexedReader provides the same functionality as CarReader but isclose()
instantiated from a path to a CAR file and also
adds a method that must be called when the reader
is no longer required, to clean up resources.
CarIndexedReader performs a single full-scan of a CAR file, collecting a listCID
of s and their block positions in the archive. It then performsget()
random-access reads when blocks are requested via and the blocks() andcids() iterators.
This class may be sutiable for random-access (primarily via has() and get())
to relatively large CAR files.
`js`
import { CarBlockIterator } from '@ipld/car/iterator'
// or
import { CarCIDIterator } from '@ipld/car/iterator'
Or alternatively:
import { CarBlockIterator, CarCIDIterator } from '@ipld/car'. CommonJSrequire will also work for the same import paths and references.
These two classes provide AsyncIterables to the blocks or just the CIDs
contained within a CAR archive. These are efficient mechanisms for scanning an
entire CAR archive, regardless of size, if random-access to blocks is not
required.
CarBlockIterator and CarCIDIterator can be instantiated from aUint8Array
single (seeCarBlockIterator.fromBytes() andCarCIDIterator.fromBytes()) or fromAsyncIterable
an of Uint8Arrays (seeCarBlockIterator.fromIterable() andCarCIDIterator.fromIterable())—note thatAsyncIterable
Node.js streams are s and can be consumed in this way.
The CarIndexer class can be used to scan a CAR archive and provide indexing
data on the contents. It can be consumed via:
`js`
import CarIndexer from '@ipld/car/indexed-reader'
Or alternatively: import { CarIndexer } from '@ipld/car'. CommonJSrequire will also work for the same import paths and references.
This class is used within CarIndexedReader and is only
useful in cases where an external index of a CAR needs to be generated and used.
The index data can also be used with
CarReader.readRaw()] to fetch block data directly from
a file descriptor using the index data for that block.
A CarWriter is used to create new CAR archives. It can be consumed via:
`js`
import CarWriter from '@ipld/car/writer'
Or alternatively: import { CarWriter } from '@ipld/car'. CommonJSrequire will also work for the same import paths and references.
Creation of a CarWriter involves a "channel", or a
{ writer:CarWriter, out:AsyncIterable pair. The writer sideput()
of the channel is used to blocks, while the out
side of the channel emits the bytes that form the encoded CAR archive.
In Node.js, you can use the
Readable.from()
API to convert the out AsyncIterable to a standard Node.js stream, or it canstream.pipeline()
be directly fed to a.
* class CarReader
* async CarReader#getRoots()
* async CarReader#has(key)
* async CarReader#get(key)
async CarReader#blocks()
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
* [](#CarBufferWriter_close)CarBufferWriter.blockLength(Block)
*
*
*
* [](#CarBufferWriter__estimateHeaderLength__rootCount______rootByteLength____)CarBufferWriter.createWriter(buffer[, options])
* [](#CarBufferWriter__createWriter__buffer______options____)async decoder.readHeader(reader)
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
Properties:
* version (number): The version number of the CAR referenced by this1
reader (should be or 2).
Provides blockstore-like access to a CAR.
Implements the RootsReader interface:getRoots(). And the BlockReader interface:get(), has(),blocks() (defined as a BlockIterator) andcids() (defined as a CIDIterator).
Load this class with either import { CarReader } from '@ipld/car/reader'
(). Orimport { CarReader } from '@ipld/car' (const { CarReader } = require('@ipld/car')).
The former will likely result in smaller bundle sizes where this is
important.
* Returns: Promise
Get the list of roots defined by the CAR referenced by this reader. May be
zero or more CIDs.
* key (CID)
* Returns: Promise
Check whether a given CID exists within the CAR referenced by this
reader.
* key (CID)
* Returns: Promise<(Block|undefined)>
Fetch a Block (a { cid:CID, bytes:Uint8Array } pair) from the CARCID
referenced by this reader matching the provided . In the case whereCID
the provided doesn't exist within the CAR, undefined will be
returned.
* Returns: AsyncGenerator
Returns a BlockIterator (AsyncIterable) that iterates over allBlock
of the s ({ cid:CID, bytes:Uint8Array } pairs) contained within
the CAR referenced by this reader.
* Returns: AsyncGenerator
Returns a CIDIterator (AsyncIterable) that iterates over all ofCID
the s contained within the CAR referenced by this reader.
* bytes (Uint8Array)
* Returns: Promise
Instantiate a CarReader from a Uint8Array blob. This performs aCarReader
decode fully in memory and maintains the decoded state in memory for full
access to the data via the API.
* asyncIterable (AsyncIterable
* Returns: Promise
Instantiate a CarReader from a AsyncIterable, such asCarReader
a modern Node.js stream.
This performs a decode fully in memory and maintains the decoded state in
memory for full access to the data via the API.
Care should be taken for large archives; this API may not be appropriate
where memory is a concern or the archive is potentially larger than the
amount of memory that the runtime can handle.
* fd (FileHandle|number): A file descriptor from thefs
Node.js module. Either an integer, from fs.open() or a FileHandle
from .blockIndex
* (BlockIndex): An index pointing to the location of theBlockIndex
Block required. This should take the form:{cid:CID, blockLength:number, blockOffset:number}
.
* Returns: Promise: A { cid:CID, bytes:Uint8Array } pair.
Reads a block directly from a file descriptor for an open CAR file. This
function is only available in Node.js and not a browser environment.
This function can be used in connection with CarIndexer which emits
the BlockIndex objects that are required by this function.
The user is responsible for opening and closing the file used in this call.
Properties:
* version (number): The version number of the CAR referenced by this1
reader (should be ).
A form of CarReader that pre-indexes a CAR archive from a file and
provides random access to blocks within the file using the index data. This
function is only available in Node.js and not a browser environment.
For large CAR files, using this form of CarReader can be singificantly moreCID
efficient in terms of memory. The index consists of a list of s andCarIndexer
their location within the archive (see ). For large numbersNODE_OPTIONS="--max-old-space-size=16384"
of blocks, this index can also occupy a significant amount of memory. In some
cases it may be necessary to expand the memory capacity of a Node.js instance
to allow this index to fit. (e.g. by running with).
As an CarIndexedReader instance maintains an open file descriptor for itsCarReader#close
CAR file, an additional method is attached. This
_must_ be called to have full clean-up of resources after use.
Load this class with either
import { CarIndexedReader } from '@ipld/car/indexed-reader'
(). Orimport { CarIndexedReader } from '@ipld/car'
(). The former will likely
result in smaller bundle sizes where this is important.
* Returns: Promise
* key (CID)
* Returns: Promise
See CarReader#has
* key (CID)
* Returns: Promise<(Block|undefined)>
See CarReader#get
* Returns: AsyncGenerator
See CarReader#blocks
* Returns: AsyncGenerator
See CarReader#cids
* Returns: Promise
Close the underlying file descriptor maintained by this CarIndexedReader.
This must be called for proper resource clean-up to occur.
* path (string)
* Returns: Promise
Instantiate an CarIndexedReader from a file with the provided
path. The CAR file is first indexed with a full path that collects CIDs
and block locations. This index is maintained in memory. Subsequent reads
operate on a read-only file descriptor, fetching the block from its in-file
location.
For large archives, the initial indexing may take some time. The returned
Promise will resolve only after this is complete.
Properties:
* version (number): The version number of the CAR referenced by this1
iterator (should be ).
Provides an iterator over all of the Blocks in a CAR. Implements aBlockIterator interface, or AsyncIterable. Where a Block is{ cid:CID, bytes:Uint8Array }
a pair.
As an implementer of AsyncIterable, this class can be used directly in afor await (const block of iterator) {} loop. Where the iterator isCarBlockiterator.fromBytes
constructed using orCarBlockiterator.fromIterable.
An iteration can only be performce _once_ per instantiation.
CarBlockIterator also implements the RootsReader interface and providesgetRoots()
the method.
Load this class with either
import { CarBlockIterator } from '@ipld/car/iterator'
(). Orimport { CarBlockIterator } from '@ipld/car'
().
* Returns: Promise
Get the list of roots defined by the CAR referenced by this iterator. May be
zero or more CIDs.
* bytes (Uint8Array)
* Returns: Promise
Instantiate a CarBlockIterator from a Uint8Array blob. RatherCarReader.fromBytes
than decoding the entire byte array prior to returning the iterator, as in, only the header is decoded and the remainderBlock
of the CAR is parsed as the s as yielded.
* asyncIterable (AsyncIterable
* Returns: Promise
Instantiate a CarBlockIterator from a AsyncIterable,CarReader.fromIterable
such as a modern Node.js stream.
Rather than decoding the entire byte array prior to returning the iterator,
as in , only the header is decoded and theBlock
remainder of the CAR is parsed as the s as yielded.
Properties:
* version (number): The version number of the CAR referenced by this1
iterator (should be ).
Provides an iterator over all of the CIDs in a CAR. Implements aCIDIterator interface, or AsyncIterable. Similar toCarBlockIterator but only yields the CIDs in the CAR.
As an implementer of AsyncIterable, this class can be used directly in afor await (const cid of iterator) {} loop. Where the iterator isCarCIDiterator.fromBytes
constructed using orCarCIDiterator.fromIterable.
An iteration can only be performce _once_ per instantiation.
CarCIDIterator also implements the RootsReader interface and providesgetRoots()
the method.
Load this class with either
import { CarCIDIterator } from '@ipld/car/iterator'
(). Orimport { CarCIDIterator } from '@ipld/car'
().
* Returns: Promise
Get the list of roots defined by the CAR referenced by this iterator. May be
zero or more CIDs.
* bytes (Uint8Array)
* Returns: Promise
Instantiate a CarCIDIterator from a Uint8Array blob. RatherCarReader.fromBytes
than decoding the entire byte array prior to returning the iterator, as in, only the header is decoded and the remainderCID
of the CAR is parsed as the s as yielded.
* asyncIterable (AsyncIterable
* Returns: Promise
Instantiate a CarCIDIterator from a AsyncIterable,CarReader.fromIterable
such as a modern Node.js stream.
Rather than decoding the entire byte array prior to returning the iterator,
as in , only the header is decoded and theCID
remainder of the CAR is parsed as the s as yielded.
Properties:
* version (number): The version number of the CAR referenced by this1
reader (should be ).
Provides an iterator over all of the Blocks in a CAR, returning their CIDsAsyncIterable
and byte-location information. Implements an .BlockIndex
Where a is a{ cid:CID, length:number, offset:number, blockLength:number, blockOffset:number }.
As an implementer of AsyncIterable, this class can be used directly in afor await (const blockIndex of iterator) {} loop. Where the iterator isCarIndexer.fromBytes
constructed using orCarIndexer.fromIterable.
An iteration can only be performce _once_ per instantiation.
CarIndexer also implements the RootsReader interface and providesgetRoots()
the method.
Load this class with either
import { CarIndexer } from '@ipld/car/indexer'
(). Orimport { CarIndexer } from '@ipld/car'
(). The former will likely
result in smaller bundle sizes where this is important.
* Returns: Promise
Get the list of roots defined by the CAR referenced by this indexer. May be
zero or more CIDs.
* bytes (Uint8Array)
* Returns: Promise
Instantiate a CarIndexer from a Uint8Array blob. Only the header
is decoded initially, the remainder is processed and emitted via the
iterator as it is consumed.
* asyncIterable (AsyncIterable
* Returns: Promise
Instantiate a CarIndexer from a AsyncIterable,
such as a modern Node.js stream.
is decoded initially, the remainder is processed and emitted via the
iterator as it is consumed.
Provides a writer interface for the creation of CAR files.
Creation of a CarWriter involves the instatiation of an input / output pairWriterChannel
in the form of a , which is a{ writer:CarWriter, out:AsyncIterable pair. These twowriter
components form what can be thought of as a stream-like interface. The component (an instantiated CarWriter), has methods toput() new blocks and close()
the writing operation (finalising the CAR archive). The component isAsyncIterable
an that yields the bytes of the archive. This can beReadable.from()
redirected to a file or other sink. In Node.js, you can use the
API to convert this to a standard Node.js stream, or it can be directly fed
to a.
The channel will provide a form of backpressure. The Promise from awrite() won't resolve until the resulting data is drained from the out
iterable.
It is also possible to ignore the Promise from write() calls and allow
the generated data to queue in memory. This should be avoided for large CAR
archives of course due to the memory costs and potential for memory overflow.
Load this class with either
import { CarWriter } from '@ipld/car/writer'
(). Orimport { CarWriter } from '@ipld/car'
(). The former will likely
result in smaller bundle sizes where this is important.
* block (Block): A { cid:CID, bytes:Uint8Array } pair.
* Returns: Promise: The returned promise will only resolve once theout
bytes this block generates are written to the iterable.
Write a Block (a { cid:CID, bytes:Uint8Array } pair) to the archive.
* Returns: Promise
Finalise the CAR archive and signal that the out iterable should end once
any remaining bytes are written.
* Returns: number
Returns the version number of the CAR file being written
* roots (CID[]|CID|void)
* Returns: WriterChannel: The channel takes the form of{ writer:CarWriter, out:AsyncIterable
.
Create a new CAR writer "channel" which consists of a
{ writer:CarWriter, out:AsyncIterable pair.
* Returns: WriterChannel: The channel takes the form of{ writer:CarWriter, out:AsyncIterable
.
Create a new CAR appender "channel" which consists of a
{ writer:CarWriter, out:AsyncIterable pair.out
This appender does not consider roots and does not produce a CAR header.
It is designed to append blocks to an _existing_ CAR archive. It is
expected that will be concatenated onto the end of an existing
archive that already has a properly formatted header.
* bytes (Uint8Array)
* (CID[]): A new list of roots to replace the existing list in
the CAR header. The new header must take up the same number of bytes as the
existing header, so the roots should collectively be the same byte length
as the existing roots.
* Returns: Promise
Update the list of roots in the header of an existing CAR as represented
in a Uint8Array.
This operation is an _overwrite_, the total length of the CAR will not be
modified. A rejection will occur if the new header will not be the same
length as the existing header, in which case the CAR will not be modified.
It is the responsibility of the user to ensure that the roots being
replaced encode as the same length as the new roots.
The byte array passed in an argument will be modified and also returned
upon successful modification.
* fd (FileHandle|number): A file descriptor from thefs
Node.js module. Either an integer, from fs.open() or a FileHandle
from .roots
* (CID[]): A new list of roots to replace the existing list in
the CAR header. The new header must take up the same number of bytes as the
existing header, so the roots should collectively be the same byte length
as the existing roots.
* Returns: Promise
Update the list of roots in the header of an existing CAR file. The first
argument must be a file descriptor for CAR file that is open in read and
write mode (not append), e.g. fs.open or fs.promises.open with 'r+'
mode.
This operation is an _overwrite_, the total length of the CAR will not be
modified. A rejection will occur if the new header will not be the same
length as the existing header, in which case the CAR will not be modified.
It is the responsibility of the user to ensure that the roots being
replaced encode as the same length as the new roots.
This function is only available in Node.js and not a browser
environment.
A simple CAR writer that writes to a pre-allocated buffer.
* root (CID)
*
* Returns: CarBufferWriter
Add a root to this writer, to be used to create a header when the CAR is
finalized with close()
* block (Block): A { cid:CID, bytes:Uint8Array } pair.
* Returns: CarBufferWriter
Write a Block (a { cid:CID, bytes:Uint8Array } pair) to the archive.
Throws if there is not enough capacity.
* options (object, optional)
* (boolean, optional)
* Returns: Uint8Array
Finalize the CAR and return it as a Uint8Array.
* block (Block)
* Returns: number
Calculates number of bytes required for storing given block in CAR. Useful in
estimating size of an ArrayBuffer for the CarBufferWriter.
* rootLengths (number[])
* Returns: number
Calculates header size given the array of byteLength for roots.
* options (object)
* (CID[])
* Returns: number
Calculates header size given the array of roots.
* rootCount (number)
* (number, optional)
* Returns: number
Estimates header size given a count of the roots and the expected byte length
of the root CIDs. The default length works for a standard CIDv1 with a
single-byte multihash code, such as SHA2-256 (i.e. the most common CIDv1).
* buffer (ArrayBuffer)
* (object, optional)
* (CID[], optional)
* (number, optional)
* (number, optional)
* (number, optional)
* Returns: CarBufferWriter
Creates synchronous CAR writer that can be used to encode blocks into a given
buffer. Optionally you could pass byteOffset and byteLength to specify aroots
range inside buffer to write into. If car file is going to have youoptions.roots
need to either pass them under (from which header size willoptions.headerSize
be calculated) or provide to allocate required spaceroots
in the buffer. You may also provide known and headerSize to
allocate space for the roots that may not be known ahead of time.
Note: Incorrect headerSize may lead to copying bytes inside a buffer
which will have a negative impact on performance.
* reader (BytesReader)
* (number, optional)
* Returns: Promise<(CarHeader|CarV2Header)>
Reads header data from a BytesReader. The header may either be in the formCarHeader
of a or CarV2Header depending on the CAR being read.
* reader (BytesReader)
* Returns: Promise
Reads the leading data of an individual block from CAR data from a
BytesReader. Returns a BlockHeader object which contains{ cid, length, blockLength } which can be used to either index the block
or read the block binary data.
* reader (BytesReader)
* Returns: CarDecoder
Creates a CarDecoder from a BytesReader. The CarDecoder is as asyncBytesReader
interface that will consume the bytes from the to yield aheader() and either blocks() or blocksIndex() data.
* bytes (Uint8Array)
* Returns: BytesReader
Creates a BytesReader from a Uint8Array.
* readChunk
* Returns: BytesReader
reusable reader for streams and files, we just need a way to read an
additional chunk (of some undetermined size) and a way to close the
reader when finished
* asyncIterable (AsyncIterable
* Returns: BytesReader
Creates a BytesReader from an AsyncIterable, which allows for
consumption of CAR data from a streaming source.
* reader (BytesReader)
* (number)
* Returns: BytesReader
Wraps a BytesReader in a limiting BytesReader which limits maximum readbyteLimit
to bytes. It _does not_ update pos of the originalBytesReader.
Properties:
* version (number): The version number of the CAR referenced by this1
reader (should be or 2).
Provides blockstore-like access to a CAR.
Implements the RootsBufferReader interface:getRoots(). And the BlockBufferReader interface:get(), has(),blocks() andcids().
Load this class with either import { CarBufferReader } from '@ipld/car/buffer-reader'
(). Orimport { CarBufferReader } from '@ipld/car' (const { CarBufferReader } = require('@ipld/car')).
The former will likely result in smaller bundle sizes where this is
important.
* Returns: CID[]
Get the list of roots defined by the CAR referenced by this reader. May be
zero or more CIDs.
* key (CID)
* Returns: boolean
Check whether a given CID exists within the CAR referenced by this
reader.
* key (CID)
* Returns: Block|undefined
Fetch a Block (a { cid:CID, bytes:Uint8Array } pair) from the CARCID
referenced by this reader matching the provided . In the case whereCID
the provided doesn't exist within the CAR, undefined will be
returned.
* Returns: Block[]
Returns a Block[] of the Blocks ({ cid:CID, bytes:Uint8Array } pairs) contained within
the CAR referenced by this reader.
* Returns: CID[]
Returns a CID[] of the CIDs contained within the CAR referenced by this reader.
* bytes (Uint8Array)
* Returns: CarBufferReader
Instantiate a CarBufferReader from a Uint8Array blob. This performs aCarReader
decode fully in memory and maintains the decoded state in memory for full
access to the data via the API.
* fd (number): A file descriptor from thefs
Node.js module. An integer, from fs.open().blockIndex
* (BlockIndex): An index pointing to the location of theBlockIndex
Block required. This should take the form:{cid:CID, blockLength:number, blockOffset:number}
.
* Returns: Block: A { cid:CID, bytes:Uint8Array } pair.
Reads a block directly from a file descriptor for an open CAR file. This
function is only available in Node.js and not a browser environment.
This function can be used in connection with CarIndexer which emits
the BlockIndex` objects that are required by this function.
The user is responsible for opening and closing the file used in this call.
Licensed under either of
- Apache 2.0, (LICENSE-APACHE /
- MIT (LICENSE-MIT /
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.